Expose memmap dtype in data config

allenai / OLMo

Modeling, training, eval, and inference code for OLMo

https://allenai.org/olmo

Apache License 2.0

4.2k stars 392 forks source link

Closed leon-g-xu closed 1 month ago

leon-g-xu commented 1 month ago

This is expose dtype in the data config so that we can support reading memmap files with different dtypes

leon-g-xu commented 1 month ago

@2015aroras Thanks for the review. Updated the PR to address the comment

leon-g-xu commented 1 month ago

There are style issue due to imports. Make sure to follow the steps here so that required automatic checks pass.

@2015aroras Went though instructions and added all the missing things. Can I get another approval so that it kicks off all the auto checks?