allenai / OLMo

Modeling, training, eval, and inference code for OLMo
https://allenai.org/olmo
Apache License 2.0
4.24k stars 399 forks source link

Add Weka storage support #575

Closed 2015aroras closed 2 months ago

2015aroras commented 2 months ago

Adds support for Weka storage, which is the primary storage in the Jupiter cluster. It might eventually be added a file system, but for now it is exposed as an S3 endpoint. The main changes are: