search

allenai / OLMo

Modeling, training, eval, and inference code for OLMo
https://allenai.org/olmo
Apache License 2.0
4.37k stars 431 forks source link

Unshard S3 checkpoints and convert to HF. #671

Closed dwadden closed 1 month ago

dwadden commented 1 month ago

Updates to script to handle unsharding and HF conversion of checkpoints stored on S3.

2015aroras commented 1 month ago

Did you mean to add me as a reviewer?

dwadden commented 1 month ago

Ah I did, sorry.

dwadden commented 1 month ago

Script to handle unsharding and HF conversion of checkpoints on Huggingface.