search

allenai / OLMo

Modeling, training, eval, and inference code for OLMo
https://allenai.org/olmo
Apache License 2.0
4.56k stars 459 forks source link

Expected Data Format #715

Open aflah02 opened 1 month ago

aflah02 commented 1 month ago

❓ The question

I was looking at the config files and noticed that the config files sometimes point to .npy files for the dataset. Is there any script to generate the same from a set of text files or any other format.

aman-17 commented 5 days ago

You can use Hugging Face to download the dataset directly or the Dolma toolkit. The Hugging Face repository provides easy access to the dataset, and the Dolma toolkit offers utilities to handle different data format. If you need further help, feel free to follow up.