search

allenai / dolma

Data and tools for generating and inspecting OLMo pre-training data.
https://allenai.github.io/dolma/
Apache License 2.0
909 stars 94 forks source link

do not overwrite tagger outputs with the same output path, fixes #113 #114

Closed peterbjorgensen closed 7 months ago

peterbjorgensen commented 7 months ago

Fix https://github.com/allenai/dolma/issues/113

soldni commented 7 months ago

ooof, this is a good catch. Adding tests before merging. Thank you!