search

allenai / dolma

Data and tools for generating and inspecting OLMo pre-training data.
https://allenai.github.io/dolma/
Apache License 2.0
894 stars 90 forks source link

fix divide by 0 in gopher tagger #148

Closed peterbjorgensen closed 4 months ago

peterbjorgensen commented 4 months ago

The gopher tagger throws divide by zero errors when the documents are empty. With this commit, all the divisors are safeguarded by max(x, 1)