Some race condition in url taggers

allenai / dolma

Data and tools for generating and inspecting OLMo pre-training data.

https://allenai.github.io/dolma/

Apache License 2.0

894 stars 90 forks source link

Some race condition in url taggers #138

Open peterbjorgensen opened 5 months ago

peterbjorgensen commented 5 months ago

Even with the latest git version some of the URL taggers crash if I run the taggers with multiprocessing. I can't figure out where this race condition happens. If I run the taggers with --processes 1 first and then do --processes 16 it works.

soldni commented 4 months ago

Hi @peterbjorgensen! which taggers are you trying to run? More information for me to reproduce would be great.