EleutherAI / openwebtext2

MIT License
86 stars 16 forks source link

add multiprocessing to process_dump_files #1

Closed sdtblck closed 4 years ago

sdtblck commented 4 years ago

The tqdm progress bars get a bit fonked up, but it is faster :)

researcher2 commented 4 years ago

Nice, I approve.