This line seems to throw an error when ring_size < len(source_paths) for division by 0.
Basically it seems that len(tokenizer_ring) will be decremented here. The inner loop is broken, but the outer loop keeps going and divides by 0.
I'm not exactly sure what the right fix is, and it seems things work fine as long as ring_size * processes >= num_files. Any clarity here would be appreciated, thanks!
Hey @davidbrandfonbrener, thank you for the report! I think the right behavior should be to reduce ring size so that the issue doesn't happen. For now, please adjust size manually.
This line seems to throw an error when
ring_size < len(source_paths)
for division by 0.Basically it seems that
len(tokenizer_ring)
will be decremented here. The inner loop is broken, but the outer loop keeps going and divides by 0.I'm not exactly sure what the right fix is, and it seems things work fine as long as
ring_size * processes >= num_files
. Any clarity here would be appreciated, thanks!