Open ccx06 opened 5 months ago
Same issue here, have you solved it? @ccx06 @sangmichaelxie
Maybe there is some kind of OOM issue or num_workers is set too high? Does this happen every time and on all datasets?
I set the num_workers as 1. I tried to reduce the RANDOM_BATCH_SIZE in dataloader.py but it does not work.
It seems the issue is related to caching
06/25/2024 03:50:31 - INFO - datasets.arrow_dataset - Caching indices mapping at ~/doremi/preprocessed/train/Pile-CC/66/cache-abce09a69c09a6c6.arrow
When running the training code, the loading cached shuffled indexes program is stuck? How to solve it?