While running preprocess.py, I have received intermittent memory access errors.
I have been using the ramen dataset from the link provided and running python preprocess.py --dataset_path data/ramen.
Randomly, it will give me memory errors like double free or corruption (!prev) Aborted (core dumped) , corrupted size vs. prev_size Aborted and additionally a pytorch error that has been patched in pytorch 2.0.1 when trying to run on multithreaded CPU.
Setting the threads to 1 with torch.set_num_threads(1) fixes the pytorch issue but makes the preprocessing speed very slow.
I was wondering if there was any advice on how to fix these issues happening during the preprocess script execution.
While running preprocess.py, I have received intermittent memory access errors.
I have been using the ramen dataset from the link provided and running
python preprocess.py --dataset_path data/ramen
.Randomly, it will give me memory errors like
double free or corruption (!prev) Aborted (core dumped)
,corrupted size vs. prev_size Aborted
and additionally a pytorch error that has been patched in pytorch 2.0.1 when trying to run on multithreaded CPU.Setting the threads to 1 with
torch.set_num_threads(1)
fixes the pytorch issue but makes the preprocessing speed very slow.I was wondering if there was any advice on how to fix these issues happening during the preprocess script execution.