I have been running CryoAI with this dataset EMPIAR-10076. However, for this particular dataset, because it stuffed all images into one big .mrcs file, CryoAI pause significantly long time during training.
Specifically, after every train_num_workers steps, the training gets stuck for about 5-10 minutes and GPU usage drops to 0%. I have compared the training time of EMPIAR-10028, where image chunks are split into multiple .mrcs, and that dataset trained smoothly without pausing.
Hi,
I have been running CryoAI with this dataset EMPIAR-10076. However, for this particular dataset, because it stuffed all images into one big
.mrcsfile, CryoAI pause significantly long time during training. Specifically, after everytrain_num_workerssteps, the training gets stuck for about 5-10 minutes and GPU usage drops to 0%. I have compared the training time of EMPIAR-10028, where image chunks are split into multiple.mrcs, and that dataset trained smoothly without pausing.Is there anyway to fix this?
Thanks!