lukashermann
commented
1 year ago Hi @mbreuss, did you maybe run the shared memory of a smaller debug dataset before?
Try to delete the shared memory in /dev/shm/, they are called /dev/shm/train_* and /dev/shm/val_*.
Also delete the train_shm_lookup.npy and the val_shm_lookup.npy in tmp or slurm_temp directory (see here).
It's weird that it takes so long without the shared memory, there definitely seems to be something wrong. Shared memory gave us a speed-up of max. 50%. From where do you load the dataset, is it maybe via a slow network mount? Try running the PyTorch-lightning profiler to see where's the bottleneck, you can also post the results here and I compare them to our cluster. For debugging, you might want to use the hydra cmd line flags +trainer.limit_train_batches=10 and +trainer.limit_val_batches=10 (or a similar number), then you don't have to wait for the whole epoch, but you still get an estimate of the time.
mbreuss
hk-zh
Hi,
I am currently facing the error, while trying to use the shared memory variant of the dataset D. The error occurs in the following line: https://github.com/lukashermann/hulc/blob/main/hulc/datasets/utils/shared_memory_utils.py#L192 where the start_idx variable does not match the current dataset. I tried to reinstall the dataset to make sure I installed everything, but it did not help and tried to fix it.
Without using the shared memory variant the code runs without any errors. However, I have some general performance issues using a Slurm cluster with 4x3090. Currently, one epoch of training Hulc on task D takes approximate 70 hours without the shared memory. I already tried experimenting with the batch size and the number of workers, but so far it did not help. Does not using the shared memory dataset causes such a huge difference in training speed? Do you have some advice to improve the performance?
Thanks in advance! Best regards.