naga-karthik
commented
1 year ago This is indeed strange. Can you also post the exact arguments that you're using with main.py?
Open louisfb01 opened 1 year ago
naga-karthik
commented
1 year ago This is indeed strange. Can you also post the exact arguments that you're using with main.py?
louisfb01
commented
1 year ago Pretty much no argument, just correct paths to data and default unet, 100 epochs, and your hard coded values.
jcohenadad
commented
1 year ago @louisfb01 can you please list in this issue thread the various discussions on this topic in MONAI GH, slack, forums, etc.
louisfb01
commented
1 year ago The source seems to be a memory leak when storing to much data from a dataloader, from this. But in his case it happens running the test set whereas here it happens when training.
Solutions tried:
- Adding this line to the training script did not work torch.multiprocessing.set_sharing_strategy('file_system'), as in this issue.
ulimit was another solution but it is limited to 2000 for a user in Romane. Increasing from 1024 to 2000 did not solve the problem.received 0 items of ancdata, similar to here and this thread.The current workaround is to use num_workes=0 as in this issue and this one, and more.
New temporary fix:
torch.multiprocessing.set_sharing_strategy('file_system'), as in this issue. I believe I previously did not add it in the right section of the code by mistake. It now works as expected!jcohenadad
commented
1 year ago @louisfb01 can you please
louisfb01
commented
1 year ago Here is more information about the environment (python version 3.9.17) and the output I get from running main.py (training with PyTorch, PyTorch lightning, MONAI).
louisfb01
commented
1 year ago Updated the answer above with a new temporary fix.
Adding this line to the training script did work torch.multiprocessing.set_sharing_strategy('file_system'), as in this issue and here. I believe I previously did not add it in the right section of the code by mistake. It now works as expected!
It is not super clean but it at least allows us to train normally for now.
jcohenadad
commented
1 year ago It is not super clean
Why is that?
louisfb01
commented
1 year ago It is not super clean
Why is that?
Nevermind on that. I didn't like the idea of having to add this line and thought it was a "hard-coded" fix to a PyTorch issue, but it seems like a normal behaviour after further research. This issue can be closed with the solution of adding the torch.multiprocessing.set_sharing_strategy('file_system') line to the beginning of your training script.
jcohenadad
commented
1 year ago I just realized this magic syntax also fixed an issue for me in the past 😅 https://github.com/jcohenadad/model-seg-ms-mp2rage-monai/commit/450f72e471b33e668109f55195db92811ef15d78
naga-karthik
commented
1 year ago Is there any deeper explanation anywhere as to why this fix is working?
louisfb01
commented
1 year ago Is there any deeper explanation anywhere as to why this fix is working?
This is what I found:
torch.multiprocessing is a wrapper around the native multiprocessing module. It registers custom reducers, that use shared memory to provide shared views on the same data in different processes. Once the tensor/storage is moved to shared_memory (see sharememory()), it will be possible to send it to other processes without making any copies. (from torch documentation)
And from what I understand, this memory issue comes from using the CacheDataset and it has to do with pytorch's sharing strategy. The function torch.multiprocessing.set_sharing_strategy('file_system') will enable a flag to not create extra file descriptors, more information here.
It still seems to be a temporary fix to "having a high enough limit" in our system. This can be done increasing ulimit, but not in our case since we are limited as a user in Romane.
Quote from PyTorch doc:
Still, if your system has high enough limits, and file_descriptor is a supported strategy, we do not recommend switching to this one.
This one referring to using the file_system flag.
naga-karthik
commented
1 year ago And from what I understand, this memory issue comes from using the CacheDataset and it has to do with pytorch's sharing strategy.
ah this is a good point! Monai also provides PyTorch's native Dataset class. Could you please try using that once and remove the multiprocessing fix to see if it is really CacheDataset that's the culprit? (you just have switch to Dataset and use the right arguments, just a line of code)
louisfb01
commented
1 year ago I tried using MONAI's Dataset class instead of CacheDataset.
Using Dataset does not allow us to remove torch.multiprocessing.set_sharing_strategy('file_system').
louisfb01
commented
1 year ago New error even with torch.multiprocessing.set_sharing_strategy('file_system').
Happens both when using MONAI's Dataset class and CacheDataset.
The error occurs with all the aggregated datasets (approx. 7k total images from train/val/test) and it only happens at 73% (with CacheDataset) and 75% (with Dataset) of the first epoch:
23 Epoch 0: 75%|███████████████████████████████████████████████████████████████████████████████████████████████████████▍ | 1069/1426 [1:01:50<20:39, 3.47s/it, v_num=ymin]
24 Exception in thread Thread-7:
25 Traceback (most recent call last):
26 File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/threading.py", line 980, in _bootstrap_inner
27 self.run()
28 File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/threading.py", line 917, in run
29 self._target(*self._args, **self._kwargs)
30 File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/torch/utils/data/_utils/pin_memory.py", line 51, in _pin_memory_loop
31 do_one_step()
32 File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/torch/utils/data/_utils/pin_memory.py", line 28, in do_one_step
33 r = in_queue.get(timeout=MP_STATUS_CHECK_INTERVAL)
34 File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/multiprocessing/queues.py", line 122, in get
35 return _ForkingPickler.loads(res)
36 File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/torch/multiprocessing/reductions.py", line 324, in rebuild_storage_filename
37 storage = torch.UntypedStorage._new_shared_filename_cpu(manager, handle, size)
38 RuntimeError: unable to mmap 160 bytes from file </torch_2814340_3051998398_49246>: Cannot allocate memory (12)
39 Traceback (most recent call last):
40 File "/home/GRAMES.POLYMTL.CA/lobouz/github/contrast-agnostic-softseg-spinalcord/monai/main.py", line 636, in <module>
41 main(args)
42 File "/home/GRAMES.POLYMTL.CA/lobouz/github/contrast-agnostic-softseg-spinalcord/monai/main.py", line 536, in main
43 trainer.fit(pl_model)
44 File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 531, in fit
45 call._call_and_handle_interrupt(
46 File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
47 return trainer_fn(*args, **kwargs)
48 File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 570, in _fit_impl
49 self._run(model, ckpt_path=ckpt_path)
50 File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 975, in _run
51 results = self._run_stage()
52 File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1018, in _run_stage
53 self.fit_loop.run()
54 File "/home/GRAMES.POLYMTL.CA/lobouz/miniconda3/envs/monai_training/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 201, in runPlus, the epoch takes over 1:30 hours to run. I think the next step is to investigate using compute Canada and train with more GPUs and memory. Anyways, as @naga-karthik and I saw, Romane is getting pretty crowded nowadays and it is hard to train when you want.
jcohenadad
commented
1 year ago I think the next step is to investigate using compute Canada and train with more GPUs and memory.
👍
louisfb01
commented
1 year ago Regarding the issue above, I looked further into it, and it seems nobody can explain it (?). For now, one solution is to put num_workers=0 once again, based on this thread, with no better solution. Reminder that using num_workers=0 is not viable in our case. One epoch is close to 5 hours instead of 1:40 hours with this setting.
Some threads are also still opened, not finding the cause of the issue with another fix (will update the comment once I've tried them):
We have no root cause but this issue disappeared now after we fixed a host mem leak issue in our project. If you also met this problem, you can set the OS mmap limitation much higher as a work around. Or set the worker = 0 in dataloader also helps. From this issue.. -> requires sudo, I don't think it is pertinent.
Another potential solution is to implement our own version of the dataset class and implement this modification to transform the data into torch tensors. -> did not work.
Now working on implementing the code for Compute Canada to train with multiple GPUs and will update if we have the same problem.
naga-karthik
commented
2 months ago closing this as it is not relevant anymore -- I was able to train the model on 11 datasets at the moment
plbenveniste
commented
1 month ago Hi @naga-karthik ! I am re-opening this issue as I am facing the same problem as @louisfb01 when training using monai and large datasets. My training crashes during the validation step even though I am using torch.multiprocessing.set_sharing_strategy('file_system'). Here is my code..
The error I get is the following:
I had to set num_workers to zero in the validation to get the code to work. Is there any cleaner one around this ? (Btw I am running this on this server: GPU/CPU Server - 512 cores-threads – Epyc 3.0GHz – 128G RAM – 4 GPUs (4x Nvidia A6000) and the memory is not reaching its limit.
Hey All! So as discussed with @naga-karthik I've had some issues with the "aggregated training".
I am running into memory issues where no fixes seem to work.
I am using Naga's training script and tried pretty much all solutions I could find, but I always get a
RuntimeError('Pin memory thread exited unexpectedly'). This can be fixed by using 0 workers in these lines, but makes the training super slow (15+ mins per epoch).I tried with the exact same configurations as Naga too. The only difference is the amount of images. Plus, it works when removing most images from the dataset.json file (so when working with a much smaller set).
I am still investigating this issue...