search

CompVis / latent-diffusion

High-Resolution Image Synthesis with Latent Diffusion Models
MIT License
11.82k stars 1.53k forks source link

multi-gpu training pickle error #212

Open lx709 opened 1 year ago

lx709 commented 1 year ago

Hello guys, thanks for sharing the code of your wonderful work. I was trying to train latent diffusion on my own datasets. When I use a single GPU, everything is ok. But when I changed to multiple GPUs, I encountered the following error. Btw, I was using PyTorch-lightening 1.8 instead of 1.4.

Traceback (most recent call last): File "train_ddpm.py", line 710, in trainer.fit(model, data) File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit call._call_and_handle_interrupt( File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs) File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/multiprocessing.py", line 113, in launch mp.start_processes( File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 179, in start_processes process.start() File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/multiprocessing/process.py", line 121, in start self._popen = self._Popen(self) File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/multiprocessing/context.py", line 284, in _Popen return Popen(process_obj) File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in init super().init(process_obj) File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/multiprocessing/popen_fork.py", line 19, in init self._launch(process_obj) File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch reduction.dump(process_obj, fp) File "/home/lix0i/miniconda3/envs/ldm_clone/lib/python3.8/multiprocessing/reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) AttributeError: Can't pickle local object 'always..inner'

lx709 commented 1 year ago

problem solved

ruairiseosamh commented 1 year ago

How did you solve this? I am having the same issue.