Hello~ Thank you for providing this SSL library and rapid response a lot of issue.
As the title described, i have encountered the error on single machine multi-gpu mode (greater then one GPU) in nnclr.sh, and it occurs the error in trainer.fit function.
The error message is shown as following figure :
Traceback (most recent call last):
File "../../../main_pretrain.py", line 207, in
main()
File "../../../main_pretrain.py", line 203, in main
trainer.fit(model, train_loader, val_loader, ckpt_path=ckpt_path)
File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 771, in fit
self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt
return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, *kwargs)
File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/strategies/launchers/spawn.py", line 82, in launch
start_method=self._start_method,
File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 179, in start_processes
process.start()
File "/opt/conda/lib/python3.7/multiprocessing/process.py", line 112, in start
self._popen = self._Popen(self)
File "/opt/conda/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
return Popen(process_obj)
File "/opt/conda/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/opt/conda/lib/python3.7/multiprocessing/popen_fork.py", line 20, in init
self._launch(process_obj)
File "/opt/conda/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/opt/conda/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'TorchGraph.create_forward_hook..after_forward_hook'
Traceback (most recent call last):
File "../../../main_pretrain.py", line 207, in
main()
File "../../../main_pretrain.py", line 203, in main
trainer.fit(model, train_loader, val_loader, ckpt_path=ckpt_path)
File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 771, in fit
self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt
return self.strategy.launcher.launch(trainer_fn, args, trainer=self, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/pytorch_lightning/strategies/launchers/spawn.py", line 82, in launch
start_method=self._start_method,
File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 179, in start_processes
process.start()
File "/opt/conda/lib/python3.7/multiprocessing/process.py", line 112, in start
self._popen = self._Popen(self)
File "/opt/conda/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
return Popen(process_obj)
File "/opt/conda/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/opt/conda/lib/python3.7/multiprocessing/popen_fork.py", line 20, in init
self._launch(process_obj)
File "/opt/conda/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/opt/conda/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'TorchGraph.create_forward_hook..after_forward_hook'
I have run the pip install -r requirement.txt
The other information about the pkg version :
However, it will go smoothing with the single GPU, the following illustrate the config of bash file
Hello~ Thank you for providing this SSL library and rapid response a lot of issue.
As the title described, i have encountered the error on single machine multi-gpu mode (greater then one GPU) in nnclr.sh, and it occurs the error in trainer.fit function. The error message is shown as following figure :
I have run the
pip install -r requirement.txt
The other information about the pkg version :
However, it will go smoothing with the single GPU, the following illustrate the config of bash file
Thanks for having a look, any suggestion will be appreciated!!