search

JonasSchult / Mask3D

Mask3D predicts accurate 3D semantic instances achieving state-of-the-art on ScanNet, ScanNet200, S3DIS and STPLS3D.
MIT License
528 stars 105 forks source link

support multi-gpus ? #26

Closed yichaoshen-MS closed 1 year ago

yichaoshen-MS commented 1 year ago

When I use multi-gpu by change the $gpus in config, I meet the error TypeError: cannot pickle 'MinkowskiConvolutionFunction' object

yichaoshen-MS commented 1 year ago

Traceback (most recent call last): File "/root/anaconda3/envs/mask3d/lib/python3.10/site-packages/hydra/_internal/utils.py", line 198, in run_and_report return func() File "/root/anaconda3/envs/mask3d/lib/python3.10/site-packages/hydra/_internal/utils.py", line 347, in lambda: hydra.run( File "/root/anaconda3/envs/mask3d/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 107, in run return run_job( File "/root/anaconda3/envs/mask3d/lib/python3.10/site-packages/hydra/core/utils.py", line 128, in run_job ret.return_value = task_function(task_cfg) File "/openseg_blob/code/rank3d/mask3d1105/Mask3D/main_instance_segmentation.py", line 97, in main train(cfg) File "/root/anaconda3/envs/mask3d/lib/python3.10/site-packages/hydra/main.py", line 27, in decorated_main return task_function(cfg_passthrough) File "/openseg_blob/code/rank3d/mask3d1105/Mask3D/main_instance_segmentation.py", line 78, in train runner.fit(model) File "/root/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 696, in fit self._call_and_handle_interrupt( File "/root/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 648, in _call_and_handle_interrupt return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs) File "/root/anaconda3/envs/mask3d/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/multiprocessing.py", line 103, in launch mp.start_processes( File "/root/anaconda3/envs/mask3d/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 189, in start_processes process.start() File "/root/anaconda3/envs/mask3d/lib/python3.10/multiprocessing/process.py", line 121, in start self._popen = self._Popen(self) File "/root/anaconda3/envs/mask3d/lib/python3.10/multiprocessing/context.py", line 288, in _Popen return Popen(process_obj) File "/root/anaconda3/envs/mask3d/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in init super().init(process_obj) File "/root/anaconda3/envs/mask3d/lib/python3.10/multiprocessing/popen_fork.py", line 19, in init self._launch(process_obj) File "/root/anaconda3/envs/mask3d/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch reduction.dump(process_obj, fp) File "/root/anaconda3/envs/mask3d/lib/python3.10/multiprocessing/reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) TypeError: cannot pickle 'MinkowskiConvolutionFunction' object

JonasSchult commented 1 year ago

Hi! Right now, the codebase does not support multi-gpu training. It sounds like a great extension! :)

Best, Jonas