Open Joker9194 opened 1 year ago
The same issue happens to me when I try to use MinkowskiEngine for multi-GPU training work
You can try build the model in the ModelEmaV2 class directly, that works for me. Eg: forward the config into the Ema class and build the model in the "init" rather than using deepcopy :)
Describe the bug I installed it according to the installation steps. I used ema in my project, which used deepcopy code, and this error occurred when the project was running. “TypeError: cannot pickle 'MinkowskiConvolutionFunction' object”
Expected behavior The project is running normally
Desktop (please complete the following information):
python -c "import MinkowskiEngine as ME; ME.print_diagnostics()"
. Otherwise, paste the output of the following command.) ==========System========== Linux-5.4.0-84-generic-x86_64-with-glibc2.10 DISTRIB_ID=Ubuntu DISTRIB_RELEASE=18.04 DISTRIB_CODENAME=bionic DISTRIB_DESCRIPTION="Ubuntu 18.04.6 LTS" 3.8.16 | packaged by conda-forge | (default, Feb 1 2023, 16:01:55) [GCC 11.3.0] ==========Pytorch========== 1.10.1 torch.cuda.is_available(): True ==========NVIDIA-SMI========== /usr/bin/nvidia-smi Driver Version 510.60.02 CUDA Version 11.6 VBIOS Version 94.04.25.80.4D Image Version G001.0000.03.03 GSP Firmware Version N/A ==========NVCC========== /usr/local/cuda-11.3/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2021 NVIDIA Corporation Built on Mon_May__3_19:15:13_PDT_2021 Cuda compilation tools, release 11.3, V11.3.109 Build cuda_11.3.r11.3/compiler.29920130_0 ==========CC========== /usr/bin/c++ c++ (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.==========MinkowskiEngine========== 0.5.4 MinkowskiEngine compiled with CUDA Support: True NVCC version MinkowskiEngine is compiled: 11030 CUDART version MinkowskiEngine is compiled: 11030