shanice-l / gdrnpp_bop2022

PyTorch Implementation of GDRNPP, winner (most of the awards) of the BOP Challenge 2022 at ECCV'22
Apache License 2.0
241 stars 51 forks source link

File "/home/mona/anaconda3/envs/gdrnpp/lib/python3.10/site-packages/numba/np/ufunc/decorators.py", line 3, in <module> from numba.np.ufunc import _internal SystemError: initialization of _internal failed without raising an exception #91

Closed monajalal closed 10 months ago

monajalal commented 10 months ago

Can you please guide how to fix this error?

(gdrnpp) mona@ada:~/gdrnpp_bop2022$ conda install numba
Channels:
 - defaults
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/mona/anaconda3/envs/gdrnpp

  added / updated specs:
    - numba

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    blas-devel-3.9.0           |   12_linux64_mkl          11 KB  conda-forge
    brotli-python-1.0.9        |  py310h6a678d5_7         907 KB
    certifi-2023.11.17         |  py310h06a4308_0         158 KB
    cffi-1.16.0                |  py310h5eee18b_0         253 KB
    cryptography-41.0.7        |  py310hdda0065_0         2.0 MB
    freetype-2.11.0            |       h70c0345_0         618 KB
    libblas-3.9.0              |   12_linux64_mkl          12 KB  conda-forge
    libcblas-3.9.0             |   12_linux64_mkl          12 KB  conda-forge
    libgfortran-3.0.0          |                1         281 KB  conda-forge
    liblapack-3.9.0            |   12_linux64_mkl          12 KB  conda-forge
    liblapacke-3.9.0           |   12_linux64_mkl          12 KB  conda-forge
    libllvm11-11.1.0           |       h9e868ea_6        24.1 MB
    libnsl-2.0.0               |       h5eee18b_0          31 KB
    libtiff-4.2.0              |       h85742a9_0         502 KB
    libuuid-2.32.1             |    h7f98852_1000          28 KB  conda-forge
    libzlib-1.2.11             |    h166bdaf_1014          60 KB  conda-forge
    llvm-openmp-14.0.4         |       he0ac6c6_0         5.8 MB  conda-forge
    llvmlite-0.39.1            |  py310he621ea3_0         2.0 MB
    mkl-devel-2021.4.0         |     h66538d2_640          27 KB
    mkl-include-2021.4.0       |     h06a4308_640         535 KB
    numba-0.56.4               |  py310h1128e8f_0         4.1 MB
    numpy-1.23.5               |  py310hd5efca6_0          10 KB
    numpy-base-1.23.5          |  py310h8e6c178_0         6.7 MB
    openblas-0.3.4             |       ha44fe06_0        13.7 MB  conda-forge
    pillow-9.0.1               |  py310h22f2fdc_0         1.1 MB
    platformdirs-3.10.0        |  py310h06a4308_0          33 KB
    pooch-1.7.0                |  py310h06a4308_0          82 KB
    python-3.10.4              |h2660328_0_cpython        28.6 MB  conda-forge
    scipy-1.10.1               |  py310hd5efca6_0        23.9 MB
    sqlite-3.38.2              |       hc218d9a_0         1.0 MB
    tk-8.6.12                  |       h27826a3_0         3.3 MB  conda-forge
    urllib3-1.26.18            |  py310h06a4308_0         198 KB
    zlib-1.2.11                |    h166bdaf_1014          88 KB  conda-forge
    zstd-1.4.9                 |       haebb681_0         480 KB
    ------------------------------------------------------------
                                           Total:       120.6 MB

The following NEW packages will be INSTALLED:

  brotli-python      pkgs/main/linux-64::brotli-python-1.0.9-py310h6a678d5_7 
  certifi            pkgs/main/linux-64::certifi-2023.11.17-py310h06a4308_0 
  cffi               pkgs/main/linux-64::cffi-1.16.0-py310h5eee18b_0 
  charset-normalizer pkgs/main/noarch::charset-normalizer-2.0.4-pyhd3eb1b0_0 
  cryptography       pkgs/main/linux-64::cryptography-41.0.7-py310hdda0065_0 
  idna               pkgs/main/linux-64::idna-3.4-py310h06a4308_0 
  libgfortran        conda-forge/linux-64::libgfortran-3.0.0-1 
  libllvm11          pkgs/main/linux-64::libllvm11-11.1.0-h9e868ea_6 
  libnsl             pkgs/main/linux-64::libnsl-2.0.0-h5eee18b_0 
  llvmlite           pkgs/main/linux-64::llvmlite-0.39.1-py310he621ea3_0 
  lz4-c              pkgs/main/linux-64::lz4-c-1.9.4-h6a678d5_0 
  mkl-devel          pkgs/main/linux-64::mkl-devel-2021.4.0-h66538d2_640 
  mkl-include        pkgs/main/linux-64::mkl-include-2021.4.0-h06a4308_640 
  numba              pkgs/main/linux-64::numba-0.56.4-py310h1128e8f_0 
  platformdirs       pkgs/main/linux-64::platformdirs-3.10.0-py310h06a4308_0 
  pooch              pkgs/main/linux-64::pooch-1.7.0-py310h06a4308_0 
  pycparser          pkgs/main/noarch::pycparser-2.21-pyhd3eb1b0_0 
  pyopenssl          pkgs/main/linux-64::pyopenssl-23.2.0-py310h06a4308_0 
  pysocks            pkgs/main/linux-64::pysocks-1.7.1-py310h06a4308_0 
  requests           pkgs/main/linux-64::requests-2.31.0-py310h06a4308_0 
  urllib3            pkgs/main/linux-64::urllib3-1.26.18-py310h06a4308_0 

The following packages will be UPDATED:

  _openmp_mutex      conda-forge::_openmp_mutex-4.5-2_kmp_~ --> pkgs/main::_openmp_mutex-5.1-1_gnu 
  libuuid              pkgs/main::libuuid-1.41.5-h5eee18b_0 --> conda-forge::libuuid-2.32.1-h7f98852_1000 

The following packages will be SUPERSEDED by a higher-priority channel:

  _libgcc_mutex      conda-forge::_libgcc_mutex-0.1-conda_~ --> pkgs/main::_libgcc_mutex-0.1-main 
  blas                     conda-forge::blas-2.120-openblas --> pkgs/main::blas-1.0-mkl 
  libgcc-ng          conda-forge::libgcc-ng-13.2.0-h807b86~ --> pkgs/main::libgcc-ng-11.2.0-h1234567_1 
  libgfortran-ng     conda-forge::libgfortran-ng-13.2.0-h6~ --> pkgs/main::libgfortran-ng-11.2.0-h00389a5_1 
  libgfortran5       conda-forge::libgfortran5-13.2.0-ha46~ --> pkgs/main::libgfortran5-11.2.0-h1234567_1 
  libgomp            conda-forge::libgomp-13.2.0-h807b86a_3 --> pkgs/main::libgomp-11.2.0-h1234567_1 
  libopenblas        conda-forge::libopenblas-0.3.25-pthre~ --> pkgs/main::libopenblas-0.3.21-h043d6bf_0 
  mkl                conda-forge::mkl-2022.2.1-h84fe81f_16~ --> pkgs/main::mkl-2021.4.0-h06a4308_640 
  mkl-service        conda-forge::mkl-service-2.4.0-py310h~ --> pkgs/main::mkl-service-2.4.0-py310h7f8727e_0 
  mkl_fft            conda-forge::mkl_fft-1.3.8-py310h88d2~ --> pkgs/main::mkl_fft-1.3.1-py310hd6ae3a3_0 
  mkl_random         conda-forge::mkl_random-1.2.4-py310h3~ --> pkgs/main::mkl_random-1.2.2-py310h00e6091_0 
  openssl             conda-forge::openssl-3.2.0-hd590300_1 --> pkgs/main::openssl-3.0.12-h7f8727e_0 
  python               pkgs/main::python-3.10.13-h955ad1f_0 --> conda-forge::python-3.10.4-h2660328_0_cpython 
  tk                        pkgs/main::tk-8.6.12-h1ccaba5_0 --> conda-forge::tk-8.6.12-h27826a3_0 
  zstd                   conda-forge::zstd-1.5.5-hfc55251_0 --> pkgs/main::zstd-1.4.9-haebb681_0 

The following packages will be DOWNGRADED:

  blas-devel                      3.9.0-20_linux64_openblas --> 3.9.0-12_linux64_mkl 
  freetype                                2.12.1-h4a9f257_0 --> 2.11.0-h70c0345_0 
  intel-openmp                      2023.1.0-hdb19cb5_46306 --> 2021.4.0-h06a4308_3561 
  libblas                         3.9.0-20_linux64_openblas --> 3.9.0-12_linux64_mkl 
  libcblas                        3.9.0-20_linux64_openblas --> 3.9.0-12_linux64_mkl 
  liblapack                       3.9.0-20_linux64_openblas --> 3.9.0-12_linux64_mkl 
  liblapacke                      3.9.0-20_linux64_openblas --> 3.9.0-12_linux64_mkl 
  libpng                                  1.6.39-h5eee18b_0 --> 1.6.37-hbc83047_0 
  libtiff                                  4.5.1-h6a678d5_0 --> 4.2.0-h85742a9_0 
  libwebp                                  1.3.2-h11a3e52_0 --> 1.2.4-h11a3e52_1 
  libwebp-base                             1.3.2-h5eee18b_0 --> 1.2.4-h5eee18b_1 
  libzlib                                 1.2.13-hd590300_5 --> 1.2.11-h166bdaf_1014 
  llvm-openmp                             17.0.6-h4dfa4b3_0 --> 14.0.4-he0ac6c6_0 
  numpy                              1.26.2-py310heeff2f4_0 --> 1.23.5-py310hd5efca6_0 
  numpy-base                         1.26.2-py310h8a23956_0 --> 1.23.5-py310h8e6c178_0 
  openblas                       0.3.25-pthreads_h7a3da1a_0 --> 0.3.4-ha44fe06_0 
  pillow                             10.0.1-py310ha6cbd5a_0 --> 9.0.1-py310h22f2fdc_0 
  scipy                              1.11.4-py310heeff2f4_0 --> 1.10.1-py310hd5efca6_0 
  sqlite                                  3.41.2-h5eee18b_0 --> 3.38.2-hc218d9a_0 
  zlib                                    1.2.13-hd590300_5 --> 1.2.11-h166bdaf_1014 

Proceed ([y]/n)? y

Downloading and Extracting Packages:

Preparing transaction: done                                                                                                                 
Verifying transaction: done                                                                                                                 
Executing transaction: done                                                                                                                 
(gdrnpp) mona@ada:~/gdrnpp_bop2022$ ./core/gdrn_modeling/train_gdrn.sh configs/gdrn/ycbv/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_ycbv.py 0                                                                                                                  
++ dirname ./core/gdrn_modeling/train_gdrn.sh                                                                                               
+ this_dir=./core/gdrn_modeling                                                                                                             
+ CFG=configs/gdrn/ycbv/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_ycbv.py                                         
+ CUDA_VISIBLE_DEVICES=0                                                                                                                    
+ IFS=,                                                                                                                                     
+ read -ra GPUS                                                                                                                             
+ NGPU=1                                                                                                                                    
+ echo 'use gpu ids: 0 num gpus: 1'                                                                                                         
use gpu ids: 0 num gpus: 1                                                                                                                  
+ NCCL_DEBUG=INFO                                                                                                                           
+ OMP_NUM_THREADS=1                                                                                                                         
+ MKL_NUM_THREADS=1                                                                                                                         
+ PYTHONPATH=./core/gdrn_modeling/../..:/home/mona/realsense-ros/install/realsense2_camera_msgs/local/lib/python3.10/dist-packages:/opt/ros/humble/lib/python3.10/site-packages:/opt/ros/humble/local/lib/python3.10/dist-packages                                                      
+ CUDA_VISIBLE_DEVICES=0                                                                                                                    
+ python ./core/gdrn_modeling/main_gdrn.py --config-file configs/gdrn/ycbv/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_ycbv.py --num-gpus 1                                                                                                                     
/home/mona/.local/lib/python3.10/site-packages/mmcv/__init__.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.        
  warnings.warn(                                                                                                                            
You requested to import horovod which is missing or not supported for your OS.                                                              
/home/mona/.local/lib/python3.10/site-packages/mmcv/device/npu/data_parallel.py:22: UserWarning: Torchaudio's I/O functions now support par-call bakcend dispatch. Importing backend implementation directly is no longer guaranteed to work. Please use `backend` keyword with load/save/info function, instead of calling the udnerlying implementation directly.                                                                 
  if hasattr(sys.modules[m], '_check_balance'):                                                                                             
Traceback (most recent call last):                                                                                                          
  File "/home/mona/gdrnpp_bop2022/./core/gdrn_modeling/main_gdrn.py", line 38, in <module>                                                  
    from core.gdrn_modeling.datasets.dataset_factory import register_datasets_in_cfg
  File "/home/mona/gdrnpp_bop2022/core/gdrn_modeling/../../core/gdrn_modeling/datasets/dataset_factory.py", line 10, in <module>
    from core.gdrn_modeling.datasets import (
  File "/home/mona/gdrnpp_bop2022/core/gdrn_modeling/../../core/gdrn_modeling/datasets/lm_pbr.py", line 20, in <module>
    from lib.pysixd import inout, misc
  File "/home/mona/gdrnpp_bop2022/core/gdrn_modeling/../../lib/pysixd/misc.py", line 13, in <module>
    from numba import jit, njit
  File "/home/mona/anaconda3/envs/gdrnpp/lib/python3.10/site-packages/numba/__init__.py", line 42, in <module>
    from numba.np.ufunc import (vectorize, guvectorize, threading_layer,
  File "/home/mona/anaconda3/envs/gdrnpp/lib/python3.10/site-packages/numba/np/ufunc/__init__.py", line 3, in <module>
    from numba.np.ufunc.decorators import Vectorize, GUVectorize, vectorize, guvectorize
  File "/home/mona/anaconda3/envs/gdrnpp/lib/python3.10/site-packages/numba/np/ufunc/decorators.py", line 3, in <module>
    from numba.np.ufunc import _internal
SystemError: initialization of _internal failed without raising an exception
monajalal commented 10 months ago

fixed

(gdrnpp) mona@ada:~/gdrnpp_bop2022$ pip install -U numba
Defaulting to user installation because normal site-packages is not writeable
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting numba
  Downloading numba-0.58.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (2.7 kB)
Collecting llvmlite<0.42,>=0.41.0dev0 (from numba)
  Downloading llvmlite-0.41.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.8 kB)
Requirement already satisfied: numpy<1.27,>=1.22 in /home/mona/.local/lib/python3.10/site-packages (from numba) (1.26.2)
Downloading numba-0.58.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 16.6 MB/s eta 0:00:00
Downloading llvmlite-0.41.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (43.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.6/43.6 MB 28.0 MB/s eta 0:00:00
DEPRECATION: pytorch-lightning 1.6.0 has a non-standard dependency specifier torch>=1.8.*. pip 24.0 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of pytorch-lightning or contact the author to suggest that they release a version with a conforming dependency specifiers. Discussion can be found at https://github.com/pypa/pip/issues/12063
Installing collected packages: llvmlite, numba
Successfully installed llvmlite-0.41.1 numba-0.58.1
(gdrnpp) mona@ada:~/gdrnpp_bop2022$ ./core/gdrn_modeling/train_gdrn.sh configs/gdrn/ycbv/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_ycbv.py 0
++ dirname ./core/gdrn_modeling/train_gdrn.sh
+ this_dir=./core/gdrn_modeling
+ CFG=configs/gdrn/ycbv/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_ycbv.py
+ CUDA_VISIBLE_DEVICES=0
+ IFS=,
+ read -ra GPUS
+ NGPU=1
+ echo 'use gpu ids: 0 num gpus: 1'
use gpu ids: 0 num gpus: 1
+ NCCL_DEBUG=INFO
+ OMP_NUM_THREADS=1
+ MKL_NUM_THREADS=1
+ PYTHONPATH=./core/gdrn_modeling/../..:/home/mona/realsense-ros/install/realsense2_camera_msgs/local/lib/python3.10/dist-packages:/opt/ros/humble/lib/python3.10/site-packages:/opt/ros/humble/local/lib/python3.10/dist-packages
+ CUDA_VISIBLE_DEVICES=0
+ python ./core/gdrn_modeling/main_gdrn.py --config-file configs/gdrn/ycbv/convnext_a6_AugCosyAAEGray_BG05_mlL1_DMask_amodalClipBox_classAware_ycbv.py --num-gpus 1
/home/mona/.local/lib/python3.10/site-packages/mmcv/__init__.py:20: UserWarning: On January 1, 2023, MMCV will release v2.0.0, in which it will remove components related to the training process and add a data transformation module. In addition, it will rename the package names mmcv to mmcv-lite and mmcv-full to mmcv. See https://github.com/open-mmlab/mmcv/blob/master/docs/en/compatibility.md for more details.
  warnings.warn(
You requested to import horovod which is missing or not supported for your OS.
/home/mona/.local/lib/python3.10/site-packages/mmcv/device/npu/data_parallel.py:22: UserWarning: Torchaudio's I/O functions now support par-call bakcend dispatch. Importing backend implementation directly is no longer guaranteed to work. Please use `backend` keyword with load/save/info function, instead of calling the udnerlying implementation directly.
  if hasattr(sys.modules[m], '_check_balance'):
/home/mona/gdrnpp_bop2022/core/gdrn_modeling/../../lib/pysixd/misc.py:586: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  def get_obj_im_c(K, t):
/home/mona/gdrnpp_bop2022/core/gdrn_modeling/../../lib/pysixd/misc.py:765: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  def compute_2d_bbox_xyxy_from_pose(points, pose, K, width=640, height=480, clip=False):
/home/mona/gdrnpp_bop2022/core/gdrn_modeling/../../lib/pysixd/misc.py:793: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  def compute_2d_bbox_xyxy_from_pose_v2(points, pose, K, width=640, height=480, clip=False):
/home/mona/gdrnpp_bop2022/core/gdrn_modeling/../../lib/pysixd/misc.py:822: NumbaDeprecationWarning: The 'nopython' keyword argument was not supplied to the 'numba.jit' decorator. The implicit default value for this argument is currently False, but it will be changed to True in Numba 0.59.0. See https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-object-mode-fall-back-behaviour-when-using-jit for details.
  def compute_2d_bbox_xywh_from_pose(points, pose, K, width=640, height=480, clip=False):
Traceback (most recent call last):
  File "/home/mona/gdrnpp_bop2022/./core/gdrn_modeling/main_gdrn.py", line 39, in <module>
    from core.gdrn_modeling.engine.engine_utils import get_renderer
  File "/home/mona/gdrnpp_bop2022/core/gdrn_modeling/../../core/gdrn_modeling/engine/engine_utils.py", line 6, in <module>
    from einops import rearrange
ModuleNotFoundError: No module named 'einops'