search

numba / numba

NumPy aware dynamic Python compiler using LLVM
https://numba.pydata.org/
BSD 2-Clause "Simplified" License
9.82k stars 1.12k forks source link

Call to cuMemFree results in CUDA_ERROR_LAUNCH_TIMEOUT #8006

Closed cainiaoshidai closed 2 years ago

cainiaoshidai commented 2 years ago

I encountered the following problem: 26/04/2022 11:02:31 Call to cuMemcpyDtoH results in CUDA_ERROR_LAUNCH_TIMEOUT Traceback (most recent call last): File "/media/gzm/1430A2F330A2DACE1/graph_color/DLMCOL_WVCP/main.py", line 74, in name_expe, File "/media/gzm/1430A2F330A2DACE1/graph_color/DLMCOL_WVCP/dlmcol/main_WVCP.py", line 336, in main_WVCP offsprings_pop = offsprings_pop_gpu_memory.copy_to_host() File "/home/gzm/anaconda3/envs/py3/lib/python3.7/site-packages/numba/cuda/cudadrv/devices.py", line 224, in _require_cuda_context return fn(*args, kws) File "/home/gzm/anaconda3/envs/py3/lib/python3.7/site-packages/numba/cuda/cudadrv/devicearray.py", line 262, in copy_to_host stream=stream) File "/home/gzm/anaconda3/envs/py3/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 2403, in device_to_host fn(host_pointer(dst), device_pointer(src), size, varargs) File "/home/gzm/anaconda3/envs/py3/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 300, in safe_cuda_api_call self._check_error(fname, retcode) File "/home/gzm/anaconda3/envs/py3/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 335, in _check_error raise CudaAPIError(retcode, msg) numba.cuda.cudadrv.driver.CudaAPIError: [702] Call to cuMemcpyDtoH results in CUDA_ERROR_LAUNCH_TIMEOUT 26/04/2022 11:02:31 add pending dealloc: module_unload ? bytes 26/04/2022 11:02:31 add pending dealloc: cuMemFree_v2 81920 bytes 26/04/2022 11:02:31 add pending dealloc: module_unload ? bytes 26/04/2022 11:02:31 add pending dealloc: cuMemFree_v2 2000 bytes 26/04/2022 11:02:31 add pending dealloc: cuMemFree_v2 62500 bytes 26/04/2022 11:02:31 add pending dealloc: cuMemFree_v2 327680 bytes 26/04/2022 11:02:31 add pending dealloc: cuMemFree_v2 20480000 bytes 26/04/2022 11:02:31 add pending dealloc: cuMemFree_v2 81920 bytes 26/04/2022 11:02:31 add pending dealloc: cuMemFree_v2 81920 bytes 26/04/2022 11:02:31 add pending dealloc: cuMemFree_v2 838860800 bytes 26/04/2022 11:02:31 add pending dealloc: cuMemFree_v2 838860800 bytes 26/04/2022 11:02:31 add pending dealloc: cuMemFree_v2 81920 bytes 26/04/2022 11:02:31 add pending dealloc: cuMemFree_v2 20480000 bytes 26/04/2022 11:02:31 dealloc: cuMemFree_v2 81920 bytes 26/04/2022 11:02:31 Call to cuMemFree results in CUDA_ERROR_LAUNCH_TIMEOUT Traceback (most recent call last): File "/home/gzm/anaconda3/envs/py3/lib/python3.7/weakref.py", line 648, in _exitfunc f() File "/home/gzm/anaconda3/envs/py3/lib/python3.7/weakref.py", line 572, in call return info.func(info.args, (info.kwargs or {})) File "/home/gzm/anaconda3/envs/py3/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 1804, in deref mem.free() File "/home/gzm/anaconda3/envs/py3/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 1617, in free self._finalizer() File "/home/gzm/anaconda3/envs/py3/lib/python3.7/weakref.py", line 572, in call return info.func(*info.args, **(info.kwargs or {})) File "/home/gzm/anaconda3/envs/py3/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 1285, in core deallocations.add_item(driver.cuMemFree, handle, size) File "/home/gzm/anaconda3/envs/py3/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 970, in add_item self.clear() File "/home/gzm/anaconda3/envs/py3/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 981, in clear dtor(handle) File "/home/gzm/anaconda3/envs/py3/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 300, in safe_cuda_api_call self._check_error(fname, retcode) File "/home/gzm/anaconda3/envs/py3/lib/python3.7/site-packages/numba/cuda/cudadrv/driver.py", line 335, in _check_error raise CudaAPIError(retcode, msg) numba.cuda.cudadrv.driver.CudaAPIError: [702] Call to cuMemFree results in CUDA_ERROR_LAUNCH_TIMEOUT

But there is no problem when I run the following code:

import torch from numba import cuda import numpy as np size_pop = 20480 size = 500 offsprings_pop_after_tabu = np.zeros((size_pop, size), dtype=np.int32) offsprings_pop_gpu_memory = cuda.to_device(offsprings_pop_after_tabu) offsprings_pop = offsprings_pop_gpu_memory.copy_to_host()

gmarkall commented 2 years ago

This can happen if a kernel runs for longer than the timeout period (usually 5 seconds) and the watchdog is enabled - do you have a display attached to the GPU you're using?

cainiaoshidai commented 2 years ago

This can happen if a kernel runs for longer than the timeout period (usually 5 seconds) and the watchdog is enabled - do you have a display attached to the GPU you're using?

No, my GPU is Tesla k40, It can't connect to a display. Maybe the GPU is too old. So is there a way to get him to work？Can you give me some advice? Thanks. Best.

gmarkall commented 2 years ago

Can you provide the output of numba -s please?

cainiaoshidai commented 2 years ago

Hi, This is the output of numba -s. Thanks.

(py3) gzm@gzm:~$ numba -s System info:

Time Stamp Report started (local time) : 2022-04-27 20:25:48.736661 UTC start time : 2022-04-27 12:25:48.736675 Running time (s) : 7.751219

Hardware Information Machine : x86_64 CPU Name : ivybridge CPU Count : 8 Number of accessible CPUs : 8 List of accessible CPUs cores : 0-7 CFS Restrictions (CPUs worth of runtime) : None

CPU Features : 64bit aes avx cmov cx16 cx8 f16c fsgsbase fxsr mmx pclmul popcnt rdrnd sahf sse sse2 sse3 sse4.1 sse4.2 ssse3 xsave xsaveopt

Memory Total (MB) : 23975 Memory Available (MB) : 21680

OS Information Platform Name : Linux-5.4.0-109-generic-x86_64-with-debian-buster-sid Platform Release : 5.4.0-109-generic OS Name : Linux OS Version : #123~18.04.1-Ubuntu SMP Fri Apr 8 09:48:52 UTC 2022 OS Specific Version : ? Libc Version : glibc 2.10

Python Information Python Compiler : GCC 7.3.0 Python Implementation : CPython Python Version : 3.7.6 Python Locale : zh_CN.UTF-8

LLVM Information LLVM Version : 10.0.1

CUDA Information CUDA Device Initialized : True CUDA Driver Version : 11030 CUDA Detect Output: Found 2 CUDA devices id 0 b'NVIDIA Tesla K40c' [SUPPORTED] compute capability: 3.5 pci device id: 0 pci bus id: 2 id 1 b'NVIDIA GeForce GT 720' [SUPPORTED] compute capability: 3.5 pci device id: 0 pci bus id: 3 Summary: 2/2 devices are supported

CUDA Libraries Test Output: Finding cublas from Conda environment named libcublas.so.10.2.2.89 trying to open library... ok Finding cusparse from Conda environment named libcusparse.so.10.3.1.89 trying to open library... ok Finding cufft from Conda environment named libcufft.so.10.1.2.89 trying to open library... ok Finding curand from Conda environment named libcurand.so.10.1.2.89 trying to open library... ok Finding nvvm from Conda environment named libnvvm.so.3.3.0 trying to open library... ok Finding cudart from Conda environment named libcudart.so.10.2.89 trying to open library... ok Finding cudadevrt from Conda environment named libcudadevrt.a Finding libdevice from Conda environment searching for compute_20... ok searching for compute_30... ok searching for compute_35... ok searching for compute_50... ok

ROC information ROC Available : False ROC Toolchains : None HSA Agents Count : 0 HSA Agents: None HSA Discrete GPUs Count : 0 HSA Discrete GPUs : None

SVML Information SVML State, config.USING_SVML : False SVML Library Loaded : False llvmlite Using SVML Patched LLVM : True SVML Operational : False

Threading Layer Information TBB Threading Layer Available : False +--> Disabled due to Unknown import problem. OpenMP Threading Layer Available : True +-->Vendor: GNU Workqueue Threading Layer Available : True +-->Workqueue imported successfully.

Numba Environment Variable Information None found.

Conda Information Conda Build : 3.20.5 Conda Env : 4.11.0 Conda Platform : linux-64 Conda Python Version : 3.8.5.final.0 Conda Root Writable : True

Installed Packages _libgcc_mutex 0.1 main https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main _openmp_mutex 4.5 1_gnu https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main absl-py 0.12.0 pypi_0 pypi addict 2.4.0 py37h89c1867_0 conda-forge aiohttp 3.7.4.post0 pypi_0 pypi alabaster 0.7.12 py37_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main alfred-py 2.9.2 pypi_0 pypi anyio 3.2.1 pypi_0 pypi appdirs 1.4.4 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main apptools 5.1.0 pypi_0 pypi argon2-cffi 20.1.0 pypi_0 pypi ase 3.21.1 pypi_0 pypi astroid 2.9.0 py37h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main async-generator 1.10 pypi_0 pypi async-timeout 3.0.1 pypi_0 pypi attrs 20.3.0 pypi_0 pypi autoflake 1.4 pypi_0 pypi babel 2.9.1 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main backcall 0.2.0 pypi_0 pypi black 19.10b0 py_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main blas 1.0 mkl https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main bleach 3.3.0 pypi_0 pypi boost-cpp 1.74.0 h9d3c048_1 conda-forge brotlipy 0.7.0 py37h27cfd23_1003 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main bzip2 1.0.8 h7b6447c_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main ca-certificates 2022.3.29 h06a4308_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main cached-property 1.5.2 pypi_0 pypi cachetools 4.2.2 pypi_0 pypi certifi 2021.10.8 py37h06a4308_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main cffi 1.14.0 py37h2e261b9_0 defaults chardet 4.0.0 py37h06a4308_1003 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main click 8.0.4 py37h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main cmake 3.3.1 0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free colorama 0.4.4 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main configobj 5.0.6 pypi_0 pypi cryptography 3.4.7 py37hd23ed53_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main cudatoolkit 10.2.89 hfd86e86_1 anaconda cudnn 7.6.5 cuda10.2_0 anaconda cycler 0.10.0 pypi_0 pypi cython 0.29.23 pypi_0 pypi dataclasses 0.8 pyh6d0b6a4_7 defaults debugpy 1.3.0 pypi_0 pypi decorator 4.4.2 pypi_0 pypi defusedxml 0.7.1 pypi_0 pypi deprecated 1.2.13 pypi_0 pypi deprecation 2.1.0 pypi_0 pypi distlib 0.3.4 pypi_0 pypi docutils 0.17.1 py37h06a4308_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main entrypoints 0.3 pypi_0 pypi envisage 6.0.1 pypi_0 pypi expat 2.4.1 h2531618_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main filelock 3.6.0 pypi_0 pypi flake8 3.9.2 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main flann 1.9.1 h2e58136_1008 conda-forge freetype 2.10.4 h5ab3b9f_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main fsspec 2021.4.0 pypi_0 pypi funcy 1.16 pypi_0 pypi future 0.18.2 py37_1 defaults fvcore 0.1.5.post20211023 pypi_0 pypi git 2.23.0 pl526hacde149_0 defaults google-auth 1.30.0 pypi_0 pypi google-auth-oauthlib 0.4.4 pypi_0 pypi googledrivedownloader 0.4 pypi_0 pypi grpcio 1.37.1 pypi_0 pypi h5py 3.2.1 pypi_0 pypi hdf5 1.10.6 hb1b8bf9_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main icu 68.1 h2531618_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main idna 2.10 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main imagesize 1.3.0 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main importlib-metadata 3.10.1 pypi_0 pypi importlib-resources 5.4.0 pypi_0 pypi intel-openmp 2021.2.0 h06a4308_610 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main iopath 0.1.9 pypi_0 pypi ipykernel 6.0.1 pypi_0 pypi ipython 7.25.0 pypi_0 pypi ipython-genutils 0.2.0 pypi_0 pypi ipywidgets 7.6.3 pypi_0 pypi isodate 0.6.0 pypi_0 pypi isort 5.9.3 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main jedi 0.18.0 pypi_0 pypi jinja2 2.11.3 pypi_0 pypi joblib 1.0.1 pypi_0 pypi jpeg 9b h024ee3a_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main json5 0.9.6 pypi_0 pypi jsonpatch 1.32 pyhd3eb1b0_0 defaults jsonpath 0.82 pypi_0 pypi jsonpointer 2.1 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main jsons 1.6.0 pypi_0 pypi jsonschema 3.2.0 pypi_0 pypi jupyter-client 6.1.12 pypi_0 pypi jupyter-console 6.4.0 pypi_0 pypi jupyter-core 4.7.1 pypi_0 pypi jupyter-packaging 0.10.4 pypi_0 pypi jupyter-server 1.9.0 pypi_0 pypi jupyterlab 3.0.16 pypi_0 pypi jupyterlab-pygments 0.1.2 pypi_0 pypi jupyterlab-server 2.6.1 pypi_0 pypi jupyterlab-widgets 1.0.0 pypi_0 pypi kiwisolver 1.3.1 pypi_0 pypi knn-cuda 0.2 pypi_0 pypi krb5 1.18.2 h173b8e3_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main lazy-object-proxy 1.6.0 py37h27cfd23_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main lcms2 2.12 h3be6417_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main ld_impl_linux-64 2.35.1 h7274673_9 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libcurl 7.71.1 h20c2e04_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libedit 3.1.20210216 h27cfd23_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libffi 3.2.1 hf484d3e_1007 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libgcc-ng 9.3.0 h5101ec6_17 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libgfortran-ng 7.5.0 ha8ba4b0_17 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libgfortran4 7.5.0 ha8ba4b0_17 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libgomp 9.3.0 h5101ec6_17 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libpng 1.6.37 hbc83047_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libsodium 1.0.18 h7b6447c_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libssh2 1.9.0 h1ba5d50_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libstdcxx-ng 9.3.0 hd4cf53a_17 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libtiff 4.2.0 h85742a9_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libuv 1.40.0 h7b6447c_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libwebp-base 1.2.0 h27cfd23_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main llvmlite 0.36.0 pypi_0 pypi loguru 0.5.3 pypi_0 pypi lxml 4.6.4 pypi_0 pypi lz4-c 1.9.3 h2531618_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main magma-cuda102 2.5.2 1 pytorch markdown 3.3.4 pypi_0 pypi markupsafe 1.1.1 pypi_0 pypi matplotlib 3.4.1 pypi_0 pypi matplotlib-inline 0.1.2 pypi_0 pypi mccabe 0.6.1 py37_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mistune 0.8.4 pypi_0 pypi mkl 2021.2.0 h06a4308_296 defaults mkl-include 2021.2.0 h06a4308_296 defaults mkl-service 2.3.0 py37h27cfd23_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mkl_fft 1.3.0 py37h42c9631_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mkl_random 1.2.1 py37ha9443f7_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main multidict 5.1.0 pypi_0 pypi mypy_extensions 0.4.3 py37h06a4308_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main nbclassic 0.3.1 pypi_0 pypi nbclient 0.5.3 pypi_0 pypi nbconvert 6.1.0 pypi_0 pypi nbformat 5.1.3 pypi_0 pypi ncurses 6.2 he6710b0_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main nest-asyncio 1.5.1 pypi_0 pypi networkx 2.5.1 pypi_0 pypi ninja 1.10.2 hff7bd54_1 defaults nose 1.3.7 pypi_0 pypi notebook 6.4.0 pypi_0 pypi numba 0.53.1 pypi_0 pypi numexpr 2.7.3 pypi_0 pypi numpy 1.20.2 pypi_0 pypi numpy-base 1.20.1 py37h7d8b39e_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main oauthlib 3.1.0 pypi_0 pypi olefile 0.46 py37_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main open3d 0.13.0 pypi_0 pypi opencv-python 4.5.1.48 pypi_0 pypi openjpeg 2.3.0 h05c96fa_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main openmesh 1.1.6 pypi_0 pypi openssl 1.1.1n h7f8727e_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main packaging 20.9 pypi_0 pypi pandas 1.2.4 pypi_0 pypi pandocfilters 1.4.3 pypi_0 pypi parso 0.8.2 pypi_0 pypi pascal-voc-writer 0.1.4 pypi_0 pypi pathspec 0.7.0 py_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pcl 1.11.1 h39b1736_3 conda-forge pcre 8.45 h295c915_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main perl 5.26.2 h14c3975_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pexpect 4.8.0 pypi_0 pypi pickleshare 0.7.5 pypi_0 pypi pillow 8.2.0 pypi_0 pypi pip 21.3.1 pypi_0 pypi platformdirs 2.4.0 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main plotly 5.4.0 pypi_0 pypi pluggy 1.0.0 pypi_0 pypi plyfile 0.7.4 pyhd8ed1ab_0 conda-forge pointnet2-ops 3.0.0 pypi_0 pypi portalocker 2.3.2 pypi_0 pypi pptk 0.1.0 pypi_0 pypi progress 1.5 py37h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main prometheus-client 0.11.0 pypi_0 pypi prompt-toolkit 3.0.19 pypi_0 pypi protobuf 3.15.8 pypi_0 pypi psbody-mesh 0.4 pypi_0 pypi ptyprocess 0.7.0 pypi_0 pypi py 1.11.0 pypi_0 pypi pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pycodestyle 2.7.0 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pycparser 2.20 py_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pyface 7.3.0 pypi_0 pypi pyflakes 2.3.1 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pygments 2.9.0 pypi_0 pypi pylint 2.12.2 py37h06a4308_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pyopengl 3.1.5 pypi_0 pypi pyopenssl 20.0.1 pyhd3eb1b0_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pyparsing 2.4.7 pypi_0 pypi pyquaternion 0.9.9 pypi_0 pypi pyrsistent 0.18.0 pypi_0 pypi pysocks 1.7.1 py37_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main python 3.7.6 h0371630_2 defaults python-dateutil 2.8.1 pypi_0 pypi python-louvain 0.15 pypi_0 pypi python-pcl 0.3.0a1 pypi_0 pypi python_abi 3.7 1_cp37m conda-forge pytorch-lightning 1.2.10 pypi_0 pypi pytorch3d 0.3.0 pypi_0 pypi pytz 2021.1 pypi_0 pypi pyyaml 5.4.1 pypi_0 pypi pyzmq 22.0.3 pypi_0 pypi qhull 2020.2 h4bd325d_1 conda-forge qtconsole 5.2.1 pypi_0 pypi qtpy 1.11.3 pypi_0 pypi rdflib 5.0.0 pypi_0 pypi readline 7.0 h7b6447c_5 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main regex 2021.11.10 pypi_0 pypi requests 2.25.1 pyhd3eb1b0_0 defaults requests-oauthlib 1.3.0 pypi_0 pypi requests-unixsocket 0.2.0 pypi_0 pypi rhash 1.4.1 h3c74f83_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main rsa 4.7.2 pypi_0 pypi scikit-learn 0.24.1 pypi_0 pypi scipy 1.6.3 pypi_0 pypi send2trash 1.7.1 pypi_0 pypi setuptools 56.0.0 pypi_0 pypi six 1.15.0 py37h06a4308_0 defaults sklearn 0.0 pypi_0 pypi sniffio 1.2.0 pypi_0 pypi snowballstemmer 2.2.0 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sphinx 4.4.0 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sphinxcontrib-applehelp 1.0.2 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sphinxcontrib-devhelp 1.0.2 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sphinxcontrib-htmlhelp 2.0.0 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sphinxcontrib-jsmath 1.0.1 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sphinxcontrib-qthelp 1.0.3 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sphinxcontrib-serializinghtml 1.1.5 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sqlite 3.33.0 h62c20be_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main tables 3.6.1 pypi_0 pypi tabulate 0.8.9 pypi_0 pypi tenacity 8.0.1 pypi_0 pypi tensorboard 2.4.1 pypi_0 pypi tensorboard-plugin-wit 1.8.0 pypi_0 pypi termcolor 1.1.0 py37h06a4308_1 defaults terminado 0.10.1 pypi_0 pypi testpath 0.5.0 pypi_0 pypi threadpoolctl 2.1.0 pypi_0 pypi tk 8.6.10 hbc83047_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main toml 0.10.2 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main tomlkit 0.7.2 pypi_0 pypi torch 1.8.0+gitunknown pypi_0 pypi torch-cluster 1.5.9 pypi_0 pypi torch-geometric 1.7.0 pypi_0 pypi torch-scatter 2.0.6 pypi_0 pypi torch-sparse 0.6.9 pypi_0 pypi torchfile 0.1.0 py_0 conda-forge torchmetrics 0.2.0 pypi_0 pypi torchvision 0.10.0a0 pypi_0 pypi tornado 6.1 py37h27cfd23_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main tox 3.25.0 pypi_0 pypi tqdm 4.60.0 pypi_0 pypi traitlets 5.0.5 pypi_0 pypi traits 6.3.2 pypi_0 pypi traitsui 7.2.1 pypi_0 pypi trimesh 3.9.35 pyh6c4a22f_0 conda-forge typed-ast 1.4.3 py37h7f8727e_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main typing-extensions 4.1.1 hd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main typing_extensions 4.1.1 pyh06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main typish 1.9.3 pypi_0 pypi urllib3 1.26.6 pyhd3eb1b0_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main virtualenv 20.14.1 pypi_0 pypi visdom 0.1.8.9 0 conda-forge vtk 9.1.0 pypi_0 pypi wcwidth 0.2.5 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi websocket-client 0.58.0 py37h06a4308_4 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main werkzeug 1.0.1 pypi_0 pypi wheel 0.36.2 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main widgetsnbextension 3.5.1 pypi_0 pypi wrapt 1.13.3 py37h7f8727e_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main wslink 1.3.0 pypi_0 pypi xz 5.2.5 h7b6447c_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main yacs 0.1.8 pypi_0 pypi yarl 1.6.3 pypi_0 pypi zeromq 4.3.4 h2531618_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main zipp 3.4.1 pypi_0 pypi zlib 1.2.11 h7b6447c_3 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main zstd 1.4.9 haebb681_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main

No errors reported.

Warning log Warning (roc): Error initialising ROC: No ROC toolchains found. Warning (roc): No HSA Agents found, encountered exception when searching: Error at driver init: NUMBA_HSA_DRIVER /opt/rocm/lib/libhsa-runtime64.so is not a valid file path. Note it must be a filepath of the .so/.dll/.dylib or the driver: Warning (psutil): psutil cannot be imported. For more accuracy, consider installing it.

If requested, please copy and paste the information between the dashed (----) lines, or from a given specific section as appropriate.

============================================================= IMPORTANT: Please ensure that you are happy with sharing the contents of the information present, any information that you wish to keep private you should remove before sharing.

gmarkall commented 2 years ago

Thanks - the Numba version you have (0.53.1) is a couple of releases old - could you upgrade to Numba 0.55.1 and provide the output of numba -s again please?

cainiaoshidai commented 2 years ago

Thanks, this is the output of numba -s of Numba 0.55.1. System info:

Time Stamp Report started (local time) : 2022-04-28 10:17:55.270429 UTC start time : 2022-04-28 02:17:55.270442 Running time (s) : 7.635697

Hardware Information Machine : x86_64 CPU Name : ivybridge CPU Count : 8 Number of accessible CPUs : 8 List of accessible CPUs cores : 0-7 CFS Restrictions (CPUs worth of runtime) : None

CPU Features : 64bit aes avx cmov cx16 cx8 f16c fsgsbase fxsr mmx pclmul popcnt rdrnd sahf sse sse2 sse3 sse4.1 sse4.2 ssse3 xsave xsaveopt

Memory Total (MB) : 23975 Memory Available (MB) : 21705

OS Information Platform Name : Linux-5.4.0-109-generic-x86_64-with-debian-buster-sid Platform Release : 5.4.0-109-generic OS Name : Linux OS Version : #123~18.04.1-Ubuntu SMP Fri Apr 8 09:48:52 UTC 2022 OS Specific Version : ? Libc Version : glibc 2.10

Python Information Python Compiler : GCC 7.3.0 Python Implementation : CPython Python Version : 3.7.6 Python Locale : zh_CN.UTF-8

Numba Toolchain Versions Numba Version : 0.55.1 llvmlite Version : 0.38.0

LLVM Information LLVM Version : 11.1.0

CUDA Information CUDA Device Initialized : True CUDA Driver Version : (11, 3) CUDA Runtime Version : 10020 CUDA NVIDIA Bindings Available : False CUDA NVIDIA Bindings In Use : False CUDA Detect Output: Found 2 CUDA devices id 0 b'NVIDIA Tesla K40c' [SUPPORTED (DEPRECATED)] Compute Capability: 3.5 PCI Device ID: 0 PCI Bus ID: 2 UUID: GPU-ce13b711-1683-6ee6-b2ad-a5f181516a86 Watchdog: Enabled FP32/FP64 Performance Ratio: 3 id 1 b'NVIDIA GeForce GT 720' [SUPPORTED (DEPRECATED)] Compute Capability: 3.5 PCI Device ID: 0 PCI Bus ID: 3 UUID: GPU-21132057-dab6-0b3a-9872-4420b19e45fd Watchdog: Enabled FP32/FP64 Performance Ratio: 24 Summary: 2/2 devices are supported

CUDA Libraries Test Output: Finding nvvm from Conda environment named libnvvm.so.3.3.0 trying to open library... ok Finding cudart from Conda environment named libcudart.so.10.2.89 trying to open library... ok Finding cudadevrt from Conda environment named libcudadevrt.a Finding libdevice from Conda environment searching for compute_20... ok searching for compute_30... ok searching for compute_35... ok searching for compute_50... ok

SVML Information SVML State, config.USING_SVML : False SVML Library Loaded : False llvmlite Using SVML Patched LLVM : True SVML Operational : False

Threading Layer Information TBB Threading Layer Available : True +-->TBB imported successfully. OpenMP Threading Layer Available : True +-->Vendor: GNU Workqueue Threading Layer Available : True +-->Workqueue imported successfully.

Numba Environment Variable Information None found.

Conda Information Conda Build : 3.20.5 Conda Env : 4.11.0 Conda Platform : linux-64 Conda Python Version : 3.8.5.final.0 Conda Root Writable : True

Installed Packages _libgcc_mutex 0.1 main https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main _openmp_mutex 4.5 1_gnu https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main absl-py 0.12.0 pypi_0 pypi addict 2.4.0 py37h89c1867_0 conda-forge aiohttp 3.7.4.post0 pypi_0 pypi alabaster 0.7.12 py37_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main alfred-py 2.9.2 pypi_0 pypi anyio 3.2.1 pypi_0 pypi appdirs 1.4.4 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main apptools 5.1.0 pypi_0 pypi argon2-cffi 20.1.0 pypi_0 pypi ase 3.21.1 pypi_0 pypi astroid 2.9.0 py37h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main async-generator 1.10 pypi_0 pypi async-timeout 3.0.1 pypi_0 pypi attrs 20.3.0 pypi_0 pypi autoflake 1.4 pypi_0 pypi babel 2.9.1 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main backcall 0.2.0 pypi_0 pypi black 19.10b0 py_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main blas 1.0 mkl https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main bleach 3.3.0 pypi_0 pypi boost-cpp 1.74.0 h9d3c048_1 conda-forge brotlipy 0.7.0 py37h27cfd23_1003 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main bzip2 1.0.8 h7b6447c_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main ca-certificates 2022.3.29 h06a4308_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main cached-property 1.5.2 pypi_0 pypi cachetools 4.2.2 pypi_0 pypi certifi 2021.10.8 py37h06a4308_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main cffi 1.14.0 py37h2e261b9_0 defaults chardet 4.0.0 py37h06a4308_1003 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main click 8.0.4 py37h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main cmake 3.3.1 0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free colorama 0.4.4 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main configobj 5.0.6 pypi_0 pypi cryptography 3.4.7 py37hd23ed53_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main cudatoolkit 10.2.89 hfd86e86_1 anaconda cudnn 7.6.5 cuda10.2_0 anaconda cycler 0.10.0 pypi_0 pypi cython 0.29.23 pypi_0 pypi dataclasses 0.8 pyh6d0b6a4_7 defaults debugpy 1.3.0 pypi_0 pypi decorator 4.4.2 pypi_0 pypi defusedxml 0.7.1 pypi_0 pypi deprecated 1.2.13 pypi_0 pypi deprecation 2.1.0 pypi_0 pypi distlib 0.3.4 pypi_0 pypi docutils 0.17.1 py37h06a4308_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main entrypoints 0.3 pypi_0 pypi envisage 6.0.1 pypi_0 pypi expat 2.4.1 h2531618_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main filelock 3.6.0 pypi_0 pypi flake8 3.9.2 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main flann 1.9.1 h2e58136_1008 conda-forge freetype 2.10.4 h5ab3b9f_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main fsspec 2021.4.0 pypi_0 pypi funcy 1.16 pypi_0 pypi future 0.18.2 py37_1 defaults fvcore 0.1.5.post20211023 pypi_0 pypi git 2.23.0 pl526hacde149_0 defaults google-auth 1.30.0 pypi_0 pypi google-auth-oauthlib 0.4.4 pypi_0 pypi googledrivedownloader 0.4 pypi_0 pypi grpcio 1.37.1 pypi_0 pypi h5py 3.2.1 pypi_0 pypi hdf5 1.10.6 hb1b8bf9_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main icu 68.1 h2531618_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main idna 2.10 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main imagesize 1.3.0 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main importlib-metadata 3.10.1 pypi_0 pypi importlib-resources 5.4.0 pypi_0 pypi intel-openmp 2021.2.0 h06a4308_610 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main iopath 0.1.9 pypi_0 pypi ipykernel 6.0.1 pypi_0 pypi ipython 7.25.0 pypi_0 pypi ipython-genutils 0.2.0 pypi_0 pypi ipywidgets 7.6.3 pypi_0 pypi isodate 0.6.0 pypi_0 pypi isort 5.9.3 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main jedi 0.18.0 pypi_0 pypi jinja2 2.11.3 pypi_0 pypi joblib 1.0.1 pypi_0 pypi jpeg 9b h024ee3a_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main json5 0.9.6 pypi_0 pypi jsonpatch 1.32 pyhd3eb1b0_0 defaults jsonpath 0.82 pypi_0 pypi jsonpointer 2.1 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main jsons 1.6.0 pypi_0 pypi jsonschema 3.2.0 pypi_0 pypi jupyter-client 6.1.12 pypi_0 pypi jupyter-console 6.4.0 pypi_0 pypi jupyter-core 4.7.1 pypi_0 pypi jupyter-packaging 0.10.4 pypi_0 pypi jupyter-server 1.9.0 pypi_0 pypi jupyterlab 3.0.16 pypi_0 pypi jupyterlab-pygments 0.1.2 pypi_0 pypi jupyterlab-server 2.6.1 pypi_0 pypi jupyterlab-widgets 1.0.0 pypi_0 pypi kiwisolver 1.3.1 pypi_0 pypi knn-cuda 0.2 pypi_0 pypi krb5 1.18.2 h173b8e3_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main lazy-object-proxy 1.6.0 py37h27cfd23_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main lcms2 2.12 h3be6417_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main ld_impl_linux-64 2.35.1 h7274673_9 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libcurl 7.71.1 h20c2e04_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libedit 3.1.20210216 h27cfd23_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libffi 3.2.1 hf484d3e_1007 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libgcc-ng 9.3.0 h5101ec6_17 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libgfortran-ng 7.5.0 ha8ba4b0_17 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libgfortran4 7.5.0 ha8ba4b0_17 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libgomp 9.3.0 h5101ec6_17 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libllvm11 11.1.0 h3826bc1_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libpng 1.6.37 hbc83047_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libsodium 1.0.18 h7b6447c_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libssh2 1.9.0 h1ba5d50_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libstdcxx-ng 9.3.0 hd4cf53a_17 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libtiff 4.2.0 h85742a9_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libuv 1.40.0 h7b6447c_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libwebp-base 1.2.0 h27cfd23_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main llvmlite 0.36.0 pypi_0 pypi loguru 0.5.3 pypi_0 pypi lxml 4.6.4 pypi_0 pypi lz4-c 1.9.3 h2531618_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main magma-cuda102 2.5.2 1 pytorch markdown 3.3.4 pypi_0 pypi markupsafe 1.1.1 pypi_0 pypi matplotlib 3.4.1 pypi_0 pypi matplotlib-inline 0.1.2 pypi_0 pypi mccabe 0.6.1 py37_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mistune 0.8.4 pypi_0 pypi mkl 2021.2.0 h06a4308_296 defaults mkl-include 2021.2.0 h06a4308_296 defaults mkl-service 2.3.0 py37h27cfd23_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mkl_fft 1.3.0 py37h42c9631_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mkl_random 1.2.1 py37ha9443f7_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main multidict 5.1.0 pypi_0 pypi mypy_extensions 0.4.3 py37h06a4308_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main nbclassic 0.3.1 pypi_0 pypi nbclient 0.5.3 pypi_0 pypi nbconvert 6.1.0 pypi_0 pypi nbformat 5.1.3 pypi_0 pypi ncurses 6.2 he6710b0_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main nest-asyncio 1.5.1 pypi_0 pypi networkx 2.5.1 pypi_0 pypi ninja 1.10.2 hff7bd54_1 defaults nose 1.3.7 pypi_0 pypi notebook 6.4.0 pypi_0 pypi numba 0.53.1 pypi_0 pypi numexpr 2.7.3 pypi_0 pypi numpy 1.20.2 pypi_0 pypi numpy-base 1.20.1 py37h7d8b39e_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main oauthlib 3.1.0 pypi_0 pypi olefile 0.46 py37_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main open3d 0.13.0 pypi_0 pypi opencv-python 4.5.1.48 pypi_0 pypi openjpeg 2.3.0 h05c96fa_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main openmesh 1.1.6 pypi_0 pypi openssl 1.1.1n h7f8727e_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main packaging 20.9 pypi_0 pypi pandas 1.2.4 pypi_0 pypi pandocfilters 1.4.3 pypi_0 pypi parso 0.8.2 pypi_0 pypi pascal-voc-writer 0.1.4 pypi_0 pypi pathspec 0.7.0 py_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pcl 1.11.1 h39b1736_3 conda-forge pcre 8.45 h295c915_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main perl 5.26.2 h14c3975_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pexpect 4.8.0 pypi_0 pypi pickleshare 0.7.5 pypi_0 pypi pillow 8.2.0 pypi_0 pypi pip 21.3.1 pypi_0 pypi platformdirs 2.4.0 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main plotly 5.4.0 pypi_0 pypi pluggy 1.0.0 pypi_0 pypi plyfile 0.7.4 pyhd8ed1ab_0 conda-forge pointnet2-ops 3.0.0 pypi_0 pypi portalocker 2.3.2 pypi_0 pypi pptk 0.1.0 pypi_0 pypi progress 1.5 py37h06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main prometheus-client 0.11.0 pypi_0 pypi prompt-toolkit 3.0.19 pypi_0 pypi protobuf 3.15.8 pypi_0 pypi psbody-mesh 0.4 pypi_0 pypi ptyprocess 0.7.0 pypi_0 pypi py 1.11.0 pypi_0 pypi pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pycodestyle 2.7.0 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pycparser 2.20 py_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pyface 7.3.0 pypi_0 pypi pyflakes 2.3.1 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pygments 2.9.0 pypi_0 pypi pylint 2.12.2 py37h06a4308_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pyopengl 3.1.5 pypi_0 pypi pyopenssl 20.0.1 pyhd3eb1b0_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pyparsing 2.4.7 pypi_0 pypi pyquaternion 0.9.9 pypi_0 pypi pyrsistent 0.18.0 pypi_0 pypi pysocks 1.7.1 py37_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main python 3.7.6 h0371630_2 defaults python-dateutil 2.8.1 pypi_0 pypi python-louvain 0.15 pypi_0 pypi python-pcl 0.3.0a1 pypi_0 pypi python_abi 3.7 1_cp37m conda-forge pytorch-lightning 1.2.10 pypi_0 pypi pytorch3d 0.3.0 pypi_0 pypi pytz 2021.1 pypi_0 pypi pyyaml 5.4.1 pypi_0 pypi pyzmq 22.0.3 pypi_0 pypi qhull 2020.2 h4bd325d_1 conda-forge qtconsole 5.2.1 pypi_0 pypi qtpy 1.11.3 pypi_0 pypi rdflib 5.0.0 pypi_0 pypi readline 7.0 h7b6447c_5 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main regex 2021.11.10 pypi_0 pypi requests 2.25.1 pyhd3eb1b0_0 defaults requests-oauthlib 1.3.0 pypi_0 pypi requests-unixsocket 0.2.0 pypi_0 pypi rhash 1.4.1 h3c74f83_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main rsa 4.7.2 pypi_0 pypi scikit-learn 0.24.1 pypi_0 pypi scipy 1.6.3 pypi_0 pypi send2trash 1.7.1 pypi_0 pypi setuptools 56.0.0 pypi_0 pypi six 1.15.0 py37h06a4308_0 defaults sklearn 0.0 pypi_0 pypi sniffio 1.2.0 pypi_0 pypi snowballstemmer 2.2.0 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sphinx 4.4.0 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sphinxcontrib-applehelp 1.0.2 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sphinxcontrib-devhelp 1.0.2 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sphinxcontrib-htmlhelp 2.0.0 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sphinxcontrib-jsmath 1.0.1 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sphinxcontrib-qthelp 1.0.3 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sphinxcontrib-serializinghtml 1.1.5 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sqlite 3.33.0 h62c20be_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main tables 3.6.1 pypi_0 pypi tabulate 0.8.9 pypi_0 pypi tbb 2021.5.0 hd09550d_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main tenacity 8.0.1 pypi_0 pypi tensorboard 2.4.1 pypi_0 pypi tensorboard-plugin-wit 1.8.0 pypi_0 pypi termcolor 1.1.0 py37h06a4308_1 defaults terminado 0.10.1 pypi_0 pypi testpath 0.5.0 pypi_0 pypi threadpoolctl 2.1.0 pypi_0 pypi tk 8.6.10 hbc83047_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main toml 0.10.2 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main tomlkit 0.7.2 pypi_0 pypi torch 1.8.0+gitunknown pypi_0 pypi torch-cluster 1.5.9 pypi_0 pypi torch-geometric 1.7.0 pypi_0 pypi torch-scatter 2.0.6 pypi_0 pypi torch-sparse 0.6.9 pypi_0 pypi torchfile 0.1.0 py_0 conda-forge torchmetrics 0.2.0 pypi_0 pypi torchvision 0.10.0a0 pypi_0 pypi tornado 6.1 py37h27cfd23_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main tox 3.25.0 pypi_0 pypi tqdm 4.60.0 pypi_0 pypi traitlets 5.0.5 pypi_0 pypi traits 6.3.2 pypi_0 pypi traitsui 7.2.1 pypi_0 pypi trimesh 3.9.35 pyh6c4a22f_0 conda-forge typed-ast 1.4.3 py37h7f8727e_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main typing-extensions 4.1.1 hd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main typing_extensions 4.1.1 pyh06a4308_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main typish 1.9.3 pypi_0 pypi urllib3 1.26.6 pyhd3eb1b0_1 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main virtualenv 20.14.1 pypi_0 pypi visdom 0.1.8.9 0 conda-forge vtk 9.1.0 pypi_0 pypi wcwidth 0.2.5 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi websocket-client 0.58.0 py37h06a4308_4 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main werkzeug 1.0.1 pypi_0 pypi wheel 0.36.2 pyhd3eb1b0_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main widgetsnbextension 3.5.1 pypi_0 pypi wrapt 1.13.3 py37h7f8727e_2 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main wslink 1.3.0 pypi_0 pypi xz 5.2.5 h7b6447c_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main yacs 0.1.8 pypi_0 pypi yarl 1.6.3 pypi_0 pypi zeromq 4.3.4 h2531618_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main zipp 3.4.1 pypi_0 pypi zlib 1.2.11 h7b6447c_3 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main zstd 1.4.9 haebb681_0 https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main

No errors reported.

Warning log Warning (psutil): psutil cannot be imported. For more accuracy, consider installing it.

If requested, please copy and paste the information between the dashed (----) lines, or from a given specific section as appropriate.

============================================================= IMPORTANT: Please ensure that you are happy with sharing the contents of the information present, any information that you wish to keep private you should remove before sharing.

gmarkall commented 2 years ago

Watchdog: Enabled

You have the watchdog enabled for both GPUs - you need to disable it for the K40. See e.g. https://nvidia.custhelp.com/app/answers/detail/a_id/3029/~/using-cuda-and-x

cainiaoshidai commented 2 years ago

Watchdog: Enabled

You have the watchdog enabled for both GPUs - you need to disable it for the K40. See e.g. https://nvidia.custhelp.com/app/answers/detail/a_id/3029/~/using-cuda-and-x

Thanks, I finally solved this error by turning off the GUI. Although I didn't find a way that turns off the watchdog of the k40. Best.