search

NVIDIA / MinkowskiEngine

Minkowski Engine is an auto-diff neural network library for high-dimensional sparse tensors
https://nvidia.github.io/MinkowskiEngine
Other
2.43k stars 360 forks source link

Fail to install in Slurm #457

Closed Iven-Wu closed 2 years ago

Iven-Wu commented 2 years ago

Describe the bug

[12/21] /usr/local/cuda/11.1/bin/nvcc -I/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/torch/include -I/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/torch/include/TH -I/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda/11.1/include -I/scratch/users/yuefanw/Retrival/MinkowskiEngine/src -I/scratch/users/yuefanw/Retrival/MinkowskiEngine/src/3rdparty -I/home/yuefanw/scratch/miniconda3/envs/mink/include -I/home/yuefanw/scratch/miniconda3/envs/mink/include/python3.8 -c -c /scratch/users/yuefanw/Retrival/MinkowskiEngine/src/spmm.cu -o /scratch/users/yuefanw/Retrival/MinkowskiEngine/build/temp.linux-x86_64-3.8/scratch/users/yuefanw/Retrival/MinkowskiEngine/src/spmm.o -DCUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -DCUDA_NO_BFLOAT16_CONVERSIONS -DCUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' --expt-relaxed-constexpr --expt-extended-lambda -O3 -Xcompiler=-fno-gnu-unique -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 -ccbin c++ -std=c++14 FAILED: /scratch/users/yuefanw/Retrival/MinkowskiEngine/build/temp.linux-x86_64-3.8/scratch/users/yuefanw/Retrival/MinkowskiEngine/src/spmm.o /usr/local/cuda/11.1/bin/nvcc -I/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/torch/include -I/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/torch/include/TH -I/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/torch/include/THC -I/usr/local/cuda/11.1/include -I/scratch/users/yuefanw/Retrival/MinkowskiEngine/src -I/scratch/users/yuefanw/Retrival/MinkowskiEngine/src/3rdparty -I/home/yuefanw/scratch/miniconda3/envs/mink/include -I/home/yuefanw/scratch/miniconda3/envs/mink/include/python3.8 -c -c /scratch/users/yuefanw/Retrival/MinkowskiEngine/src/spmm.cu -o /scratch/users/yuefanw/Retrival/MinkowskiEngine/build/temp.linux-x86_64-3.8/scratch/users/yuefanw/Retrival/MinkowskiEngine/src/spmm.o -DCUDA_NO_HALF_OPERATORS -DCUDA_NO_HALF_CONVERSIONS -DCUDA_NO_BFLOAT16_CONVERSIONS -DCUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' --expt-relaxed-constexpr --expt-extended-lambda -O3 -Xcompiler=-fno-gnu-unique -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=compute_75 -gencode=arch=compute_75,code=sm_75 -ccbin c++ -std=c++14 /scratch/users/yuefanw/Retrival/MinkowskiEngine/src/spmm.cu(102): warning: variable "is_int64" was declared but never referenced

/scratch/users/yuefanw/Retrival/MinkowskiEngine/src/spmm.cu(364): warning: variable "is_int64" was declared but never referenced

/scratch/users/yuefanw/Retrival/MinkowskiEngine/src/spmm.cu(473): warning: variable "num_unique_keys" was set but never used detected during instantiation of "std::vector<at::Tensor, std::allocator> minkowski::coo_spmm_average(const at::Tensor &, const at::Tensor &, int64_t, int64_t, const at::Tensor &, int64_t) [with th_int_type=int32_t]" (601): here

/scratch/users/yuefanw/Retrival/MinkowskiEngine/src/spmm.cu(473): warning: variable "num_unique_keys" was set but never used detected during instantiation of "std::vector<at::Tensor, std::allocator> minkowski::coo_spmm_average(const at::Tensor &, const at::Tensor &, int64_t, int64_t, const at::Tensor &, int64_t) [with th_int_type=int32_t]" (601): here

/scratch/users/yuefanw/Retrival/MinkowskiEngine/src/spmm.cu: In instantiation of 'minkowski::coo_spmm(const at::Tensor&, const at::Tensor&, const at::Tensor&, int64_t, int64_t, const at::Tensor&, int64_t, bool)::<lambda()>::<lambda()> [with th_int_type = int]': /scratch/users/yuefanw/Retrival/MinkowskiEngine/src/spmm.cu:212:0: required from 'struct minkowski::coo_spmm(const at::Tensor&, const at::Tensor&, const at::Tensor&, int64_t, int64_t, const at::Tensor&, int64_t, bool)::<lambda()> [with th_int_type = int]::<lambda()>' /scratch/users/yuefanw/Retrival/MinkowskiEngine/src/spmm.cu:212:0: required from 'minkowski::coo_spmm(const at::Tensor&, const at::Tensor&, const at::Tensor&, int64_t, int64_t, const at::Tensor&, int64_t, bool)::<lambda()> [with th_int_type = int]' /scratch/users/yuefanw/Retrival/MinkowskiEngine/src/spmm.cu:212:0: required from 'struct minkowski::coo_spmm(const at::Tensor&, const at::Tensor&, const at::Tensor&, int64_t, int64_t, const at::Tensor&, int64_t, bool) [with th_int_type = int; int64_t = long int]::<lambda()>' /scratch/users/yuefanw/Retrival/MinkowskiEngine/src/spmm.cu:212:0: required from 'at::Tensor minkowski::coo_spmm(const at::Tensor&, const at::Tensor&, const at::Tensor&, int64_t, int64_t, const at::Tensor&, int64_t, bool) [with th_int_type = int; int64_t = long int]' /scratch/users/yuefanw/Retrival/MinkowskiEngine/src/spmm.cu:344:232: required from here /scratch/users/yuefanw/Retrival/MinkowskiEngine/src/spmm.cu:212:1292: internal compiler error: in maybe_undo_parenthesized_ref, at cp/semantics.c:1705 AT_DISPATCH_FLOATING_TYPES(vals.scalar_type(), "coo_spmm", [&] { ^ 0x68a069 maybe_undo_parenthesized_ref(tree_node) ../../gcc-7.2.0/gcc/cp/semantics.c:1704 0x6ce4a1 cp_fold ../../gcc-7.2.0/gcc/cp/cp-gimplify.c:2141 0x6cf0fc cp_fold_maybe_rvalue ../../gcc-7.2.0/gcc/cp/cp-gimplify.c:2003 0x6ce046 cp_fold ../../gcc-7.2.0/gcc/cp/cp-gimplify.c:2110 0x666b8f cp_build_binary_op(unsigned int, tree_code, tree_node, tree_node, int) ../../gcc-7.2.0/gcc/cp/typeck.c:5247 0x5af28c build_new_op_1 ../../gcc-7.2.0/gcc/cp/call.c:5982 0x5afd6e build_new_op(unsigned int, tree_code, int, tree_node, tree_node, tree_node, tree_node, int) ../../gcc-7.2.0/gcc/cp/call.c:6026 0x65e9f2 build_x_binary_op(unsigned int, tree_code, tree_node, tree_code, tree_node, tree_code, tree_node, int) ../../gcc-7.2.0/gcc/cp/typeck.c:3928 0x5eb6ff tsubst_copy_and_build(tree_node, tree_node, int, tree_node, bool, bool) ../../gcc-7.2.0/gcc/cp/pt.c:16937 0x5eb6a8 tsubst_copy_and_build(tree_node, tree_node, int, tree_node, bool, bool) ../../gcc-7.2.0/gcc/cp/pt.c:16934 0x5ebd06 tsubst_copy_and_build(tree_node, tree_node, int, tree_node, bool, bool) ../../gcc-7.2.0/gcc/cp/pt.c:17248 0x5eb908 tsubst_copy_and_build(tree_node, tree_node, int, tree_node, bool, bool) ../../gcc-7.2.0/gcc/cp/pt.c:17593 0x5ebfc5 tsubst_copy_and_build(tree_node, tree_node, int, tree_node, bool, bool) ../../gcc-7.2.0/gcc/cp/pt.c:16833 0x5ea7a9 tsubst_copy_and_build(tree_node, tree_node, int, tree_node, bool, bool) ../../gcc-7.2.0/gcc/cp/pt.c:17073 0x5e4327 tsubst_expr(tree_node, tree_node, int, tree_node, bool) ../../gcc-7.2.0/gcc/cp/pt.c:16550 0x5e4185 tsubst_expr(tree_node, tree_node, int, tree_node, bool) ../../gcc-7.2.0/gcc/cp/pt.c:15815 0x5e41d5 tsubst_expr(tree_node, tree_node, int, tree_node, bool) ../../gcc-7.2.0/gcc/cp/pt.c:15801 0x5e406b tsubst_expr(tree_node, tree_node, int, tree_node, bool) ../../gcc-7.2.0/gcc/cp/pt.c:16027 0x5e4be5 tsubst_expr(tree_node, tree_node, int, tree_node, bool) ../../gcc-7.2.0/gcc/cp/pt.c:15997 0x5e41d5 tsubst_expr(tree_node, tree_node, int, tree_node, bool) ../../gcc-7.2.0/gcc/cp/pt.c:15801 Please submit a full bug report, with preprocessed source if appropriate. Please include the complete backtrace with any bug report. See https://gcc.gnu.org/bugs/ for instructions. ninja: build stopped: subcommand failed. Traceback (most recent call last): File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1666, in _run_ninja_build subprocess.run( File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/subprocess.py", line 516, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['ninja', '-v', '-j', '1']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "setup.py", line 317, in setup( File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/setuptools/init.py", line 153, in setup return distutils.core.setup(**attrs) File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/distutils/core.py", line 148, in setup dist.run_commands() File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/distutils/dist.py", line 966, in run_commands self.run_command(cmd) File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/setuptools/command/install.py", line 67, in run self.do_egg_install() File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/setuptools/command/install.py", line 109, in do_egg_install self.run_command('bdist_egg') File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 164, in run cmd = self.call_command('install_lib', warn_dir=0) File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/setuptools/command/bdist_egg.py", line 150, in call_command self.run_command(cmdname) File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/setuptools/command/install_lib.py", line 11, in run self.build() File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/distutils/command/install_lib.py", line 107, in build self.run_command('build_ext') File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 79, in run _build_ext.run(self) File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/distutils/command/build_ext.py", line 340, in run self.build_extensions() File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 709, in build_extensions build_ext.build_extensions(self) File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions self._build_extensions_serial() File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial self.build_extension(ext) File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 202, in build_extension _build_ext.build_extension(self, ext) File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension objects = self.compiler.compile(sources, File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 530, in unix_wrap_ninja_compile _write_ninja_file_and_compile_objects( File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1355, in _write_ninja_file_and_compile_objects _run_ninja_build( File "/home/yuefanw/scratch/miniconda3/envs/mink/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1682, in _run_ninja_build raise RuntimeError(message) from e RuntimeError: Error compiling objects for extension

To Reproduce

conda create -n py3-mink python=3.8 conda activate py3-mink

conda install openblas-devel -c anaconda conda install pytorch=1.9.0 torchvision cudatoolkit=11.1 -c pytorch -c nvidia

git clone https://github.com/NVIDIA/MinkowskiEngine.git cd MinkowskiEngine python setup.py install --blas_include_dirs=${CONDA_PREFIX}/include --blas=openblas

Expected behavior

It is supposed to be installed correctly.

Desktop (please complete the following information):

Additional context Below is the result of code: wget -q https://raw.githubusercontent.com/NVIDIA/MinkowskiEngine/master/MinkowskiEngine/diagnostics.py ; python diagnostics.py

==========System========== Linux-3.10.0-1160.42.2.el7.x86_64-x86_64-with-glibc2.17 cat: /etc/lsb-release: No such file or directory 3.8.13 (default, Mar 28 2022, 11:38:47) [GCC 7.5.0] ==========Pytorch========== 1.9.0 torch.cuda.is_available(): True ==========NVIDIA-SMI========== /usr/bin/nvidia-smi Driver Version 510.47.03 CUDA Version 11.6 VBIOS Version 90.02.17.00.79 Image Version G001.0000.02.04 GSP Firmware Version N/A ==========NVCC========== /usr/local/cuda/11.1/bin/nvcc nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2020 NVIDIA Corporation Built on Mon_Oct_12_20:09:46_PDT_2020 Cuda compilation tools, release 11.1, V11.1.105 Build cuda_11.1.TC455_06.29190527_0 ==========CC========== CC=c++ /usr/local/gcc/7.2.0/bin/c++ c++ (GCC) 7.2.0 Copyright (C) 2017 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

==========MinkowskiEngine========== /scratch/users/yuefanw/Retrival/MinkowskiEngine/MinkowskiEngine/init.py:36: UserWarning: The environment variable OMP_NUM_THREADS not set. MinkowskiEngine will automatically set OMP_NUM_THREADS=16. If you want to set OMP_NUM_THREADS manually, please export it on the command line before running a python script. e.g. export OMP_NUM_THREADS=12; python your_program.py. It is recommended to set it below 24. warnings.warn( MinkowskiEngine not installed

tyhuang0428 commented 2 years ago

have you solved this error?

Iven-Wu commented 2 years ago

This is due to an old version of GCC. You can either update the GCC or use conda to install a proper version of gcc for your environment.

tyhuang0428 commented 2 years ago

This is due to an old version of GCC. You can either update the GCC or use conda to install a proper version of gcc for your environment.

Thanks for quick reply! So your current version of gcc is?