mmaGroupGitHub commented 3 years ago

I followd installation instructions and ran bash make.sh with no errors, however when I try to run the train.py script I get the following error: from ._dcn_v2 import lib as _lib, ffi as _ffi undefined symbol __cudaPopCallConfiguration any ideas as to why this is happening?

ssk1997 commented 3 years ago

you need gcc-5 and g++5

mmaGroupGitHub commented 3 years ago

@ssk1997 I have both of these now getting the following:

Traceback (most recent call last): File "build.py", line 21, in raise ValueError('CUDA is not available') ValueError: CUDA is not available Traceback (most recent call last): File "build_double.py", line 21, in raise ValueError('CUDA is not available') ValueError: CUDA is not available

mmaGroupGitHub commented 3 years ago

@ssk1997 ran sudo reboot because pytorch was not recognizing my gpu -- gpu is visible to pytorch tried to recompile the make file which ran without issues and when I try to train the model I am back to getting the ImportError undefined symbol: __cudaPopCallConfiguration

and I have both gcc5 and g++5 installed.

mmaGroupGitHub commented 3 years ago

@ssk1997

If it is helpful here is the output I am getting from running bash make.sh

/home/alec/MOC-Detector/src/network/DCNv2 Including CUDA code. /home/alec/MOC-Detector/src/network/DCNv2 generating /tmp/tmpf_7d41if/_dcn_v2.c setting the current directory to '/tmp/tmpf_7d41if' running build_ext building '_dcn_v2' extension creating home creating home/alec creating home/alec/MOC-Detector creating home/alec/MOC-Detector/src creating home/alec/MOC-Detector/src/network creating home/alec/MOC-Detector/src/network/DCNv2 creating home/alec/MOC-Detector/src/network/DCNv2/src gcc -pthread -B /home/alec/anaconda3/envs/MOC/compiler_compat -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/alec/anaconda3/envs/MOC/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include -I/home/alec/anaconda3/envs/MOC/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include/TH -I/home/alec/anaconda3/envs/MOC/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include/THC -I/home/alec/anaconda3/envs/MOC/include/python3.5m -c _dcn_v2.c -o ./_dcn_v2.o -std=c99 -fopenmp -std=c99 gcc -pthread -B /home/alec/anaconda3/envs/MOC/compiler_compat -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/alec/anaconda3/envs/MOC/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include -I/home/alec/anaconda3/envs/MOC/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include/TH -I/home/alec/anaconda3/envs/MOC/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include/THC -I/home/alec/anaconda3/envs/MOC/include/python3.5m -c /home/alec/MOC-Detector/src/network/DCNv2/src/dcn_v2.c -o ./home/alec/MOC-Detector/src/network/DCNv2/src/dcn_v2.o -std=c99 -fopenmp -std=c99 gcc -pthread -B /home/alec/anaconda3/envs/MOC/compiler_compat -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/alec/anaconda3/envs/MOC/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include -I/home/alec/anaconda3/envs/MOC/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include/TH -I/home/alec/anaconda3/envs/MOC/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include/THC -I/home/alec/anaconda3/envs/MOC/include/python3.5m -c /home/alec/MOC-Detector/src/network/DCNv2/src/dcn_v2_cuda.c -o ./home/alec/MOC-Detector/src/network/DCNv2/src/dcn_v2_cuda.o -std=c99 -fopenmp -std=c99 gcc -pthread -shared -L/home/alec/anaconda3/envs/MOC/lib -B /home/alec/anaconda3/envs/MOC/compiler_compat -Wl,-rpath=/home/alec/anaconda3/envs/MOC/lib,--no-as-needed ./_dcn_v2.o ./home/alec/MOC-Detector/src/network/DCNv2/src/dcn_v2.o ./home/alec/MOC-Detector/src/network/DCNv2/src/dcn_v2_cuda.o /home/alec/MOC-Detector/src/network/DCNv2/src/cuda/dcn_v2_im2col_cuda.cu.o /home/alec/MOC-Detector/src/network/DCNv2/src/cuda/dcn_v2_psroi_pooling_cuda.cu.o -o ./_dcn_v2.so Including CUDA code. /home/alec/MOC-Detector/src/network/DCNv2 generating /tmp/tmpp2xs_zyc/_dcn_v2_double.c setting the current directory to '/tmp/tmpp2xs_zyc' running build_ext building '_dcn_v2_double' extension creating home creating home/alec creating home/alec/MOC-Detector creating home/alec/MOC-Detector/src creating home/alec/MOC-Detector/src/network creating home/alec/MOC-Detector/src/network/DCNv2 creating home/alec/MOC-Detector/src/network/DCNv2/src gcc -pthread -B /home/alec/anaconda3/envs/MOC/compiler_compat -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/alec/anaconda3/envs/MOC/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include -I/home/alec/anaconda3/envs/MOC/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include/TH -I/home/alec/anaconda3/envs/MOC/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include/THC -I/home/alec/anaconda3/envs/MOC/include/python3.5m -c _dcn_v2_double.c -o ./_dcn_v2_double.o -std=c99 -fopenmp -std=c99 gcc -pthread -B /home/alec/anaconda3/envs/MOC/compiler_compat -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/alec/anaconda3/envs/MOC/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include -I/home/alec/anaconda3/envs/MOC/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include/TH -I/home/alec/anaconda3/envs/MOC/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include/THC -I/home/alec/anaconda3/envs/MOC/include/python3.5m -c /home/alec/MOC-Detector/src/network/DCNv2/src/dcn_v2_double.c -o ./home/alec/MOC-Detector/src/network/DCNv2/src/dcn_v2_double.o -std=c99 -fopenmp -std=c99 gcc -pthread -B /home/alec/anaconda3/envs/MOC/compiler_compat -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/alec/anaconda3/envs/MOC/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include -I/home/alec/anaconda3/envs/MOC/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include/TH -I/home/alec/anaconda3/envs/MOC/lib/python3.5/site-packages/torch/utils/ffi/../../lib/include/THC -I/home/alec/anaconda3/envs/MOC/include/python3.5m -c /home/alec/MOC-Detector/src/network/DCNv2/src/dcn_v2_cuda_double.c -o ./home/alec/MOC-Detector/src/network/DCNv2/src/dcn_v2_cuda_double.o -std=c99 -fopenmp -std=c99 gcc -pthread -shared -L/home/alec/anaconda3/envs/MOC/lib -B /home/alec/anaconda3/envs/MOC/compiler_compat -Wl,-rpath=/home/alec/anaconda3/envs/MOC/lib,--no-as-needed ./_dcn_v2_double.o ./home/alec/MOC-Detector/src/network/DCNv2/src/dcn_v2_double.o ./home/alec/MOC-Detector/src/network/DCNv2/src/dcn_v2_cuda_double.o /home/alec/MOC-Detector/src/network/DCNv2/src/cuda/dcn_v2_im2col_cuda_double.cu.o /home/alec/MOC-Detector/src/network/DCNv2/src/cuda/dcn_v2_psroi_pooling_cuda_double.cu.o -o ./_dcn_v2_double.so

ssk1997 commented 3 years ago

Are you sure you have cuda 9.0, gcc-5 and g++-5 installed ? From the message you are using gcc-7. Try the command "gcc -v".

mmaGroupGitHub commented 3 years ago

Yea I do now:

Name Version Build Channel

cudatoolkit 9.0 h13b8566_0

Thread model: posix gcc version 5.5.0 20171010 (Ubuntu 5.5.0-12ubuntu1)

Unfortunately I now get a bunch of syntactical errors during compilation:

bash make.sh /usr/lib/gcc/x86_64-linux-gnu/7/include/stddef.h(444): error: identifier "nullptr" is undefined

/usr/lib/gcc/x86_64-linux-gnu/7/include/stddef.h(444): error: expected a ";"

/usr/include/x86_64-linux-gnu/c++/7/bits/c++config.h(235): error: expected a ";"

/usr/include/c++/7/bits/exception.h(63): error: expected a ";"

/usr/include/c++/7/bits/exception.h(69): error: expected a ";"

/usr/include/c++/7/exception(49): error: expected a ";"

/usr/include/c++/7/exception(57): error: expected a ";"

/usr/include/c++/7/exception(67): error: expected a "{"

/usr/include/c++/7/bits/cxxabi_init_exception.h(63): error: expected a "{"

/usr/include/c++/7/typeinfo(99): error: expected a ";"

/usr/include/c++/7/typeinfo(187): error: not a class or struct name

/usr/include/c++/7/typeinfo(190): error: expected a ";"

/usr/include/c++/7/typeinfo(197): error: expected a ";"

/usr/include/c++/7/typeinfo(204): error: not a class or struct name

/usr/include/c++/7/typeinfo(207): error: expected a ";"

/usr/include/c++/7/typeinfo(214): error: expected a ";"

/usr/include/c++/7/bits/exception_ptr.h(63): error: function "__cxxabiv1::std::current_exception" returns incomplete type "cxxabiv1::std::exception_ptr::exception_ptr"

/usr/include/c++/7/bits/exception_ptr.h(63): error: expected a "{"

/usr/include/c++/7/bits/exception_ptr.h(73): error: namespace "__cxxabiv1::std" has no member "rethrow_exception"

/usr/include/c++/7/bits/exception_ptr.h(83): error: expected a ";"

/usr/include/c++/7/bits/exception_ptr.h(85): error: expected a ";"

/usr/include/c++/7/bits/exception_ptr.h(86): error: expected a ";"

/usr/include/c++/7/bits/exception_ptr.h(88): error: expected a ";"

/usr/include/c++/7/bits/exception_ptr.h(90): error: declaration is incompatible with previous "__cxxabiv1::std::current_exception" (63): here

/usr/include/c++/7/bits/exception_ptr.h(90): error: use of a local type to declare a function

/usr/include/c++/7/bits/exception_ptr.h(90): error: expected a ";"

/usr/include/c++/7/bits/exception_ptr.h(91): error: use of a local type to declare a function

/usr/include/c++/7/bits/exception_ptr.h(93): error: namespace "__cxxabiv1::std" has no member "make_exception_ptr"

/usr/include/c++/7/bits/exception_ptr.h(93): error: a template friend declaration cannot be declared in a local class

/usr/include/c++/7/bits/exception_ptr.h(93): error: expected a ";"

/usr/include/c++/7/bits/exception_ptr.h(96): error: expected a ";"

/usr/include/c++/7/bits/exception_ptr.h(98): error: expected a ";"

/usr/include/c++/7/bits/exception_ptr.h(101): error: incomplete type is not allowed

/usr/include/c++/7/bits/exception_ptr.h(101): error: expected a ";"

/usr/include/c++/7/bits/exception_ptr.h(122): error: expected a ";"

/usr/include/c++/7/bits/exception_ptr.h(132): error: expected a ";"

/usr/include/c++/7/bits/exception_ptr.h(149): error: use of a local type to declare a function

/usr/include/c++/7/bits/exception_ptr.h(150): error: expected a ";"

/usr/include/c++/7/bits/exception_ptr.h(153): error: expected a ";"

/usr/include/c++/7/bits/exception_ptr.h(158): error: use of a local type to declare a function

/usr/include/c++/7/bits/exception_ptr.h(159): error: expected a ";"

/usr/include/c++/7/bits/exception_ptr.h(162): error: use of a local type to declare a function

/usr/include/c++/7/bits/exception_ptr.h(163): error: expected a ";"

/usr/include/c++/7/bits/exception_ptr.h(166): error: use of a local type to declare a function

/usr/include/c++/7/bits/exception_ptr.h(167): error: expected a ";"

/usr/include/c++/7/bits/exception_ptr.h(179): error: expected a ";"

/usr/include/c++/7/bits/exception_ptr.h(220): error: expected a ";"

/usr/include/c++/7/bits/move.h(46): error: identifier "constexpr" is undefined

/usr/include/c++/7/bits/move.h(46): error: "_Tp" is not a function or static data member

/usr/include/c++/7/bits/move.h(51): error: expected a ";"

/usr/include/c++/7/type_traits(71): error: identifier "constexpr" is undefined

/usr/include/c++/7/type_traits(71): error: template parameter "_Tp" may not be redeclared in this scope

/usr/include/c++/7/type_traits(71): error: expected a ";"

/usr/include/c++/7/type_traits(72): error: member "cxxabiv1::std::integral_constant<_Tp, v>::_Tp" is not a type name

/usr/include/c++/7/type_traits(73): error: member "cxxabiv1::std::integral_constant<_Tp, v>::_Tp" is not a type name

/usr/include/c++/7/type_traits(74): error: identifier "constexpr" is undefined

/usr/include/c++/7/type_traits(74): error: expected a ";"

/usr/include/c++/7/type_traits(84): error: identifier "constexpr" is undefined

/usr/include/c++/7/type_traits(84): error: "_Tp" is not a function or static data member

/usr/include/c++/7/type_traits(93): error: expected a declaration

/usr/include/c++/7/type_traits(93): error: expected a ";"

/usr/include/c++/7/type_traits(126): error: space required between adjacent ">" delimiters of nested template argument lists (">>" is the right shift operator)

/usr/include/c++/7/type_traits(154): error: __bool_constant is not a template

/usr/include/c++/7/type_traits(154): error: not a class or struct name

/usr/include/c++/7/type_traits(245): error: identifier "char16_t" is undefined

/usr/include/c++/7/type_traits(249): error: identifier "char32_t" is undefined

/usr/include/c++/7/type_traits(249): error: class "cxxabiv1::std::is_integral_helper<>" has already been defined

/usr/include/c++/7/type_traits(362): error: namespace "__cxxabiv1::std" has no member "size_t"

/usr/include/c++/7/type_traits(463): error: expected a ">"

/usr/include/c++/7/type_traits(467): error: expected a ">"

/usr/include/c++/7/type_traits(475): error: expected a ">"

/usr/include/c++/7/type_traits(479): error: expected a ">"

/usr/include/c++/7/type_traits(487): error: expected a ">"

/usr/include/c++/7/type_traits(491): error: expected a ">"

/usr/include/c++/7/type_traits(499): error: expected a ">"

/usr/include/c++/7/type_traits(503): error: expected a ">"

/usr/include/c++/7/type_traits(511): error: expected a ">"

/usr/include/c++/7/type_traits(515): error: expected a ">"

/usr/include/c++/7/type_traits(523): error: expected a ">"

/usr/include/c++/7/type_traits(527): error: expected a ">"

/usr/include/c++/7/type_traits(535): error: expected a ">"

/usr/include/c++/7/type_traits(539): error: expected a ">"

/usr/include/c++/7/type_traits(547): error: expected a ">"

/usr/include/c++/7/type_traits(551): error: expected a ">"

/usr/include/c++/7/type_traits(561): error: namespace "__cxxabiv1::std" has no member "nullptr_t"

/usr/include/c++/7/type_traits(582): error: space required between adjacent ">" delimiters of nested template argument lists (">>" is the right shift operator)

/usr/include/c++/7/type_traits(588): error: space required between adjacent ">" delimiters of nested template argument lists (">>" is the right shift operator)

/usr/include/c++/7/type_traits(595): error: space required between adjacent ">" delimiters of nested template argument lists (">>" is the right shift operator)

/usr/include/c++/7/type_traits(602): error: space required between adjacent ">" delimiters of nested template argument lists (">>" is the right shift operator)

/usr/include/c++/7/type_traits(612): error: space required between adjacent ">" delimiters of nested template argument lists (">>" is the right shift operator)

/usr/include/c++/7/type_traits(638): error: space required between adjacent ">" delimiters of nested template argument lists (">>" is the right shift operator)

/usr/include/c++/7/type_traits(748): error: space required between adjacent ">" delimiters of nested template argument lists (">>" is the right shift operator)

/usr/include/c++/7/type_traits(762): error: expected a ";"

/usr/include/c++/7/type_traits(777): error: space required between adjacent ">" delimiters of nested template argument lists (">>" is the right shift operator)

/usr/include/c++/7/type_traits(787): error: expected a ")"

/usr/include/c++/7/type_traits(798): error: an explicit template argument list is not allowed on this declaration

/usr/include/c++/7/type_traits(798): error: expected a type specifier

/usr/include/c++/7/type_traits(798): error: function returning function is not allowed

/usr/include/c++/7/type_traits(798): error: expected a ";"

/usr/include/c++/7/type_traits(804): error: space required between adjacent ">" delimiters of nested template argument lists (">>" is the right shift operator)

Error limit reached. 100 errors detected in the compilation of "/tmp/tmpxft_00002bac_00000000-6_dcn_v2_im2col_cuda.cpp1.ii". Compilation terminated. /usr/lib/gcc/x86_64-linux-gnu/7/include/stddef.h(444): error: identifier "nullptr" is undefined