search

Jittor / jittor

Jittor is a high-performance deep learning framework based on JIT compiling and meta-operators.
https://cg.cs.tsinghua.edu.cn/jittor/
Apache License 2.0
3.08k stars 311 forks source link

assert jit_utils.cc AssertionError #310

Open hamzam0n opened 2 years ago

hamzam0n commented 2 years ago

i'm trying to run the next script: python3.8 -m jittor.test.test_example

and it gives the following error:

python3.8 -m jittor.test.test_example [i 0411 19:32:38.470641 92 lock.py:84] Create lock file:/home/killua/.cache/jittor/jt1.3.2/g++11.2.0/py3.8.0/Linux-5.16.0-kx4c/IntelRCoreTMi7xb1/jittor.lock [i 0411 19:32:38.484028 92 compiler.py:951] Jittor(1.3.2.6) src: /home/killua/anaconda3/envs/myenv/lib/python3.8/site-packages/jittor [i 0411 19:32:38.487111 92 compiler.py:952] g++ at /usr/bin/g++(11.2.0) [i 0411 19:32:38.487279 92 compiler.py:953] cache_path: /home/killua/.cache/jittor/jt1.3.2/g++11.2.0/py3.8.0/Linux-5.16.0-kx4c/IntelRCoreTMi7xb1/default [i 0411 19:32:38.503039 92 init.py:411] Found nvcc(11.4.152) at /usr/bin/nvcc. [i 0411 19:32:38.512382 92 init.py:411] Found addr2line(2.38) at /usr/bin/addr2line. [i 0411 19:32:38.644834 92 compiler.py:1006] cuda key:cu11.4.152_sm_50 [i 0411 19:32:38.666229 92 compiler.py:34] Create cache dir: /home/killua/.cache/jittor/jt1.3.2/g++11.2.0/py3.8.0/Linux-5.16.0-kx4c/IntelRCoreTMi7xb1/default/cu11.4.152_sm_50 [i 0411 19:32:38.666401 92 compiler.py:34] Create cache dir: /home/killua/.cache/jittor/jt1.3.2/g++11.2.0/py3.8.0/Linux-5.16.0-kx4c/IntelRCoreTMi7xb1/default/cu11.4.152_sm_50/jit [i 0411 19:32:38.666806 92 compiler.py:34] Create cache dir: /home/killua/.cache/jittor/jt1.3.2/g++11.2.0/py3.8.0/Linux-5.16.0-kx4c/IntelRCoreTMi7xb1/default/cu11.4.152_sm_50/obj_files [i 0411 19:32:38.666900 92 compiler.py:34] Create cache dir: /home/killua/.cache/jittor/jt1.3.2/g++11.2.0/py3.8.0/Linux-5.16.0-kx4c/IntelRCoreTMi7xb1/default/cu11.4.152_sm_50/gen [i 0411 19:32:38.666994 92 compiler.py:34] Create cache dir: /home/killua/.cache/jittor/jt1.3.2/g++11.2.0/py3.8.0/Linux-5.16.0-kx4c/IntelRCoreTMi7xb1/default/cu11.4.152_sm_50/tmp [i 0411 19:32:38.667082 92 compiler.py:34] Create cache dir: /home/killua/.cache/jittor/jt1.3.2/g++11.2.0/py3.8.0/Linux-5.16.0-kx4c/IntelRCoreTMi7xb1/default/cu11.4.152_sm_50/checkpoints Traceback (most recent call last): File "/home/killua/anaconda3/envs/myenv/lib/python3.8/runpy.py", line 183, in _run_module_as_main mod_name, mod_spec, code = _get_module_details(mod_name, _Error) File "/home/killua/anaconda3/envs/myenv/lib/python3.8/runpy.py", line 109, in _get_module_details import(pkg_name) File "/home/killua/anaconda3/envs/myenv/lib/python3.8/site-packages/jittor/init.py", line 18, in from . import compiler File "/home/killua/anaconda3/envs/myenv/lib/python3.8/site-packages/jittor/compiler.py", line 1171, in check_cache_compile() File "/home/killua/anaconda3/envs/myenv/lib/python3.8/site-packages/jittor/compiler.py", line 887, in check_cache_compile assert jit_utils.cc AssertionError

i have tried to change to python3.9 and 3.7 and stills same problem also i have tried to execute "rm -rf ~/.cache/jittor" but still same problem..

my cuda version is: 11.4 my system: x86_64 GNU/Linux ps: i'm using anaconda3

MenghaoGuo commented 2 years ago

This issue may be helpful for this problem.

I think it may be caused by the conda environment.

hamzam0n commented 2 years ago

when i try to run it without conda i get the following error

python3.9 -m jittor.test.test_example

[i 0414 16:22:36.553732 44 compiler.py:944] Jittor(1.3.1.45) src: /home/killua/.local/lib/python3.9/site-packages/jittor [i 0414 16:22:36.556234 44 compiler.py:945] g++ at /usr/bin/g++(11.2.0) [i 0414 16:22:36.556300 44 compiler.py:946] cache_path: /home/killua/.cache/jittor/jt1.3.1/g++11.2.0/py3.9.10/Linux-5.16.0-kx37/IntelRCoreTMi7xb1/default [i 0414 16:22:36.562929 44 install_cuda.py:51] cuda_driver_version: [11, 4] [i 0414 16:22:36.572933 44 init.py:376] Found /home/killua/.cache/jittor/jtcuda/cuda11.2_cudnn8_linux/bin/nvcc(11.2.152) at /home/killua/.cache/jittor/jtcuda/cuda11.2_cudnn8_linux/bin/nvcc. [i 0414 16:22:36.576367 44 init.py:376] Found addr2line(2.38) at /usr/bin/addr2line. [i 0414 16:22:36.819203 44 compiler.py:999] cuda key:cu11.2.152_sm_50 [i 0414 16:22:37.048374 44 init.py:191] Total mem: 31.06GB, using 10 procs for compiling. /usr/include/c++/11/type_traits(1406): error: type name is not allowed

/usr/include/c++/11/type_traits(1406): error: type name is not allowed

/usr/include/c++/11/type_traits(1406): error: identifier "__is_same" is undefined

/home/killua/.local/lib/python3.9/site-packages/jittor/src/misc/cstr.h(19): error: no instance of overloaded function "std::unique_ptr<_Tp [], _Dp>::reset [with _Tp=char, _Dp=std::default_delete<char []>]" matches the argument list argument types are: (char *) object type is: jittor::unique_ptr<char []>

/home/killua/.local/lib/python3.9/site-packages/jittor/src/misc/cstr.h(25): error: no instance of overloaded function "std::unique_ptr<_Tp [], _Dp>::reset [with _Tp=char, _Dp=std::default_delete<char []>]" matches the argument list argument types are: (char *) object type is: jittor::unique_ptr<char []>

/usr/include/c++/11/ext/string_conversions.h(85): error: no instance of overloaded function "_Range_chk::_S_chk" matches the argument list argument types are: (const long, std::is_same<int, int>) detected during instantiation of "_Ret __gnu_cxx::__stoa(_TRet ()(const _CharT , _CharT *, _Base...), const char , const _CharT , std::size_t , _Base...) [with _TRet=long, _Ret=int, _CharT=char, _Base=]" /usr/include/c++/11/bits/basic_string.h(6620): here

/usr/include/c++/11/ext/string_conversions.h(85): error: no instance of overloaded function "_Range_chk::_S_chk" matches the argument list argument types are: (const long, std::is_same<long, int>) detected during instantiation of "_Ret __gnu_cxx::__stoa(_TRet ()(const _CharT , _CharT *, _Base...), const char , const _CharT , std::size_t , _Base...) [with _TRet=long, _Ret=long, _CharT=char, _Base=]" /usr/include/c++/11/bits/basic_string.h(6625): here

/usr/include/c++/11/ext/string_conversions.h(85): error: no instance of overloaded function "_Range_chk::_S_chk" matches the argument list argument types are: (const unsigned long, std::is_same<unsigned long, int>) detected during instantiation of "_Ret __gnu_cxx::__stoa(_TRet ()(const _CharT , _CharT *, _Base...), const char , const _CharT , std::size_t , _Base...) [with _TRet=unsigned long, _Ret=unsigned long, _CharT=char, _Base=]" /usr/include/c++/11/bits/basic_string.h(6630): here

/usr/include/c++/11/ext/string_conversions.h(85): error: no instance of overloaded function "_Range_chk::_S_chk" matches the argument list argument types are: (const long long, std::is_same<long long, int>) detected during instantiation of "_Ret __gnu_cxx::__stoa(_TRet ()(const _CharT , _CharT *, _Base...), const char , const _CharT , std::size_t , _Base...) [with _TRet=long long, _Ret=long long, _CharT=char, _Base=]" /usr/include/c++/11/bits/basic_string.h(6635): here

/usr/include/c++/11/ext/string_conversions.h(85): error: no instance of overloaded function "_Range_chk::_S_chk" matches the argument list argument types are: (const unsigned long long, std::is_same<unsigned long long, int>) detected during instantiation of "_Ret __gnu_cxx::__stoa(_TRet ()(const _CharT , _CharT *, _Base...), const char , const _CharT , std::size_t , _Base...) [with _TRet=unsigned long long, _Ret=unsigned long long, _CharT=char, _Base=]" /usr/include/c++/11/bits/basic_string.h(6640): here

/usr/include/c++/11/ext/string_conversions.h(85): error: no instance of overloaded function "_Range_chk::_S_chk" matches the argument list argument types are: (const float, std::is_same<float, int>) detected during instantiation of "_Ret __gnu_cxx::__stoa(_TRet ()(const _CharT , _CharT *, _Base...), const char , const _CharT , std::size_t , _Base...) [with _TRet=float, _Ret=float, _CharT=char, _Base=<>]" /usr/include/c++/11/bits/basic_string.h(6646): here

/usr/include/c++/11/ext/string_conversions.h(85): error: no instance of overloaded function "_Range_chk::_S_chk" matches the argument list argument types are: (const double, std::is_same<double, int>) detected during instantiation of "_Ret __gnu_cxx::__stoa(_TRet ()(const _CharT , _CharT *, _Base...), const char , const _CharT , std::size_t , _Base...) [with _TRet=double, _Ret=double, _CharT=char, _Base=<>]" /usr/include/c++/11/bits/basic_string.h(6650): here

/usr/include/c++/11/ext/string_conversions.h(85): error: no instance of overloaded function "_Range_chk::_S_chk" matches the argument list argument types are: (const long double, std::is_same<long double, int>) detected during instantiation of "_Ret __gnu_cxx::__stoa(_TRet ()(const _CharT , _CharT *, _Base...), const char , const _CharT , std::size_t , _Base...) [with _TRet=long double, _Ret=long double, _CharT=char, _Base=<>]" /usr/include/c++/11/bits/basic_string.h(6654): here

/usr/include/c++/11/ext/string_conversions.h(85): error: no instance of overloaded function "_Range_chk::_S_chk" matches the argument list argument types are: (const long, std::is_same<int, int>) detected during instantiation of "_Ret __gnu_cxx::__stoa(_TRet ()(const _CharT , _CharT *, _Base...), const char , const _CharT , std::size_t , _Base...) [with _TRet=long, _Ret=int, _CharT=wchar_t, _Base=]" /usr/include/c++/11/bits/basic_string.h(6751): here

/usr/include/c++/11/ext/string_conversions.h(85): error: no instance of overloaded function "_Range_chk::_S_chk" matches the argument list argument types are: (const long, std::is_same<long, int>) detected during instantiation of "_Ret __gnu_cxx::__stoa(_TRet ()(const _CharT , _CharT *, _Base...), const char , const _CharT , std::size_t , _Base...) [with _TRet=long, _Ret=long, _CharT=wchar_t, _Base=]" /usr/include/c++/11/bits/basic_string.h(6756): here

/usr/include/c++/11/ext/string_conversions.h(85): error: no instance of overloaded function "_Range_chk::_S_chk" matches the argument list argument types are: (const unsigned long, std::is_same<unsigned long, int>) detected during instantiation of "_Ret __gnu_cxx::__stoa(_TRet ()(const _CharT , _CharT *, _Base...), const char , const _CharT , std::size_t , _Base...) [with _TRet=unsigned long, _Ret=unsigned long, _CharT=wchar_t, _Base=]" /usr/include/c++/11/bits/basic_string.h(6761): here

/usr/include/c++/11/ext/string_conversions.h(85): error: no instance of overloaded function "_Range_chk::_S_chk" matches the argument list argument types are: (const long long, std::is_same<long long, int>) detected during instantiation of "_Ret __gnu_cxx::__stoa(_TRet ()(const _CharT , _CharT *, _Base...), const char , const _CharT , std::size_t , _Base...) [with _TRet=long long, _Ret=long long, _CharT=wchar_t, _Base=]" /usr/include/c++/11/bits/basic_string.h(6766): here

/usr/include/c++/11/ext/string_conversions.h(85): error: no instance of overloaded function "_Range_chk::_S_chk" matches the argument list argument types are: (const unsigned long long, std::is_same<unsigned long long, int>) detected during instantiation of "_Ret __gnu_cxx::__stoa(_TRet ()(const _CharT , _CharT *, _Base...), const char , const _CharT , std::size_t , _Base...) [with _TRet=unsigned long long, _Ret=unsigned long long, _CharT=wchar_t, _Base=]" /usr/include/c++/11/bits/basic_string.h(6771): here

/usr/include/c++/11/ext/string_conversions.h(85): error: no instance of overloaded function "_Range_chk::_S_chk" matches the argument list argument types are: (const float, std::is_same<float, int>) detected during instantiation of "_Ret __gnu_cxx::__stoa(_TRet ()(const _CharT , _CharT *, _Base...), const char , const _CharT , std::size_t , _Base...) [with _TRet=float, _Ret=float, _CharT=wchar_t, _Base=<>]" /usr/include/c++/11/bits/basic_string.h(6777): here

/usr/include/c++/11/ext/string_conversions.h(85): error: no instance of overloaded function "_Range_chk::_S_chk" matches the argument list argument types are: (const double, std::is_same<double, int>) detected during instantiation of "_Ret __gnu_cxx::__stoa(_TRet ()(const _CharT , _CharT *, _Base...), const char , const _CharT , std::size_t , _Base...) [with _TRet=double, _Ret=double, _CharT=wchar_t, _Base=<>]" /usr/include/c++/11/bits/basic_string.h(6781): here

/usr/include/c++/11/ext/string_conversions.h(85): error: no instance of overloaded function "_Range_chk::_S_chk" matches the argument list argument types are: (const long double, std::is_same<long double, int>) detected during instantiation of "_Ret __gnu_cxx::__stoa(_TRet ()(const _CharT , _CharT *, _Base...), const char , const _CharT , std::size_t , _Base...) [with _TRet=long double, _Ret=long double, _CharT=wchar_t, _Base=<>]" /usr/include/c++/11/bits/basic_string.h(6785): here

21 errors detected in the compilation of "/home/killua/.local/lib/python3.9/site-packages/jittor/src/misc/nan_checker.cu". multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/usr/lib/python3.9/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "/home/killua/.local/lib/python3.9/site-packages/jittor_utils/init.py", line 161, in do_compile return cc.cache_compile(cmd, cache_path, jittor_path) RuntimeError: [f 0414 16:22:38.618127 44 log.cc:608] Check failed ret(256) == 0(0) Run cmd failed: "/home/killua/.cache/jittor/jtcuda/cuda11.2_cudnn8_linux/bin/nvcc" "/home/killua/.local/lib/python3.9/site-packages/jittor/src/misc/nan_checker.cu" -std=c++14 -Xcompiler -fPIC -Xcompiler -march=native -Xcompiler -fdiagnostics-color=always -I"/home/killua/.local/lib/python3.9/site-packages/jittor/src" -I/usr/include/python3.9 -I/usr/include/python3.9 -DHAS_CUDA -I"/home/killua/.cache/jittor/jtcuda/cuda11.2_cudnn8_linux/include" -I"/home/killua/.local/lib/python3.9/site-packages/jittor/extern/cuda/inc" -I"/home/killua/.cache/jittor/jt1.3.1/g++11.2.0/py3.9.10/Linux-5.16.0-kx37/IntelRCoreTMi7xb1/default/cu11.2.152_sm_50" -O2 -c -o "/home/killua/.cache/jittor/jt1.3.1/g++11.2.0/py3.9.10/Linux-5.16.0-kx37/IntelRCoreTMi7xb1/default/cu11.2.152_sm_50/obj_files/nan_checker.cu.o" -x cu --cudart=shared -ccbin="/usr/bin/g++" -w -I"/home/killua/.local/lib/python3.9/site-packages/jittor/extern/cuda/inc" """

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/lib/python3.9/runpy.py", line 188, in _run_module_as_main mod_name, mod_spec, code = _get_module_details(mod_name, _Error) File "/usr/lib/python3.9/runpy.py", line 111, in _get_module_details import(pkg_name) File "/home/killua/.local/lib/python3.9/site-packages/jittor/init.py", line 18, in from . import compiler File "/home/killua/.local/lib/python3.9/site-packages/jittor/compiler.py", line 1305, in compile(cc_path, cc_flags+opt_flags, files, 'jittor_core'+extension_suffix) File "/home/killua/.local/lib/python3.9/site-packages/jittor/compiler.py", line 147, in compile jit_utils.run_cmds(cmds, cache_path, jittor_path, "Compiling "+base_output) File "/home/killua/.local/lib/python3.9/site-packages/jittor_utils/init.py", line 215, in runcmds for i, in enumerate(p.imap_unordered(do_compile, cmds)): File "/usr/lib/python3.9/multiprocessing/pool.py", line 870, in next raise value RuntimeError: [f 0414 16:22:38.618127 44 log.cc:608] Check failed ret(256) == 0(0) Run cmd failed: "/home/killua/.cache/jittor/jtcuda/cuda11.2_cudnn8_linux/bin/nvcc" "/home/killua/.local/lib/python3.9/site-packages/jittor/src/misc/nan_checker.cu" -std=c++14 -Xcompiler -fPIC -Xcompiler -march=native -Xcompiler -fdiagnostics-color=always -I"/home/killua/.local/lib/python3.9/site-packages/jittor/src" -I/usr/include/python3.9 -I/usr/include/python3.9 -DHAS_CUDA -I"/home/killua/.cache/jittor/jtcuda/cuda11.2_cudnn8_linux/include" -I"/home/killua/.local/lib/python3.9/site-packages/jittor/extern/cuda/inc" -I"/home/killua/.cache/jittor/jt1.3.1/g++11.2.0/py3.9.10/Linux-5.16.0-kx37/IntelRCoreTMi7xb1/default/cu11.2.152_sm_50" -O2 -c -o "/home/killua/.cache/jittor/jt1.3.1/g++11.2.0/py3.9.10/Linux-5.16.0-kx37/IntelRCoreTMi7xb1/default/cu11.2.152_sm_50/obj_files/nan_checker.cu.o" -x cu --cudart=shared -ccbin="/usr/bin/g++" -w -I"/home/killua/.local/lib/python3.9/site-packages/jittor/extern/cuda/inc"

Xunius commented 2 years ago

@hamzam0n I guess your g++ version is too high.

Try using g++8. You can install one in a conda env using conda install -c conda-forge gxx=8, mpich.