search

taichi-dev / taichi

Productive, portable, and performant GPU programming in Python.
https://taichi-lang.org
Apache License 2.0
25.37k stars 2.27k forks source link

RuntimeError: [taichi/backends/cuda/cuda_driver.h:taichi #2054

Closed Fin-Del-Mundo closed 3 years ago

Fin-Del-Mundo commented 3 years ago

when run examples/fractal.py, i find the bug: taichi::lang::CUDADriverFunction<void ,char const ,unsigned int,unsigned int ,void >::operator ()@80] CUDA Error CUDA_ERROR_INVALID_PTX: a PTX JIT compilation failed while calling module_load_data_ex (cuModuleLoadDataEx)

CAESIUS-TIM commented 3 years ago

image

RonDingDing commented 3 years ago

When I init with cpu it's fine, but init with gpu gives me this

[Taichi] materializing...
[E 06/14/21 14:24:26.811] [taichi/backends/cuda/cuda_driver.h:taichi::lang::CUDADriverFunction<void * *,char const *,unsigned int,unsigned int *,void * *>::operator ()@81] CUDA Error CUDA_ERROR_INVALID_PTX: a PTX JIT compilation failed while calling module_load_data_ex (cuModuleLoadDataEx)

***********************************
* Taichi Compiler Stack Traceback *
***********************************
    0x7fff0bc383e7: taichi::create_instance_placement<taichi::Benchmark>
 in taichi_core.pyd
0x7fff0bb29449: taichi::create_instance_placement<taichi::Benchmark>
 in taichi_core.pyd
0x7fff0bd7200b: taichi::create_instance_placement<taichi::Benchmark>
 in taichi_core.pyd
0x7fff0bbb4d89: taichi::create_instance_placement<taichi::Benchmark>
 in taichi_core.pyd
0x7fff0bbc22ef: taichi::create_instance_placement<taichi::Benchmark>
 in taichi_core.pyd
0x7fff0bc225a7: taichi::create_instance_placement<taichi::Benchmark>
 in taichi_core.pyd
0x7fff0bbfe8c4: taichi::create_instance_placement<taichi::Benchmark>
 in taichi_core.pyd
0x7fff0bb65989: taichi::create_instance_placement<taichi::Benchmark>
 in taichi_core.pyd
0x7fff0ba508a0: taichi::create_instance_placement<taichi::Benchmark>
 in taichi_core.pyd
0x7fff0b9df046: taichi::create_instance_placement<taichi::Benchmark>
 in taichi_core.pyd
0x7fff0b9d50e7: taichi::create_instance_placement<taichi::Benchmark>
 in taichi_core.pyd
0x7fff12463cc4: PyCFunction_Call in python38.dll
0x7fff12444dca: PyEval_EvalFrameDefault in python38.dll
0x7fff1243e618: PyEval_EvalCodeWithName in python38.dll
0x7fff124439b2: PyEval_EvalFrameDefault in python38.dll
0x7fff1243e618: PyEval_EvalCodeWithName in python38.dll
0x7fff12445675: PyEval_EvalFrameDefault in python38.dll
0x7fff1243e618: PyEval_EvalCodeWithName in python38.dll
0x7fff1243fd5f: PyFunction_Vectorcall in python38.dll
0x7fff1243e24c: PyObject_GetBuffer in python38.dll
0x7fff1245a6a2: PyVectorcall_Call in python38.dll
0x7fff1245a533: PySequence_GetItem in python38.dll
0x7fff124467a5: PyEval_EvalFrameDefault in python38.dll
0x7fff1243e618: PyEval_EvalCodeWithName in python38.dll
0x7fff1243fd5f: PyFunction_Vectorcall in python38.dll
0x7fff124330cc: PyObject_FastCallDict in python38.dll
0x7fff12535ca6: PyObject_Call_Prepend in python38.dll
0x7fff12535c15: PyNumber_InPlaceMultiply in python38.dll
0x7fff1245a5af: PySequence_GetItem in python38.dll
0x7fff124467a5: PyEval_EvalFrameDefault in python38.dll
0x7fff1243e618: PyEval_EvalCodeWithName in python38.dll
0x7fff12444a7c: PyEval_EvalFrameDefault in python38.dll
0x7fff1243e618: PyEval_EvalCodeWithName in python38.dll
0x7fff1245315b: PyEval_EvalCodeEx in python38.dll
0x7fff124530b9: PyEval_EvalCode in python38.dll
0x7fff12452ac6: PyArena_New in python38.dll
0x7fff12452a55: PyArena_New in python38.dll
0x7fff124fa1a3:                            Py_wfopen in python38.dll
0x7fff124f85a8: PyUnicode_CompareWithASCIIString in python38.dll    
0x7fff124f7837: PyRun_SimpleFileExFlags in python38.dll
0x7fff126173ff: PyRun_AnyFileExFlags in python38.dll
0x7fff125c6453: Py_gitversion in python38.dll
0x7fff124fb494: Py_RunMain in python38.dll
0x7fff124fb31d: Py_RunMain in python38.dll
0x7fff124faecd: Py_Main in python38.dll
0x7ff66bfc1258: Unknown Function in python.exe
0x7fff79f77bd4: BaseThreadInitThunk in KERNEL32.DLL
0x7fff7bdcce71: RtlUserThreadStart in ntdll.dll

Internal error occurred. Check out this page for possible solutions:
https://taichi.readthedocs.io/en/stable/install.html#troubleshooting
    Traceback (most recent call last):
  File "1.py", line 29, in <module>
    paint(i * 0.03)
  File "D:\venv\lib\site-packages\taichi\lang\kernel_impl.py", line 634, in wrapped
    return primal(*args, **kwargs)
  File "D:\venv\lib\site-packages\taichi\lang\kernel_impl.py", line 561, in __call__
    key = self.ensure_compiled(*args)
  File "D:\venv\lib\site-packages\taichi\lang\kernel_impl.py", line 552, in ensure_compiled
    self.materialize(key=key, args=args, arg_features=arg_features)
  File "D:\venv\lib\site-packages\taichi\lang\kernel_impl.py", line 354, in materialize
    self.runtime.materialize()
  File "D:\venv\lib\site-packages\taichi\lang\impl.py", line 253, in materialize
    _ti_core.layout(layout)
RuntimeError: [taichi/backends/cuda/cuda_driver.h:taichi::lang::CUDADriverFunction<void * *,char const *,unsigned int,unsigned int *,void * *>::operator ()@81] CUDA Error CUDA_ERROR_INVALID_PTX: a PTX JIT compilation failed while calling module_load_data_ex (cuModuleLoadDataEx)
dwxrycb123 commented 3 years ago

I met the same problem. It's fine when I use ti.opengl, while it reports this when using ti.gpu or ti.cuda:

RuntimeError: [taichi/backends/cuda/cuda_driver.h:taichi::lang::CUDADriverFunction<void ,char const ,unsigned int,unsigned int ,void >::operator ()@81] CUDA Error CUDA_ERROR_INVALID_PTX: a PTX JIT compilation failed while calling module_load_data_ex (cuModuleLoadDataEx)

k-ye commented 3 years ago

Could you report your CUDA version, taichi version etc? (Possibly attach the output of ti diagnose)

There are a few similar issues before: https://github.com/taichi-dev/taichi/issues/1980, https://github.com/taichi-dev/taichi/issues/2084.

Also, try turn off the unified memory with ti.init(use_unified_memory=False) ? https://github.com/taichi-dev/taichi/issues/2466

dwxrycb123 commented 3 years ago

The CUDA version is 9.2, the taichi version is 0.7.17 ti.init(use_unified_memory=False) doesn't work.

The output of ti.diagnose: `[Taichi] mode=release [Taichi] version 0.7.17, llvm 10.0.0, commit 489586af, win, python 3.8.8

Taichi Programming Language

Docs: https://taichi.rtfd.io/zh_CN/latest GitHub: https://github.com/taichi-dev/taichi Forum: https://forum.taichi.graphics

Taichi system diagnose:

python: 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 15:50:08) [MSC v.1916 64 bit (AMD64)] system: win32 executable: d:\anaconda3\envs\cg\python.exe platform: Windows-10-10.0.18362-SP0 architecture: 64bit WindowsPE uname: uname_result(system='Windows', node='LAPTOP-xxxxx', release='10', version='10.0.18362', machine='AMD64', processor='Intel64 Family 6 Model 142 Stepping 10, GenuineIntel') locale: zh_CN.cp936

lsb_release not available: [WinError 2] 系统找不到指定的文件。 TAICHI_REPO_DIR=

import: ERROR list index out of range

cc: ERROR list index out of range cpu: ERROR list index out of range metal: ERROR list index out of range opengl: ERROR list index out of range cuda: ERROR list index out of range

glewinfo not available: [WinError 2] 系统找不到指定的文件。

Sat Jul 03 15:10:42 2021 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 399.24 Driver Version: 399.24 | |-------------------------------+----------------------+----------------------+ | GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 GeForce MX150 WDDM | 00000000:01:00.0 Off | N/A | | N/A 47C P3 N/A / N/A | 64MiB / 2048MiB | 0% Default | +-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 1308 C+G Insufficient Permissions N/A | | 0 2800 C+G ...iles (x86)\Tencent\WeChat\WeChatApp.exe N/A | | 0 9972 C+G ...osoft.LockApp_cw5n1h2txyewy\LockApp.exe N/A | | 0 10252 C+G ...rogram Files\Microsoft VS Code\Code.exe N/A | | 0 10976 C+G ...hell.Experiences.TextInput.InputApp.exe N/A | | 0 11004 C+G ...5n1h2txyewy\StartMenuExperienceHost.exe N/A | | 0 12056 C+G ....143.0_x64__8wekyb3d8bbwe\YourPhone.exe N/A | | 0 14976 C+G ...m Files (x86)\Tencent\WeChat\WeChat.exe N/A | +-----------------------------------------------------------------------------+

[Taichi] mode=release [Taichi] version 0.7.17, llvm 10.0.0, commit 489586af, win, python 3.8.8

[Taichi] mode=release [Taichi] version 0.7.17, llvm 10.0.0, commit 489586af, win, python 3.8.8 [Taichi] Starting on arch=x64

[Taichi] mode=release [Taichi] version 0.7.17, llvm 10.0.0, commit 489586af, win, python 3.8.8 [Taichi] Starting on arch=opengl

[Taichi] mode=release [Taichi] version 0.7.17, llvm 10.0.0, commit 489586af, win, python 3.8.8 [Taichi] Starting on arch=cuda

[Taichi] mode=release [Taichi] version 0.7.17, llvm 10.0.0, commit 489586af, win, python 3.8.8

Taichi Programming Language

Docs: https://taichi.rtfd.io/zh_CN/latest GitHub: https://github.com/taichi-dev/taichi Forum: https://forum.taichi.graphics

Running example minimal ... [Taichi] Starting on arch=x64 [Taichi] materializing...

Running time: 1.20s 42

Consider attaching this log when maintainers ask about system information.

Running time: 21.90s`

The error information: `[Taichi] materializing... [E 07/03/21 15:08:17.729] [taichi/backends/cuda/cuda_driver.h:taichi::lang::CUDADriverFunction<void ,char const ,unsigned int,unsigned int ,void >::operator ()@81] CUDA Error CUDA_ERROR_INVALID_PTX: a PTX JIT compilation failed while calling module_load_data_ex (cuModuleLoadDataEx)

  Taichi Compiler Stack Traceback  0x7ffd0c6fea87: taichi::create_instance_placement in taichi_core.pyd 0x7ffd0c5ef899: taichi::create_instance_placement in taichi_core.pyd 0x7ffd0c8371fb: taichi::create_instance_placement in taichi_core.pyd 0x7ffd0c67da79: taichi::create_instance_placement in taichi_core.pyd 0x7ffd0c68afdf: taichi::create_instance_placement in taichi_core.pyd 0x7ffd0c6e8c47: taichi::create_instance_placement in taichi_core.pyd 0x7ffd0c6c4ee7: taichi::create_instance_placement in taichi_core.pyd 0x7ffd0c62c259: taichi::create_instance_placement in taichi_core.pyd 0x7ffd0c51e1b0: taichi::create_instance_placement in taichi_core.pyd 0x7ffd0c4ae456: taichi::create_instance_placement in taichi_core.pyd 0x7ffd0c4a4607: taichi::create_instance_placement in taichi_core.pyd 0x7ffd17766c80: PyMethodDef_RawFastCallKeywords in python38.dll 0x7ffd17765c46: PyObject_MakeTpCall in python38.dll 0x7ffd17843138: PyEval_GetFuncDesc in python38.dll 0x7ffd1783fcb1: PyEval_EvalFrameDefault in python38.dll 0x7ffd17841d59: PyEval_EvalCodeWithName in python38.dll 0x7ffd1776640f: PyFunction_Vectorcall in python38.dll 0x7ffd17843113: PyEval_GetFuncDesc in python38.dll 0x7ffd1783fcca: PyEval_EvalFrameDefault in python38.dll 0x7ffd17841d59: PyEval_EvalCodeWithName in python38.dll 0x7ffd1776640f: PyFunction_Vectorcall in python38.dll 0x7ffd17768f89: PyMethod_Self in python38.dll 0x7ffd17843113: PyEval_GetFuncDesc in python38.dll 0x7ffd1783fcf1: PyEval_EvalFrameDefault in python38.dll 0x7ffd17841d59: PyEval_EvalCodeWithName in python38.dll 0x7ffd1776640f: PyFunction_Vectorcall in python38.dll 0x7ffd17765ab4: PyObject_FastCallDict in python38.dll 0x7ffd17767070: PyObject_Call_Prepend in python38.dll 0x7ffd177d38ea: PyType_Ready in python38.dll 0x7ffd17765eda: PyObject_Call in python38.dll 0x7ffd178433d1: PyEval_GetFuncDesc in python38.dll 0x7ffd1783fdf1: PyEval_EvalFrameDefault in python38.dll 0x7ffd17841d59: PyEval_EvalCodeWithName in python38.dll 0x7ffd1776640f: PyFunction_Vectorcall in python38.dll 0x7ffd17843113: PyEval_GetFuncDesc in python38.dll 0x7ffd1783dabc: PyEval_EvalFrameDefault in python38.dll 0x7ffd17841d59: PyEval_EvalCodeWithName in python38.dll 0x7ffd178b055f: PyRun_FileExFlags in python38.dll 0x7ffd178b0651: PyRun_FileExFlags in python38.dll 0x7ffd178b0138: PyRun_StringFlags in python38.dll 0x7ffd178ae3d1: PyRun_InteractiveOneFlags in python38.dll 0x7ffd178ae5a0: PyRun_SimpleFileExFlags in python38.dll 0x7ffd176d744d: Py_hashtable_copy in python38.dll 0x7ffd176d80b1: Py_hashtable_copy in python38.dll 0x7ffd176d8fa2: Py_RunMain in python38.dll 0x7ffd176d9016: Py_Main in python38.dll 0x7ff63b3914f8: OPENSSL_Applink in python.exe 0x7ffd800e7c24: BaseThreadInitThunk in KERNEL32.DLL 0x7ffd81c2d721: RtlUserThreadStart in ntdll.dll  Internal error occurred. Check out this page for possible solutions: https://taichi.readthedocs.io/en/stable/install.html#troubleshooting Traceback (most recent call last): File "test.py", line 456, in reset_cloth() File "D:\anaconda3\envs\cg\lib\site-packages\taichi\lang\kernel_impl.py", line 576, in wrapped return primal(*args, *kwargs) File "D:\anaconda3\envs\cg\lib\site-packages\taichi\lang\kernel_impl.py", line 503, in call self.materialize(key=key, args=args, arg_features=arg_features) File "D:\anaconda3\envs\cg\lib\site-packages\taichi\lang\kernel_impl.py", line 304, in materialize self.runtime.materialize() File "D:\anaconda3\envs\cg\lib\site-packages\taichi\lang\impl.py", line 239, in materialize _ti_core.layout(layout) RuntimeError: [taichi/backends/cuda/cuda_driver.h:taichi::lang::CUDADriverFunction<void ,char const ,unsigned int,unsigned int ,void *>::operator ()@81] CUDA Error CUDA_ERROR_INVALID_PTX: a PTX JIT compilation failed while calling module_load_data_ex (cuModuleLoadDataEx)`

k-ye commented 3 years ago

Hmm, Taichi only supports CUDA 10+..

dwxrycb123 commented 3 years ago

Oh I didn't notice that, I'm sorry... I tried with CUDA 11.0 and now it works.

guoyuanhao-gyh commented 1 year ago

Try ti.lang.gpu, maybe it cause by new versions.

dongfeicui commented 1 year ago

I tested on GTX 4080, cuda 11.6 and python 3.8 by using taichi 1.5.0 and taichi 1.4.0. It works.