Error: CUDAError: Check failed: ret == 0 (-1 vs. 0) : cuModuleLoadData(&(module_[device_id]), data_.c_str()) failed with error: CUDA_ERROR_INVALID_PTX

[expectopatronm]

Hi!,

I get this error when I run the example from the Github description.

jetson@jetson:~/fast-depth/deploy$ python3 tx2_run_tvm.py --input-fp data/rgb.npy --output-fp data/pred.npy --model-dir ../results/tvm_compiled/tx2_gpu_mobilenet_nnconv5dw_skipadd_pruned/ --cuda True
=> [TVM on TX2] using model files in ../results/tvm_compiled/tx2_gpu_mobilenet_nnconv5dw_skipadd_pruned/
=> [TVM on TX2] loading model lib and ptx
=> [TVM on TX2] loading model graph and params
=> [TVM on TX2] creating TVM runtime module
=> [TVM on TX2] feeding inputs and params into TVM module
=> [TVM on TX2] running TVM module, saving output
Traceback (most recent call last):

  File "tx2_run_tvm.py", line 91, in <module>
    main()

  File "tx2_run_tvm.py", line 88, in main
    run_model(args.model_dir, args.input_fp, args.output_fp, args.warmup, args.run, args.cuda,  try_randin=args.randin)

  File "tx2_run_tvm.py", line 36, in run_model
    run() # not gmodule.run()

  File "/home/jetson/tvm/python/tvm/_ffi/_ctypes/function.py", line 207, in __call__
    raise get_last_ffi_error()

tvm._ffi.base.TVMError: Traceback (most recent call last):
  [bt] (3) /home/jetson/tvm/build/libtvm.so(TVMFuncCall+0x70) [0x7fad7ccec0]
  [bt] (2) /home/jetson/tvm/build/libtvm.so(std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::detail::PackFuncVoidAddr_<4, tvm::runtime::CUDAWrappedFunc>(tvm::runtime::CUDAWrappedFunc, std::vector<tvm::runtime::detail::ArgConvertCode, std::allocator<tvm::runtime::detail::ArgConvertCode> > const&)::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)+0xe8) [0x7fad850b08]
  [bt] (1) /home/jetson/tvm/build/libtvm.so(tvm::runtime::CUDAWrappedFunc::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*, void**) const+0x6cc) [0x7fad85093c]
  [bt] (0) /home/jetson/tvm/build/libtvm.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x4c) [0x7facfdebac]
  File "/home/jetson/tvm/src/runtime/cuda/cuda_module.cc", line 110
  File "/home/jetson/tvm/src/runtime/library_module.cc", line 91
CUDAError: Check failed: ret == 0 (-1 vs. 0) : cuModuleLoadData(&(module_[device_id]), data_.c_str()) failed with error: CUDA_ERROR_INVALID_PTX

I can't seem to find a solution to this. Could you please help?

[dwofk]

Hi @expectopatronm

This seems like a compilation issue. The compiled models provided in this repo were compiled for the following environment:

NVIDIA Jetson TX2 CPU/GPU CUDA version 8.0 LLVM version 4.0 TVM runtime commit ab4946c8b80da510a5a518dca066d8159473345f

Does this match your environment?

Note that for a different hardware target or a more recent version of TVM, you would need to redo the compilation process to ensure that compiled models are compatible with your environment.

[expectopatronm]

Hi!,

Thank you for the reply. I am using a Jetson Nano with CUDA 10.0, LLVM 4.0 and the latest TVM. I redid the compilation and results in this error.

Regards, Shankar

On Fri, Jan 31, 2020 at 6:59 PM Diana Wofk notifications@github.com wrote:

Hi @expectopatronm https://github.com/expectopatronm

This seems like a compilation issue. The compiled models provided in this repo were compiled for the following environment:

NVIDIA Jetson TX2 CPU/GPU CUDA version 8.0 LLVM version 4.0 TVM runtime commit ab4946c8b80da510a5a518dca066d8159473345f

Does this match your environment?

Note that for a different hardware target or a more recent version of TVM, you would need to redo the compilation process to ensure that compiled models are compatible with your environment.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dwofk/fast-depth/issues/22?email_source=notifications&email_token=AHAJHHONVMHPLZ7VVKFMS53RARRI3A5CNFSM4KNYFVN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKPPCTQ#issuecomment-580841806, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHAJHHKX4RPXJSKHSY3WEZTRARRI3ANCNFSM4KNYFVNQ .

[dwofk]

Hi @expectopatronm

For steps to correctly redo the compilation, visit https://tvm.apache.org/docs/tutorials/index.html#compile-deep-learning-models. The TVM source code has undergone several updates and so the documentation now differs from that which was used in our work.

If you have been encountering further difficulties with compilation using TVM, it would be best to refer to the TVM discussion forum at https://discuss.tvm.ai/.