search

apache / tvm

Open deep learning compiler stack for cpu, gpu and specialized accelerators
https://tvm.apache.org/
Apache License 2.0
11.75k stars 3.47k forks source link

NVIDIA GPU inference on TVM Docker #8232

Closed YongtaoHuang1994 closed 3 years ago

YongtaoHuang1994 commented 3 years ago

OS: ubuntu 18.04 NVIDIA GPU: GM107GL [Quadro K620]

First, the inference code can run in the on the physical machine (no docker env)

(tvm-build) root@hyongtao-Precision-Tower-5810:/home/hyongtao/build_docker/tvm# python3 tune_relay_cuda_test.py
Extract tasks...
One or more operators have not been tuned. Please tune your model for better performance. Use DEBUG logging level to see more details.
Evaluate inference time cost...
Mean inference time (std dev): 11.10 ms (0.34 ms)
(tvm-build) root@hyongtao-Precision-Tower-5810:/home/hyongtao/build_docker/tvm#

Then, we wanna test TVM in docker environment. I have build the docker as https://github.com/apache/tvm/tree/main/docker

$ ./docker/build.sh ci_gpu make -j$(nproc)
$ ./docker/bash.sh ci_gpu

Docker building and runing are both successful.

But when I infer the resnet-18 inside the docker 'ci_gpu', I got these error:

root@hyongtao-Precision-Tower-5810:~# python3 tune_relay_cuda_test.py
Extract tasks...
Traceback (most recent call last):
  File "tune_relay_cuda_test.py", line 91, in <module>
    tune_and_evaluate()
  File "tune_relay_cuda_test.py", line 72, in tune_and_evaluate
    lib = relay.build_module.build(mod, target=target, params=params)
  File "/workspace/python/tvm/relay/build_module.py", line 333, in build
    mod=ir_mod, target=target, params=params, executor=executor
  File "/workspace/python/tvm/relay/build_module.py", line 148, in build
    self._build(mod, target, target_host, executor)
  File "/workspace/python/tvm/_ffi/_ctypes/packed_func.py", line 237, in __call__
    raise get_last_ffi_error()
tvm._ffi.base.TVMError: Traceback (most recent call last):
  53: TVMFuncCall
  52: std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::relay::backend::RelayBuildModule::GetFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tvm::runtime::ObjectPtr<tvm::runtime::Object> const&)::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#3}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)
  51: tvm::relay::backend::RelayBuildModule::BuildRelay(tvm::IRModule, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tvm::runtime::NDArray, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, tvm::runtime::NDArray> > > const&)
  50: tvm::relay::backend::RelayBuildModule::Optimize(tvm::IRModule, tvm::runtime::Map<tvm::Integer, tvm::Target, void, void> const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tvm::runtime::NDArray, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, tvm::runtime::NDArray> > > const&)
  49: tvm::transform::Pass::operator()(tvm::IRModule) const
  48: tvm::transform::Pass::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
  47: tvm::transform::SequentialNode::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
  46: tvm::transform::Pass::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
  45: tvm::relay::transform::FunctionPassNode::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
  44: std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::TypedPackedFunc<tvm::relay::Function (tvm::relay::Function, tvm::IRModule, tvm::transform::PassContext)>::AssignTypedLambda<tvm::relay::transform::DynamicToStatic()::{lambda(tvm::relay::Function, tvm::IRModule, tvm::transform::PassContext)#1}>(tvm::relay::transform::DynamicToStatic()::{lambda(tvm::relay::Function, tvm::IRModule, tvm::transform::PassContext)#1})::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)
  43: tvm::relay::DynamicToStatic(tvm::relay::Function, tvm::IRModule)
  42: tvm::relay::DynamicToStaticMutator::PrepareInput(tvm::RelayExpr const&)
  41: tvm::transform::Pass::operator()(tvm::IRModule) const
  40: tvm::transform::Pass::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
  39: tvm::relay::transform::FunctionPassNode::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
  38: std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::TypedPackedFunc<tvm::relay::Function (tvm::relay::Function, tvm::IRModule, tvm::transform::PassContext)>::AssignTypedLambda<tvm::relay::transform::FoldConstant()::{lambda(tvm::relay::Function, tvm::IRModule, tvm::transform::PassContext)#1}>(tvm::relay::transform::FoldConstant()::{lambda(tvm::relay::Function, tvm::IRModule, tvm::transform::PassContext)#1})::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)
  37: tvm::relay::FoldConstant(tvm::RelayExpr const&, tvm::IRModule const&)
  36: tvm::relay::MixedModeMutator::VisitExpr(tvm::RelayExpr const&)
  35: tvm::relay::MixedModeMutator::VisitLeaf(tvm::RelayExpr const&)
  34: _ZN3tvm5relay16MixedModeMutator17DispatchVisitExprERKNS_9Re
  33: tvm::relay::ExprMutator::VisitExpr(tvm::RelayExpr const&)
  32: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::VisitExpr(tvm::RelayExpr const&)
  31: _ZZN3tvm5relay11ExprFunctorIFNS_9RelayExprERKS2_EE10InitVTableEvENUlR
  30: tvm::relay::ConstantFolder::VisitExpr_(tvm::relay::FunctionNode const*)
  29: tvm::relay::ExprMutator::VisitExpr_(tvm::relay::FunctionNode const*)
  28: tvm::relay::MixedModeMutator::VisitExpr(tvm::RelayExpr const&)
  27: tvm::relay::MixedModeMutator::VisitLeaf(tvm::RelayExpr const&)
  26: _ZN3tvm5relay16MixedModeMutator17DispatchVisitExprERKNS_9Re
  25: tvm::relay::ExprMutator::VisitExpr(tvm::RelayExpr const&)
  24: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::VisitExpr(tvm::RelayExpr const&)
  23: _ZZN3tvm5relay11ExprFunctorIFNS_9RelayExprERKS2_EE10InitVTableEvENUlR
  22: tvm::relay::MixedModeMutator::VisitExpr_(tvm::relay::CallNode const*)
  21: tvm::relay::ConstantFolder::Rewrite_(tvm::relay::CallNode const*, tvm::RelayExpr const&)
  20: tvm::relay::ConstantFolder::ConstEvaluate(tvm::RelayExpr)
  19: std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::TypedPackedFunc<tvm::runtime::ObjectRef (tvm::RelayExpr)>::AssignTypedLambda<tvm::relay::CreateInterpreter(tvm::IRModule, DLDevice, tvm::Target)::{lambda(tvm::RelayExpr)#1}>(tvm::relay::CreateInterpreter(tvm::IRModule, DLDevice, tvm::Target)::{lambda(tvm::RelayExpr)#1})::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)
  18: tvm::relay::ExprFunctor<tvm::runtime::ObjectRef (tvm::RelayExpr const&)>::VisitExpr(tvm::RelayExpr const&)
  17: _ZZN3tvm5relay11ExprFunctorIFNS_7runtime9ObjectRefERKNS_9RelayExprEEE
  16: tvm::relay::Interpreter::VisitExpr_(tvm::relay::LetNode const*)
  15: tvm::relay::ExprFunctor<tvm::runtime::ObjectRef (tvm::RelayExpr const&)>::VisitExpr(tvm::RelayExpr const&)
  14: _ZZN3tvm5relay11ExprFunctorIFNS_7runtime9ObjectRefERKNS_9RelayExprEEE
  13: tvm::relay::Interpreter::VisitExpr_(tvm::relay::LetNode const*)
  12: tvm::relay::ExprFunctor<tvm::runtime::ObjectRef (tvm::RelayExpr const&)>::VisitExpr(tvm::RelayExpr const&)
  11: _ZZN3tvm5relay11ExprFunctorIFNS_7runtime9ObjectRefERKNS_9RelayExprEEE
  10: tvm::relay::Interpreter::VisitExpr_(tvm::relay::LetNode const*)
  9: tvm::relay::ExprFunctor<tvm::runtime::ObjectRef (tvm::RelayExpr const&)>::VisitExpr(tvm::RelayExpr const&)
  8: _ZZN3tvm5relay11ExprFunctorIFNS_7runtime9ObjectRefERKNS_9RelayExprEEE
  7: tvm::relay::Interpreter::VisitExpr_(tvm::relay::LetNode const*)
  6: tvm::relay::ExprFunctor<tvm::runtime::ObjectRef (tvm::RelayExpr const&)>::VisitExpr(tvm::RelayExpr const&)
  5: _ZZN3tvm5relay11ExprFunctorIFNS_7runtime9ObjectRefERKNS_9RelayExprEEE
  4: tvm::relay::Interpreter::VisitExpr_(tvm::relay::CallNode const*)
  3: tvm::relay::Interpreter::Invoke(tvm::relay::InterpreterClosure const&, tvm::runtime::Array<tvm::runtime::ObjectRef, void> const&, tvm::relay::Var const&)
  2: tvm::relay::Interpreter::InvokePrimitiveOp(tvm::relay::Function const&, tvm::runtime::Array<tvm::runtime::ObjectRef, void> const&)
  1: tvm::relay::CompileEngineImpl::JIT(tvm::relay::CCacheKey const&)
  0: std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), TVMFuncCreateFromCFunc::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#2}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)
  File "/workspace/python/tvm/_ffi/_ctypes/packed_func.py", line 81, in cfun
    rv = local_pyfunc(*pyargs)
  File "/workspace/python/tvm/relay/backend/_backend.py", line 84, in build
    return tvm.driver.build(mod, target=target)
  File "/workspace/python/tvm/driver/build_module.py", line 473, in build
    rt_mod_host = codegen.build_module(mod_host_all, target_host)
  File "/workspace/python/tvm/target/codegen.py", line 39, in build_module
    return _ffi_api.Build(mod, target)
  File "/workspace/python/tvm/_ffi/_ctypes/packed_func.py", line 237, in __call__
    raise get_last_ffi_error()
  2: TVMFuncCall
  1: tvm::runtime::TypedPackedFunc<tvm::runtime::Module (tvm::IRModule, tvm::Target)>::AssignTypedLambda<tvm::runtime::Module (*)(tvm::IRModule, tvm::Target)>(tvm::runtime::Module (*)(tvm::IRModule, tvm::Target), std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}::operator()(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*) const
  0: tvm::codegen::Build(tvm::IRModule, tvm::Target)
  File "/workspace/src/target/codegen.cc", line 53
TVMError:
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
  Check failed: (bf != nullptr) is false: target.build.llvm is not enabled
root@hyongtao-Precision-Tower-5810:~#

Could you help me solve the problem? Thanks a lot.

vinx13 commented 3 years ago

You need to enable llvm when compiling it. See https://github.com/apache/tvm/blob/main/cmake/config.cmake#L122