Closed butterluo closed 9 months ago
Did you buid with DCMAKE_BUILD_TYPE =Debug
or Release
?
Same issue encountered here. CUDA runtime API doc says:
CUDNN_STATUS_MAPPING_ERROR
Mismatch between the user stream and the cuDNN handle context.
But I check the code and add some extra logs.
current_stream
is created right after cudnn_handle_
is created. I can't find out any reason why they are not matched.
Code in HugeCTR/include/gpu_resource.hpp
:
void set_stream(const std::string& name, int priority = 0) {
// Get and create stream here.
cudaStream_t current_stream =
stream_event_manager_.get_stream(name, cudaStreamNonBlocking, priority);
stream_name_ = name;
HCTR_LIB_THROW(curandSetStream(replica_variant_curand_generator_, current_stream));
HCTR_LIB_THROW(cublasSetStream(cublas_handle_, current_stream));
HCTR_LIB_THROW(cudnnSetStream(cudnn_handle_, current_stream));
}
Hi @zpcalan, we have encountered this CUDNN error when building with Debug
mode before 22.05 release, and we have fixed it in 22.05. So I'd like to know how did you build the source codes.
Can you try the latest image(22.06) or build HugeCTR with Release mode with 22.02 release?
Hi @zpcalan, we have encountered this CUDNN error when building with
Debug
mode before 22.05 release, and we have fixed it in 22.05. So I'd like to know how did you build the source codes.Can you try the latest image(22.06) or build HugeCTR with Release mode with 22.02 release?
Do you mean 23.06? I was using 23.02 image and its built-in HugeCTR when encountering this issue. So I did not build source codes. But never mind, I change some hyper parameters like slot array size and workspace size and error disappeared. It could be some gpu memory issue.
Glad to hear that. Out of curiosity, can you help confirm if this error does happen in 23.06 if you don't chagne any hyper-params?
OK, I'll try it later.
Describe the bug When I run samples/wdl/wdl_parquet.py according to its Readme.md, an Error thowen: terminate called after throwing an instance of 'std::_Nested_exception'
what(): Runtime error: CUDNN_STATUS_MAPPING_ERROR
cudnnSetStream(cudnnhandle, current_stream) at set_stream(/home/SRC/MLSys/RecsysAdSrch/Merlin/HugeCTR/HugeCTR/include/gpu_resource.hpp:79)
[b:117125] Process received signal
[b:117125] Signal: Aborted (6)
[b:117125] Signal code: (-6)
[b:117125] [ 0] /usr/lib/x86_64-linux-gnu/libc.so.6(+0x43090)[0x7fb889e50090]
[b:117125] [ 1] /usr/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7fb889e5000b]
[b:117125] [ 2] /usr/lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7fb889e2f859]
[b:117125] [ 3] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x9e911)[0x7fb879f4c911]
[b:117125] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa38c)[0x7fb879f5838c]
[b:117125] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xa9369)[0x7fb879f57369]
[b:117125] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(gxx_personality_v0+0x2a1)[0x7fb879f57d21]
[b:117125] [ 7] /usr/lib/x86_64-linux-gnu/libgcc_s.so.1(+0x10bef)[0x7fb879ea3bef]
[b:117125] [ 8] /usr/lib/x86_64-linux-gnu/libgcc_s.so.1(_Unwind_Resume+0x12a)[0x7fb879ea45aa]
[b:117125] [ 9] /usr/local/hugectr/lib/libhuge_ctr_shared.so(_ZN7HugeCTR11GPUResource10set_streamERKNSt7cxx1112basic_stringIcSt11char_traitsIcESaIcEEEi+0x28e)[0x7fb87b7c8b9c]
[b:117125] [10] /usr/local/hugectr/lib/libhuge_ctr_shared.so(_ZN7HugeCTR13StreamContextD1Ev+0x37)[0x7fb87bd4bfa7]
[b:117125] [11] /usr/local/hugectr/lib/libhuge_ctr_shared.so(_ZN7HugeCTR25StreamContextScheduleable3runESt10shared_ptrINS_11GPUResourceEEbb+0x548)[0x7fb87bd4b2aa]
[b:117125] [12] /usr/local/hugectr/lib/libhuge_ctr_shared.so(+0x1cae798)[0x7fb87bd4b798]
[b:117125] [13] /usr/local/hugectr/lib/libhuge_ctr_shared.so(+0x1caeae2)[0x7fb87bd4bae2]
[b:117125] [14] /usr/local/hugectr/lib/libhuge_ctr_shared.so(_ZNKSt8functionIFvP11CUstreamstEEclES1+0x4d)[0x7fb87b7cbdaf]
[b:117125] [15] /usr/local/hugectr/lib/libhuge_ctr_shared.so(_ZN7HugeCTR12GraphWrapper7captureESt8functionIFvP11CUstreamstEES3+0xe8)[0x7fb87b7cb8dc]
[b:117125] [16] /usr/local/hugectr/lib/libhuge_ctr_shared.so(_ZN7HugeCTR8Pipeline9run_graphEv+0xe7)[0x7fb87bd4b8d9]
[b:117125] [17] /usr/local/hugectr/lib/libhuge_ctr_shared.so(+0x1d73666)[0x7fb87be10666]
[b:117125] [18] /usr/lib/x86_64-linux-gnu/libgomp.so.1(GOMP_parallel+0x46)[0x7fb8086038e6]
[b:117125] [19] /usr/local/hugectr/lib/libhuge_ctr_shared.so(_ZN7HugeCTR5Model5trainEb+0x500)[0x7fb87be05fa0]
[b:117125] [20] /usr/local/hugectr/lib/libhuge_ctr_shared.so(_ZN7HugeCTR5Model3fitEiiiiiNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x26f5)[0x7fb87be03df3]
[b:117125] [21] /usr/local/hugectr/lib/hugectr.so(+0x3e42a3)[0x7fb8895d22a3]
[b:117125] [22] /usr/local/hugectr/lib/hugectr.so(+0x4cbc73)[0x7fb8896b9c73]
[b:117125] [23] /usr/local/hugectr/lib/hugectr.so(+0x48873e)[0x7fb88967673e]
[b:117125] [24] /usr/local/hugectr/lib/hugectr.so(+0x433e38)[0x7fb889621e38]
[b:117125] [25] /usr/local/hugectr/lib/hugectr.so(+0x43420e)[0x7fb88962220e]
[b:117125] [26] /usr/local/hugectr/lib/hugectr.so(+0x35e190)[0x7fb88954c190]
[b:117125] [27] python3(PyCFunction_Call+0x59)[0x5f5b39]
[b:117125] [28] python3(_PyObject_MakeTpCall+0x296)[0x5f6706]
[b:117125] [29] python3[0x50ba83]
[b:117125] End of error message
To Reproduce Steps to reproduce the behavior:
Expected behavior A clear and concise description of what you expected to happen.
Screenshots If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
Additional context Add any other context about the problem here. Below are the call stack when ERROR happen. I found that the cudnnSetStream() will be call many times, but only throws ERROR when capturing Graph. CallStack: libc.so.6!raise (Unknown Source:0) libc.so.6!abort (Unknown Source:0) libstdc++.so.6![Unknown/Just-In-Time compiled code] (Unknown Source:0) libstdc++.so.6!gxx_personality_v0 (Unknown Source:0) libgcc_s.so.1![Unknown/Just-In-Time compiled code] (Unknown Source:0) libgcc_s.so.1!_Unwind_Resume (Unknown Source:0) libhuge_ctr_shared.so!HugeCTR::GPUResource::set_stream(HugeCTR::GPUResource const this, const std::string & name, int priority) (/home/SRC/MLSys/RecsysAdSrch/Merlin/HugeCTR/HugeCTR/include/gpu_resource.hpp:79) libhuge_ctr_shared.so!HugeCTR::StreamContext::~StreamContext(HugeCTR::StreamContext const this) (/home/SRC/MLSys/RecsysAdSrch/Merlin/HugeCTR/HugeCTR/include/gpu_resource.hpp:114) libhuge_ctr_shared.so!HugeCTR::StreamContextScheduleable::run(HugeCTR::StreamContextScheduleable const this, std::shared_ptr gpu, bool use_graph, bool prepare_resource) (/home/SRC/MLSys/RecsysAdSrch/Merlin/HugeCTR/HugeCTR/src/pipeline.cpp:72)
libhuge_ctr_shared.so!HugeCTR::Pipeline::<lambda(cudaStream_t)>::operator()(cudaStream_t) const(const HugeCTR::Pipeline::<lambda(cudaStream_t)> const closure) (/home/SRC/MLSys/RecsysAdSrch/Merlin/HugeCTR/HugeCTR/src/pipeline.cpp:112)
libhuge_ctr_shared.so!std::_Function_handler<void(CUstream_st), HugeCTR::Pipeline::run_graph()::<lambda(cudaStream_t)> >::_M_invoke(const std::_Any_data &, CUstream_st &&)(const std::_Any_data & functor, args#0) (/usr/include/c++/9/bits/std_function.h:300)
libhuge_ctr_shared.so!std::function<void (CUstream_st)>::operator()(CUstream_st) const(const std::function<void(CUstream_st)> const this, args#0) (/usr/include/c++/9/bits/std_function.h:688)
libhuge_ctr_shared.so!HugeCTR::GraphWrapper::capture(std::function<void (CUstream_st)>, CUstream_st)(HugeCTR::GraphWrapper const this, std::function<void(CUstream_st)> workload, cudaStream_t stream) (/home/SRC/MLSys/RecsysAdSrch/Merlin/HugeCTR/HugeCTR/src/graph_wrapper.cpp:31)
libhuge_ctr_shared.so!HugeCTR::Pipeline::run_graph(HugeCTR::Pipeline const this) (/home/SRC/MLSys/RecsysAdSrch/Merlin/HugeCTR/HugeCTR/src/pipeline.cpp:118)
libhuge_ctr_shared.so!HugeCTR::Model::_ZN7HugeCTR5Model5trainEb._omp_fn.1(void)() (/home/SRC/MLSys/RecsysAdSrch/Merlin/HugeCTR/HugeCTR/src/pybind/model.cpp:2393)
libgomp.so.1!GOMP_parallel (Unknown Source:0)
libhuge_ctr_shared.so!HugeCTR::Model::train(HugeCTR::Model const this, bool is_first_batch) (/home/SRC/MLSys/RecsysAdSrch/Merlin/HugeCTR/HugeCTR/src/pybind/model.cpp:2387)
libhuge_ctr_shared.so!HugeCTR::Model::fit(HugeCTR::Model * const this, int num_epochs, int max_iter, int display, int eval_interval, int snapshot, std::string snapshot_prefix) (/home/SRC/MLSys/RecsysAdSrch/Merlin/HugeCTR/HugeCTR/src/pybind/model.cpp:2185)
hugectr.so!pybind11::cpp_function::cpp_function<void, HugeCTR::Model, int, int, int, int, int, std::cxx11::basic_string<char, std::char_traits, std::allocator >, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(void (HugeCTR::Model::)(int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(HugeCTR::Model , int, int, int, int, int, std::cxx11::basic_string<char, std::char_traits, std::allocator >)#1}::operator()(HugeCTR::Model*, int, int, int, int, int, std:: cxx11::basic_string<char, std::char_traits, std::allocator >) const(const pybind11::cpp_function::<lambda(HugeCTR::Model, int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >)> const this, HugeCTR::Model c, args#0, args#1, args#2, args#3, args#4, args#5) (/home/SRC/MLSys/RecsysAdSrch/Merlin/HugeCTR/build/_deps/pybind11_sources-src/include/pybind11/pybind11.h:110)
hugectr.so!pybind11::detail::argument_loader<HugeCTR::Model, int, int, int, int, int, std::cxx11::basic_string<char, std::char_traits, std::allocator > >::call_impl<void, pybind11::cpp_function::cpp_function<void, HugeCTR::Model, int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(void (HugeCTR::Model::*)(int, int, int, int, int, std:: cxx11::basic_string<char, std::char_traits, std::allocator >), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(HugeCTR::Model, int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >)#1}&, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, pybind11::detail::void_type>(pybind11::cpp_function::cpp_function<void, HugeCTR::Model, int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(void (HugeCTR::Model:: )(int, int, int, int, int, std::cxx11::basic_string<char, std::char_traits, std::allocator >), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(HugeCTR::Model, int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >)#1}&, std::integer_sequence<unsigned long, 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul>, pybind11::detail::void_type&&) &&(pybind11::detail::argument_loader<HugeCTR::Model , int, int, int, int, int, std:: cxx11::basic_string<char, std::char_traits, std::allocator > > const this, pybind11::cpp_function::<lambda(HugeCTR::Model, int, int, int, int, int, std::cxx11::basic_string<char, std::char_traits, std::allocator >)> & f) (/home/SRC/MLSys/RecsysAdSrch/Merlin/HugeCTR/build/_deps/pybind11_sources-src/include/pybind11/cast.h:1448)
hugectr.so!pybind11::detail::argument_loader<HugeCTR::Model*, int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >::call<void, pybind11::detail::void_type, pybind11::cpp_function::cpp_function<void, HugeCTR::Model, int, int, int, int, int, std:: cxx11::basic_string<char, std::char_traits, std::allocator >, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(void (HugeCTR::Model::)(int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(HugeCTR::Model , int, int, int, int, int, std::cxx11::basic_string<char, std::char_traits, std::allocator >)#1}&>(pybind11::cpp_function::cpp_function<void, HugeCTR::Model, int, int, int, int, int, std:: cxx11::basic_string<char, std::char_traits, std::allocator >, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(void (HugeCTR::Model::)(int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(HugeCTR::Model , int, int, int, int, int, std::cxx11::basic_string<char, std::char_traits, std::allocator >)#1}&) &&(pybind11::detail::argument_loader<HugeCTR::Model, int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > const this, pybind11::cpp_function::<lambda(HugeCTR::Model*, int, int, int, int, int, std:: cxx11::basic_string<char, std::char_traits, std::allocator >)> & f) (/home/SRC/MLSys/RecsysAdSrch/Merlin/HugeCTR/build/_deps/pybind11_sources-src/include/pybind11/cast.h:1422)
hugectr.so!pybind11::cpp_function::initialize<pybind11::cpp_function::initialize<void, HugeCTR::Model, int, int, int, int, int, std::cxx11::basic_string<char, std::char_traits, std::allocator >, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(void (HugeCTR::Model::)(int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(HugeCTR::Model , int, int, int, int, int, std:: cxx11::basic_string<char, std::char_traits, std::allocator >)#1}, void, HugeCTR::Model, int, int, int, int, int, std::cxx11::basic_string<char, std::char_traits, std::allocator >, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(pybind11::cpp_function::initialize<void, HugeCTR::Model, int, int, int, int, int, std:: cxx11::basic_string<char, std::char_traits, std::allocator >, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(void (HugeCTR::Model:: )(int, int, int, int, int, std::cxx11::basic_string<char, std::char_traits, std::allocator >), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(HugeCTR::Model, int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >)#1}&&, void ( )(HugeCTR::Model, int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::operator()(pybind11::detail::function_call&) const(const pybind11::cpp_function::<lambda(pybind11::detail::function_call&)> const this, pybind11::detail::function_call & call) (/home/SRC/MLSys/RecsysAdSrch/Merlin/HugeCTR/build/_deps/pybind11_sources-src/include/pybind11/pybind11.h:248)
hugectr.so!pybind11::cpp_function::initialize<pybind11::cpp_function::initialize<void, HugeCTR::Model, int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(void (HugeCTR::Model::*)(int, int, int, int, int, std:: cxx11::basic_string<char, std::char_traits, std::allocator >), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(HugeCTR::Model, int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >)#1}, void, HugeCTR::Model , int, int, int, int, int, std::cxx11::basic_string<char, std::char_traits, std::allocator >, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(pybind11::cpp_function::initialize<void, HugeCTR::Model, int, int, int, int, int, std:: cxx11::basic_string<char, std::char_traits, std::allocator >, pybind11::name, pybind11::is_method, pybind11::sibling, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v, pybind11::arg_v>(void (HugeCTR::Model::)(int, int, int, int, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(HugeCTR::Model , int, int, int, int, int, std::cxx11::basic_string<char, std::char_traits, std::allocator >)#1}&&, void ()(HugeCTR::Model, int, int, int, int, int, std:: cxx11::basic_string<char, std::char_traits, std::allocator >), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&)() (/home/SRC/MLSys/RecsysAdSrch/Merlin/HugeCTR/build/_deps/pybind11_sources-src/include/pybind11/pybind11.h:223)
hugectr.so!pybind11::cpp_function::dispatcher(PyObject self, PyObject args_in, PyObject * kwargs_in) (/home/SRC/MLSys/RecsysAdSrch/Merlin/HugeCTR/build/_deps/pybind11_sources-src/include/pybind11/pybind11.h:939)
PyCFunction_Call (Unknown Source:0)
_PyObject_MakeTpCall (Unknown Source:0)
[Unknown/Just-In-Time compiled code] (Unknown Source:0)
_PyEval_EvalFrameDefault (Unknown Source:0)
_PyEval_EvalCodeWithName (Unknown Source:0)
PyEval_EvalCode (Unknown Source:0)
[Unknown/Just-In-Time compiled code] (Unknown Source:0)
PyRun_SimpleFileExFlags (Unknown Source:0)
Py_RunMain (Unknown Source:0)
Py_BytesMain (Unknown Source:0)
libc.so.6!__libc_start_main (Unknown Source:0)