NVIDIA / NeMo

A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)
https://docs.nvidia.com/nemo-framework/user-guide/latest/overview.html
Apache License 2.0
11.83k stars 2.46k forks source link

Quartznet onnx model can't run on onnxruntime(CUDA backend) benchmark #8160

Closed wwiwush closed 8 months ago

wwiwush commented 9 months ago

The develop ENV is as follows: cuda 11.8 cudnn 8.2.4 onnxruntime(cuda backend) 1.16.3

When I run my exported Quartznet onnx model on onnxruntime(CUDA backend) benchmark, the following error occurs: [E:onnxruntime:benchmark, cuda_call.cc:116 CudaCall] CUDNN failure 3: CUDNN_STATUS_BAD_PARAM ; GPU=0 ; file=/onnxruntime_src/onnxruntime/contrib_ops/cuda/fusedconv.cc ; line=85 ; expr=cudnnAddTensor(cudnnHandle, &alpha, Base::s.ztensor, Base::s.zdata, &alpha, Base::s.ytensor, Base::s.y_data) [E:onnxruntime:, sequential_executor.cc:514 ExecuteKernel] Non-zero status code returned while running FusedConv node. Name:'Conv_41' Status Message: CUDNN failure 3: CUDNN_STATUS_BAD_PARAM ; GPU=0 ;file=/onnxruntime_src/onnxruntime/contrib_ops/cuda/fusedconv.cc ; line=85 ; expr=cudnnAddTensor(cudnnHandle, &alpha, Base::s.ztensor, Base::s.zdata, &alpha, Base::s.ytensor, Base::s.y_data); terminate called after throwing an instance of 'Ort::Exception' what(): Non-zero status code returned while running FusedConv node. Name:'Conv_41' Status Message: CUDNN failure 3: CUDNN_STATUS_BAD_PARAM ; GPU=0 ;file=/onnxruntime_src/onnxruntime/contrib_ops/cuda/fusedconv.cc ; line=85 ; expr=cudnnAddTensor(cudnnHandle, &alpha, Base::s.ztensor, Base::s.zdata, &alpha, Base::s.ytensor, Base::s.y_data);

Thank you very much for help!

github-actions[bot] commented 8 months ago

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

github-actions[bot] commented 8 months ago

This issue was closed because it has been inactive for 7 days since being marked as stale.