search

k2-fsa / sherpa-onnx

Speech-to-text, text-to-speech, speaker diarization, and VAD using next-gen Kaldi with onnxruntime without Internet connection. Support embedded systems, Android, iOS, Raspberry Pi, RISC-V, x86_64 servers, websocket server/client, C/C++, Python, Kotlin, C#, Go, NodeJS, Java, Swift, Dart, JavaScript, Flutter, Object Pascal, Lazarus, Rust
https://k2-fsa.github.io/sherpa/onnx/index.html
Apache License 2.0
3.62k stars 424 forks source link

Whisper medium error with DirectML #1240

Open thewh1teagle opened 3 months ago

thewh1teagle commented 3 months ago

When running whisper medium (int8 / normal) on DirectML provider it failed. on CPU it works. same build.

The error

Non-zero status code returned while running MemcpyToHost node

Reproduce

cmake -B build -DCMAKE_BUILD_TYPE=Release -DSHERPA_ONNX_ENABLE_DIRECTML=ON -DSHERPA_ONNX_ENABLE_PORTAUDIO=OFF -DSHERPA_ONNX_ENABLE_WEBSOCKET=OFF -DSHERPA_ONNX_ENABLE_TTS=OFF -DSHERPA_ONNX_ENABLE_BINARY=ON -DSHERPA_ONNX_BUILD_C_API_EXAMPLES=ON .

Change provider to DirectML in whiser-c-api and medium.int8

cmake --build build --config Release --target whisper-c-api

Result (failed)

Time taken for create recognizer: 13.49 seconds
2024-08-09 17:22:45.9754547 [E:onnxruntime:, sequential_executor.cc:494 onnxruntime::ExecuteKernel] Non-zero status code returned while running MemcpyToHost node. Name:'Memcpy_token_244' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2429)\onnxruntime.dll!00007FF9B54FE1D6: (caller: 00007FF9B5A25FC7) Exception(3) tid(2f08) 887A0006 The GPU will not respond to more commands, most likely because of an invalid command passed by the calling application.

D:\sherpa\sherpa-onnx\sherpa-onnx/csrc/offline-recognizer-whisper-impl.h:DecodeStream:176 

Caught exception:

Non-zero status code returned while running MemcpyToHost node. Name:'Memcpy_token_244' Status Message: D:\a\_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\MLOperatorAuthorImpl.cpp(2429)\onnxruntime.dll!00007FF9B54FE1D6: (caller: 00007FF9B5A25FC7) Exception(3) tid(2f08) 887A0006 The GPU will not respond to more commands, most likely because of an invalid command passed by the calling application.

Return an empty result. Number of input frames: 663, Current tail paddings: 1000. If you see a lot of such exceptions, please consider using a larger --whisper-tail-paddings
Decoded text:
Time taken: 2.83 seconds
Decoded text:

Change provider to CPU in whisper-c-api and medium.int8

cmake --build build --config Release --target whisper-c-api

Result (success)

Time taken for create recognizer: 10.05 seconds
Decoded text:  After early nightfall the yellow lamps would light up here and there the squalid quarter of the brothels.
Time taken: 15.26 seconds
Decoded text:  After early nightfall the yellow lamps would light up here and there the squalid quarter of the brothels.

Change provider to DirectML and model to tiny.int8

Result (success) 

Time taken for create recognizer: 1.39 seconds
Decoded text:  After early nightfall, the yellow lamps would light up here and there the squalid quarter of the brothels.
Time taken: 2.25 seconds
Decoded text:  After early nightfall, the yellow lamps would light up here and there the squalid quarter of the brothels.

Maybe related: https://github.com/dog-qiuqiu/Ultralight-SimplePose/issues/5 https://github.com/microsoft/onnxruntime/issues/3205 https://github.com/k2-fsa/sherpa-onnx/pull/1037

By the way I can that it loads some of the model into the GPU memory. like 2GB and then fail (I have 8GB GPU)

Happens also in test whisper script

Maybe a fix: https://github.com/onnx/models/issues/156#issuecomment-690847276

thewh1teagle commented 2 months ago

Turns out it happens with scripts/whisper/test.py as well. I created new issue in onnxruntime https://github.com/microsoft/onnxruntime/issues/21805