search

PaddlePaddle / PaddleSpeech

Easy-to-use Speech Toolkit including Self-Supervised Learning model, SOTA/Streaming ASR with punctuation, Streaming TTS with text frontend, Speaker Verification System, End-to-End Speech Translation and Keyword Spotting. Won NAACL2022 Best Demo Award.
https://paddlespeech.readthedocs.io
Apache License 2.0
11.19k stars 1.86k forks source link

[Hint: 'CUFFT_INTERNAL_ERROR'. Driver or internal cuFFT library error] 多卡时指定非0卡报错 #3420

Open PC-god opened 1 year ago

PC-god commented 1 year ago

paddle-bfloat 0.1.7 paddle2onnx 1.0.6 paddleaudio 1.1.0 paddlefsl 1.1.0 paddlenlp 2.5.2 paddlepaddle-gpu 2.5.0.post117 paddleslim 2.4.1 paddlespeech 0.0.0 paddlespeech-ctcdecoders 0.2.1 paddlespeech-feat 0.1.0

问题描述： 非流式ASR服务启动； 当服务器上有多张卡时，不修改默认参数即使用paddlespeech/server/conf/applicationdiy.yaml，是运行再cpu上； 当指定gpu:3 时，起服务时在3号卡上；当推理时调用到paddlespeech/audio/transform/transformation.py 下 Transformation的 call方法时，会调用到paddlespeech/audio/transform/spectrogram.py 下LogMelSpectrogramKaldi的 call方法， 最终调用到kaldi.fbank时，会自动挂在0号上，导致报错_，而且即使在前面指定 paddle.device.set_device('gpu:3') 或者 os.environ["CUDA_VISIBLE_DEVICES"] = "3" 或者 启动程序前 export CUDA_VISIBLE_DEVICES=3 都是一样报错。

解决办法： paddlespeech/cli/asr/infer.py 下preprocess方法中， 



``` paddle.device.set_device('cpu') # 尝试fft不报错，在cpu上跑
    audio = preprocessing(audio, **preprocess_args)
    paddle.device.set_device('gpu:3')

在“audio = preprocessing(audio, **preprocess_args)” 前后分别指定位置——计算fft在cpu上，然后再指定回gpu上。
解决
stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.