train.py PreconditionNotMetError: warp-ctc [version 2] Error in get_workspace_size: execution failed 报错

[DannyWang920]

您好，我跟着您发布在Bilibili的视频一步步做下来，发现我在运行train.py时会报错。 F:\software\CONDA\envs\TS\python.exe E:/ASR/PaddlePaddle-DeepSpeech-master/train.py ----------- Configuration Arguments ----------- augment_conf_path: None batch_size: 16 learning_rate: 0.0005 max_duration: 20.0 mean_std_path: ./dataset/mean_std.npz min_duration: 0.5 num_conv_layers: 2 num_epoch: 50 num_rnn_layers: 3 output_model_dir: ./models/param pretrained_model: None resume_model: None rnn_layer_size: 1024 shuffle_method: batch_shuffle_clipped test_manifest: ./dataset/manifest.test test_off: False train_manifest: ./dataset/manifest.train use_gpu: True vocab_path: ./dataset/zh_vocab.txt

[2023-07-05 13:00:45.732899] 训练数据数量：13362

[2023-07-05 13:00:45.748826] 测试数据数量：26

W0705 13:00:46.269068 12752 device_context.cc:404] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 12.2, Runtime API Version: 10.2 W0705 13:00:46.284031 12752 device_context.cc:422] device: 0, cuDNN Version: 7.6. Traceback (most recent call last): File "E:/ASR/PaddlePaddle-DeepSpeech-master/train.py", line 105, in main() File "E:/ASR/PaddlePaddle-DeepSpeech-master/train.py", line 101, in main train() File "E:/ASR/PaddlePaddle-DeepSpeech-master/train.py", line 96, in train test_off=args.test_off) File "E:\ASR\PaddlePaddle-DeepSpeech-master\model_utils\model.py", line 269, in train fetch = exe.run(program=train_compiled_prog, fetch_list=[ctc_loss.name], return_numpy=False) File "F:\software\CONDA\envs\TS\lib\site-packages\paddle\fluid\executor.py", line 1110, in run six.reraise(*sys.exc_info()) File "F:\software\CONDA\envs\TS\lib\site-packages\six.py", line 719, in reraise raise value File "F:\software\CONDA\envs\TS\lib\site-packages\paddle\fluid\executor.py", line 1108, in run return_merged=return_merged) File "F:\software\CONDA\envs\TS\lib\site-packages\paddle\fluid\executor.py", line 1252, in _run_impl return_merged=return_merged) File "F:\software\CONDA\envs\TS\lib\site-packages\paddle\fluid\executor.py", line 913, in _run_parallel tensors = exe.run(fetch_var_names, return_merged)._move_to_list() RuntimeError: In user code:

File "E:/ASR/PaddlePaddle-DeepSpeech-master/train.py", line 105, in <module>
  main()
File "E:/ASR/PaddlePaddle-DeepSpeech-master/train.py", line 101, in main
  train()
File "E:/ASR/PaddlePaddle-DeepSpeech-master/train.py", line 96, in train
  test_off=args.test_off)
File "E:\ASR\PaddlePaddle-DeepSpeech-master\model_utils\model.py", line 220, in train
  train_reader, _, ctc_loss = self.create_network()
File "E:\ASR\PaddlePaddle-DeepSpeech-master\model_utils\model.py", line 138, in create_network
  blank=self._blank)
File "E:\ASR\PaddlePaddle-DeepSpeech-master\model_utils\network.py", line 200, in deep_speech_v2_network
  reduction='sum', input_lengths=None, label_lengths=None)
File "F:\software\CONDA\envs\TS\lib\site-packages\paddle\nn\functional\loss.py", line 1088, in ctc_loss
  input_lengths, label_lengths)
File "F:\software\CONDA\envs\TS\lib\site-packages\paddle\fluid\layers\loss.py", line 634, in warpctc
  'norm_by_times': norm_by_times,
File "F:\software\CONDA\envs\TS\lib\site-packages\paddle\fluid\layer_helper.py", line 43, in append_op
  return self.main_program.current_block().append_op(*args, **kwargs)
File "F:\software\CONDA\envs\TS\lib\site-packages\paddle\fluid\framework.py", line 2967, in append_op
  attrs=kwargs.get("attrs", None))
File "F:\software\CONDA\envs\TS\lib\site-packages\paddle\fluid\framework.py", line 2039, in __init__
  for frame in traceback.extract_stack():

PreconditionNotMetError: warp-ctc [version 2] Error in get_workspace_size: execution failed
  [Hint: Expected CTC_STATUS_SUCCESS == status, but received CTC_STATUS_SUCCESS:0 != status:3.] (at C:\home\workspace\Paddle_release\paddle/fluid/operators/warpctc_op.h:154)
  [operator < warpctc > error]

Process finished with exit code -1073740791 (0xC0000409) 请问该如何解决呢 望回复

[DannyWang920]

我的paddlepaddle是2.1.3版本

[yeyupiaoling]

你用PPASR这个项目吧，这个项目也有这个模型。

[yeyupiaoling]

你跟我的版本一样试一下，2.2.0，paddlepaddle本身是正常的吗？

[DannyWang920]

W0705 15:02:46.374073 3632 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 12.2, Runtime API Version: 10.2 W0705 15:02:46.408979 3632 device_context.cc:465] device: 0, cuDNN Version: 7.6. W0705 15:04:18.495910 3632 operator.cc:248] gaussian_random raises an exception class thrust::system::system_error, parallel_for failed: cudaErrorNoKernelImageForDevice: no kernel image is available for execution on the device Traceback (most recent call last): File "E:/ASR/PaddlePaddle-DeepSpeech-master/train.py", line 105, in main() File "E:/ASR/PaddlePaddle-DeepSpeech-master/train.py", line 101, in main train() File "E:/ASR/PaddlePaddle-DeepSpeech-master/train.py", line 96, in train test_off=args.test_off) File "E:\ASR\PaddlePaddle-DeepSpeech-master\model_utils\model.py", line 232, in train exe.run(startup_prog) File "F:\software\CONDA\envs\TS\lib\site-packages\paddle\fluid\executor.py", line 1262, in run six.reraise(*sys.exc_info()) File "F:\software\CONDA\envs\TS\lib\site-packages\six.py", line 719, in reraise raise value File "F:\software\CONDA\envs\TS\lib\site-packages\paddle\fluid\executor.py", line 1260, in run return_merged=return_merged) File "F:\software\CONDA\envs\TS\lib\site-packages\paddle\fluid\executor.py", line 1402, in _run_impl use_program_cache=use_program_cache) File "F:\software\CONDA\envs\TS\lib\site-packages\paddle\fluid\executor.py", line 1492, in _run_program [fetch_var_name]) RuntimeError: parallel_for failed: cudaErrorNoKernelImageForDevice: no kernel image is available for execution on the device

Process finished with exit code 1 安装2.2.0之后运行train报错是这样

[yeyupiaoling]

你显卡驱动正常吗？或者你用Linux系统试一下。