search

mindspore-lab / mindaudio

A toolbox of audio models and algorithms based on MindSpore
Apache License 2.0
39 stars 12 forks source link

deepspeech2模型ascend执行训练报错 #164

Closed Venomfly closed 1 year ago

Venomfly commented 1 year ago
  1. 【Document Link】/【文档链接】 https://github.com/mindspore-lab/mindaudio/blob/main/recipes/LibriSpeech/ASR/README_CN.md
  2. 【Issues Section】/【问题文档片段】 python train.py -c "./deepspeech2.yaml"
  3. 【Existing Issues】/【存在的问题】 执行训练报错：python train.py -c "./deepspeech2.yaml"

[ERROR] DEVICE(44814,7f3352ffd700,python):2023-06-01-22:47:58.744.915 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_graph_executor.cc:195] RunGraph] run task error! [ERROR] DEVICE(44814,7f3352ffd700,python):2023-06-01-22:47:58.744.961 [mindspore/ccsrc/plugin/device/ascend/hal/hardware/ascend_utils.cc:35] ReportErrorMessage] Ascend error occurred, error message: E39999: Inner Error! E39999 Aicpu kernel execute failed, device_id=0, stream_id=12, task_id=2.[FUNC:PrintAicpuErrorInfo][FILE:task.cc][LINE:747] Aicpu kernel execute failed, device_id=0, stream_id=12, task_id=2, fault op_name=[FUNC:GetError][FILE:stream.cc][LINE:846] rtStreamSynchronize execute failed, reason=[the model stream execute failed][FUNC:FuncErrorReason][FILE:error_message_manage.cc][LINE:45]

Traceback (most recent call last): File "/data1/rhx/mindaudio/recipes/LibriSpeech/ASR/train.py", line 124, in train(args=args) File "/data1/rhx/mindaudio/recipes/LibriSpeech/ASR/train.py", line 92, in train model.train( File "/opt/buildtools/python-3.9.11/lib/python3.9/site-packages/mindspore/train/model.py", line 1044, in train self._train(epoch, File "/opt/buildtools/python-3.9.11/lib/python3.9/site-packages/mindspore/train/model.py", line 98, in wrapper func(self, *args, kwargs) File "/opt/buildtools/python-3.9.11/lib/python3.9/site-packages/mindspore/train/model.py", line 622, in _train self._train_dataset_sink_process(epoch, train_dataset, list_callback, File "/opt/buildtools/python-3.9.11/lib/python3.9/site-packages/mindspore/train/model.py", line 701, in _train_dataset_sink_process outputs = train_network(inputs) File "/opt/buildtools/python-3.9.11/lib/python3.9/site-packages/mindspore/nn/cell.py", line 578, in call out = self.compile_and_run(args) File "/opt/buildtools/python-3.9.11/lib/python3.9/site-packages/mindspore/nn/cell.py", line 988, in compile_and_run return _cell_graph_executor(self, new_inputs, phase=self.phase) File "/opt/buildtools/python-3.9.11/lib/python3.9/site-packages/mindspore/common/api.py", line 1202, in call return self.run(obj, args, phase=phase) File "/opt/buildtools/python-3.9.11/lib/python3.9/site-packages/mindspore/common/api.py", line 1239, in run return self._exec_pip(obj, args, phase=phase_real) File "/opt/buildtools/python-3.9.11/lib/python3.9/site-packages/mindspore/common/api.py", line 98, in wrapper results = fn(arg, kwargs) File "/opt/buildtools/python-3.9.11/lib/python3.9/site-packages/mindspore/common/api.py", line 1221, in _exec_pip return self._graph_executor(args, phase) RuntimeError: Launch graph failed, graph id: 1

vigo999 commented 1 year ago

recipe folder has been removed. please check examples/deepspeech2