search

PaddlePaddle / PaddleSpeech

Easy-to-use Speech Toolkit including Self-Supervised Learning model, SOTA/Streaming ASR with punctuation, Streaming TTS with text frontend, Speaker Verification System, End-to-End Speech Translation and Keyword Spotting. Won NAACL2022 Best Demo Award.
https://paddlespeech.readthedocs.io
Apache License 2.0
10.56k stars 1.81k forks source link

Out of memory error on GPU 0 #1934

Closed qiuyuzhao closed 2 years ago

qiuyuzhao commented 2 years ago

运行deepspeech2offline_aishell模型的时候提示：

Connected to pydev debugger (build 201.8538.36) /home/ssd/suimang/miniconda3/envs/paddle/lib/python3.7/site-packages/librosa/core/constantq.py:1059: DeprecationWarning: np.complex is a deprecated alias for the builtin complex. To silence this warning, use complex by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.complex128 here. Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations dtype=np.complex, [2022-05-23 09:27:06,773] [ INFO] - start to init the model [2022-05-23 09:27:06,774] [ INFO] - File /root/.paddlespeech/models/deepspeech2offline_aishell-zh-16k/asr0_deepspeech2_aishell_ckpt_0.1.1.model.tar.gz md5 checking... [2022-05-23 09:27:11,201] [ INFO] - Use pretrained model stored in: /root/.paddlespeech/models/deepspeech2offline_aishell-zh-16k/asr0_deepspeech2_aishell_ckpt_0.1.1.model.tar [2022-05-23 09:27:11,201] [ INFO] - /root/.paddlespeech/models/deepspeech2offline_aishell-zh-16k/asr0_deepspeech2_aishell_ckpt_0.1.1.model.tar [2022-05-23 09:27:11,201] [ INFO] - /root/.paddlespeech/models/deepspeech2offline_aishell-zh-16k/asr0_deepspeech2_aishell_ckpt_0.1.1.model.tar/model.yaml [2022-05-23 09:27:11,201] [ INFO] - /root/.paddlespeech/models/deepspeech2offline_aishell-zh-16k/asr0_deepspeech2_aishell_ckpt_0.1.1.model.tar/exp/deepspeech2/checkpoints/avg_1.pdparams [2022-05-23 09:27:13,483] [ INFO] - unique_endpoints {''} [2022-05-23 09:27:13,484] [ INFO] - File /root/.paddlespeech/models/language_model/data/lm/zh_giga.no_cna_cmn.prune01244.klm md5 checking... [2022-05-23 09:27:20,268] [ INFO] - Found /root/.paddlespeech/models/language_model/data/lm/zh_giga.no_cna_cmn.prune01244.klm /home/ssd/suimang/miniconda3/envs/paddle/lib/python3.7/site-packages/_distutils_hack/init.py:30: UserWarning: Setuptools is replacing distutils. warnings.warn("Setuptools is replacing distutils.") W0523 09:27:20.295001 1187855 gpu_context.cc:278] Please NOTE: device: 0, GPU Compute Capability: 8.6, Driver API Version: 11.5, Runtime API Version: 10.2 W0523 09:27:20.299127 1187855 gpu_context.cc:306] device: 0, cuDNN Version: 8.3. [2022-05-23 09:27:30,077] [ INFO] - Preprocess audio_file:/home/ssd/suimang/project/PaddleSpeech/1.wav [2022-05-23 09:27:30,091] [ INFO] - audio feat shape: [1, 707, 161] [2022-05-23 09:27:30,092] [ INFO] - audio feat process success [2022-05-23 09:27:58,393] [ INFO] - start to infer the model to get the output Traceback (most recent call last): File "/home/ssd/suimang/miniconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/dygraph/base.py", line 354, in _decorate_function return func(*args, kwargs) File "/home/ssd/suimang/project/PaddleSpeech/paddlespeech/cli/asr/infer.py", line 302, in infer result_transcripts = self.model.decode(audio, audio_len) File "/home/ssd/suimang/miniconda3/envs/paddle/lib/python3.7/site-packages/decorator.py", line 232, in fun return caller(func, *(extras + args), *kw) File "/home/ssd/suimang/miniconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/dygraph/base.py", line 354, in _decorate_function return func(args, kwargs) File "/home/ssd/suimang/project/PaddleSpeech/paddlespeech/s2t/models/ds2/deepspeech2.py", line 171, in decode eouts, eouts_len = self.encoder(audio, audio_len) File "/home/ssd/suimang/miniconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 930, in call return self._dygraph_call_func(*inputs, kwargs) File "/home/ssd/suimang/miniconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, *kwargs) File "/home/ssd/suimang/project/PaddleSpeech/paddlespeech/s2t/models/ds2/deepspeech2.py", line 77, in forward x, x_lens = self.conv(x, x_lens) File "/home/ssd/suimang/miniconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 930, in call return self._dygraph_call_func(inputs, kwargs) File "/home/ssd/suimang/miniconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func outputs = self.forward(*inputs, kwargs) File "/home/ssd/suimang/project/PaddleSpeech/paddlespeech/s2t/models/ds2/conv.py", line 168, in forward x, x_len = self.conv_in(x, x_len) File "/home/ssd/suimang/miniconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 930, in call return self._dygraph_call_func(*inputs, *kwargs) File "/home/ssd/suimang/miniconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 915, in _dygraph_call_func outputs = self.forward(inputs, kwargs) File "/home/ssd/suimang/project/PaddleSpeech/paddlespeech/s2t/models/ds2/conv.py", line 114, in forward masks = make_non_pad_mask(x_len) #[B, T] File "/home/ssd/suimang/project/PaddleSpeech/paddlespeech/s2t/modules/mask.py", line 90, in make_non_pad_mask return make_pad_mask(lengths).logical_not() File "/home/ssd/suimang/project/PaddleSpeech/paddlespeech/s2t/modules/mask.py", line 61, in make_pad_mask seq_range = paddle.arange(0, max_len, dtype=paddle.int64) File "/home/ssd/suimang/miniconda3/envs/paddle/lib/python3.7/site-packages/paddle/tensor/creation.py", line 567, in arange return paddle.fluid.layers.range(start, end, step, dtype, name) File "/home/ssd/suimang/miniconda3/envs/paddle/lib/python3.7/site-packages/paddle/fluid/layers/tensor.py", line 1505, in range out = _C_ops.range(start, end, step) SystemError: (Fatal) Operator range raises an paddle::memory::allocation::BadAlloc exception. The exception content is :ResourceExhaustedError:

Out of memory error on GPU 0. Cannot allocate 1.035075EB memory on GPU 0, 1.563965GB memory has been allocated and available memory is only 22.135559GB.

Please check whether there is any other process using GPU 0.

  1. If yes, please stop them, or start PaddlePaddle on another GPU.
  2. If no, please decrease the batch size of your model. If the above ways do not solve the out of memory problem, you can try to use CUDA managed memory. The command is export FLAGS_use_cuda_managed_memory=false. (at /paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:87) . (at /paddle/paddle/fluid/imperative/tracer.cc:307) 但是我0卡的显存应该够用啊，是我输入的不对么，下面的代码： import paddle from paddlespeech.cli import ASRExecutor

asr_executor = ASRExecutor() text = asr_executor( model='deepspeech2offline_aishell', lang='zh', sample_rate=16000, config=None, # Set config and ckpt_path to None to use pretrained model. ckpt_path=None, audio_file='1.wav', force_yes=False, device=paddle.get_device()) print('ASR Result: \n{}'.format(text))

yt605155624 commented 2 years ago

你输入的音频是不是很大，竟然要求 1.035075EB 的显存，请尝试输入 50s 以内的音频

qiuyuzhao commented 2 years ago

我推理的7秒的音频，转成cpu就可以运行了，这是为什么？

yt605155624 commented 2 years ago

你可以按照提示试试，可能 GPU 有问题？

Please check whether there is any other process using GPU 0.
If yes, please stop them, or start PaddlePaddle on another GPU.
If no, please decrease the batch size of your model.
If the above ways do not solve the out of memory problem, you can try to use CUDA managed memory. The command is export FLAGS_use_cuda_managed_memory=false.
(at /paddle/paddle/fluid/memory/allocation/cuda_allocator.cc:87)
. (at /paddle/paddle/fluid/imperative/tracer.cc:307)
ianZzzzzz commented 1 year ago

我在运行demo时遇到了同样的问题，export FLAGS_use_cuda_managed_memory=false无用。paddlespeech asr --lang zh --input zh.wav ，Out of memory error on GPU 0. Cannot allocate 655.142555PB memory on GPU 0, 1.729004GB memory has been allocated and available memory is only 21.970520GB

leolle commented 1 year ago

paddle降级到2.3.0试试