rsxdalv / tts-generation-webui

TTS Generation Web UI (Bark, MusicGen + AudioGen, Tortoise, RVC, Vocos, Demucs, SeamlessM4T, MAGNet, StyleTTS2, MMS)
https://rsxdalv.github.io/tts-generation-webui/
MIT License
1.46k stars 160 forks source link

RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR #306

Open Envyana opened 2 months ago

Envyana commented 2 months ago

Fresh installation of Ubuntu 22.04, with cuda 11.7 I encountered this error when generating any model in webui.how to fix it?

"Traceback (most recent call last): File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/queueing.py", line 407, in call_prediction output = await route_utils.call_process_api( File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/route_utils.py", line 226, in call_process_api output = await app.get_blocks().process_api( File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/blocks.py", line 1550, in process_api result = await self.call_function( File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/blocks.py", line 1185, in call_function prediction = await anyio.to_thread.run_sync( File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread return await future File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 851, in run result = context.run(func, args) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/utils.py", line 661, in wrapper response = f(args, kwargs) File "/home/ptp-ai-001/voiceai/tts-generation-webui/src/musicgen/musicgen_tab.py", line 215, in generate outputs_diffusion = mbd.tokens_to_wav(tokens) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/models/multibanddiffusion.py", line 190, in tokens_to_wav wav_diffusion = self.generate(emb=condition, size=wav_encodec.size()) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/models/multibanddiffusion.py", line 148, in generate out += DP.generate(condition=emb, step_list=step_list, initial_noise=torch.randn_like(out)) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/models/multibanddiffusion.py", line 44, in generate return self.schedule.generate_subsampled(model=self.model, initial=initial_noise, step_list=step_list, File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/modules/diffusion_schedule.py", line 272, in generate_subsampled return self.sample_processor.return_sample(previous) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/modules/diffusion_schedule.py", line 106, in return_sample bands = self.split_bands(x) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/julius/bands.py", line 84, in forward lows = self.lowpass(input) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/julius/lowpass.py", line 107, in forward out = fft_conv1d(input, self.filters, stride=self.stride) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/julius/fftconv.py", line 119, in fft_conv1d weight_z = _rfft(weight) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/julius/fftconv.py", line 26, in _new_rfft z = new_fft.rfft(x, dim=-1) RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR { type: 'status', endpoint: '/musicgen', fn_index: 155, time: 2024-04-30T03:10:49.133Z, queue: true, message: null, stage: 'error', code: undefined, success: false }"

rsxdalv commented 2 months ago

Hi, thanks for the report!

According to my search, CUDA 11.8 should be your savior. This is the link to what I assume is related: https://github.com/pytorch/pytorch/issues/88038

On Tue, Apr 30, 2024, 6:13 AM Envy42 @.***> wrote:

Fresh installation of Ubuntu 22.04, with cuda 11.7 I encountered this error when generating any model in webui.

"Traceback (most recent call last): File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/queueing.py", line 407, in call_prediction output = await route_utils.call_process_api( File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/route_utils.py", line 226, in call_process_api output = await app.get_blocks().process_api( File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/blocks.py", line 1550, in process_api result = await self.call_function( File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/blocks.py", line 1185, in call_function prediction = await anyio.to_thread.run_sync( File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync return await get_async_backend().run_sync_in_worker_thread( File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2144, in run_sync_in_worker_thread return await future File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 851, in run result = context.run(func, args) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/gradio/utils.py", line 661, in wrapper response = f(args, kwargs) File "/home/ptp-ai-001/voiceai/tts-generation-webui/src/musicgen/musicgen_tab.py", line 215, in generate outputs_diffusion = mbd.tokens_to_wav(tokens) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/models/multibanddiffusion.py", line 190, in tokens_to_wav wav_diffusion = self.generate(emb=condition, size=wav_encodec.size()) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/models/multibanddiffusion.py", line 148, in generate out += DP.generate(condition=emb, step_list=step_list, initial_noise=torch.randn_like(out)) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/models/multibanddiffusion.py", line 44, in generate return self.schedule.generate_subsampled(model=self.model, initial=initial_noise, step_list=step_list, File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/modules/diffusion_schedule.py", line 272, in generate_subsampled return self.sample_processor.return_sample(previous) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/audiocraft/modules/diffusion_schedule.py", line 106, in return_sample bands = self.split_bands(x) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/julius/bands.py", line 84, in forward lows = self.lowpass(input) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/julius/lowpass.py", line 107, in forward out = fft_conv1d(input, self.filters, stride=self.stride) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/julius/fftconv.py", line 119, in fft_conv1d weight_z = _rfft(weight) File "/home/ptp-ai-001/voiceai/installer_files/env/lib/python3.10/site-packages/julius/fftconv.py", line 26, in _new_rfft z = new_fft.rfft(x, dim=-1) RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR { type: 'status', endpoint: '/musicgen', fn_index: 155, time: 2024-04-30T03:10:49.133Z, queue: true, message: null, stage: 'error', code: undefined, success: false }"

— Reply to this email directly, view it on GitHub https://github.com/rsxdalv/tts-generation-webui/issues/306, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTRXI66SNTUW3RGUV57ZZDY74D7JAVCNFSM6AAAAABG7N2WAWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3TANBRGQ2DCMQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>