search

yxlu-0102 / MP-SENet

Explicit Estimation of Magnitude and Phase Spectra in Parallel for High-Quality Speech Enhancement
MIT License
315 stars 45 forks source link

OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. #13

Open risqaliyevds opened 11 months ago

risqaliyevds commented 11 months ago

I have 24 GB GPU but I get this error. I try to use enhance my own audio with yours best checkpoints what is problem or it needs more gpu?

OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. Process 2674591 has 16.90 GiB memory in use. Including non-PyTorch memory, this process has 1.26 GiB memory in use.

yxlu-0102 commented 11 months ago

I used two 24 GB GPUs to train the model, so you just used one or somethings ?

risqaliyevds commented 10 months ago

I solved this problem. I have another question can I train a 48k rate audio model? If yes what I should change or it is impossible? I changed config.json file (to 48k) and trained my own model (get checkpoint) but it wasn't different from your best checkpoint.

yxlu-0102 commented 10 months ago

Our model was trained on a corpus with a 16 kHz sampling rate, where the original data was downsampled from 48 kHz to 16 kHz. It is feasible to train our model with a 48 kHz noisy dataset. You just need to change the sampling_rate in the config.json to 48000.

QQQQQQQQY commented 6 months ago

I have 24 GB GPU but I get this error. I try to use enhance my own audio with yours best checkpoints what is problem or it needs more gpu?

OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. Process 2674591 has 16.90 GiB memory in use. Including non-PyTorch memory, this process has 1.26 GiB memory in use.

Hi, this has happened to me too, how did you solve it

yxlu-0102 commented 6 months ago

I have 24 GB GPU but I get this error. I try to use enhance my own audio with yours best checkpoints what is problem or it needs more gpu? OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. Process 2674591 has 16.90 GiB memory in use. Including non-PyTorch memory, this process has 1.26 GiB memory in use.

Hi, this has happened to me too, how did you solve it

When did you encounter this issue? in training or inference?

QQQQQQQQY commented 6 months ago

I have 24 GB GPU but I get this error. I try to use enhance my own audio with yours best checkpoints what is problem or it needs more gpu? OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. Process 2674591 has 16.90 GiB memory in use. Including non-PyTorch memory, this process has 1.26 GiB memory in use.

Hi, this has happened to me too, how did you solve it

When did you encounter this issue? in training or inference?

Hi, I'm happy to hear from you, I came across it in training

risqaliyevds commented 6 months ago

I have 24 GB GPU but I get this error. I try to use enhance my own audio with yours best checkpoints what is problem or it needs more gpu? OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. Process 2674591 has 16.90 GiB memory in use. Including non-PyTorch memory, this process has 1.26 GiB memory in use.

Hi, this has happened to me too, how did you solve it

Hi what is your audio sample rate?

QQQQQQQQY commented 6 months ago

I have 24 GB GPU but I get this error. I try to use enhance my own audio with yours best checkpoints what is problem or it needs more gpu? OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. Process 2674591 has 16.90 GiB memory in use. Including non-PyTorch memory, this process has 1.26 GiB memory in use.

Hi, this has happened to me too, how did you solve it

Hi what is your audio sample rate?

16 kHz

yxlu-0102 commented 6 months ago

I have 24 GB GPU but I get this error. I try to use enhance my own audio with yours best checkpoints what is problem or it needs more gpu? OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. Process 2674591 has 16.90 GiB memory in use. Including non-PyTorch memory, this process has 1.26 GiB memory in use.

Hi, this has happened to me too, how did you solve it

Hi what is your audio sample rate?

16 kHz

How many batches did you set on each GPU?

risqaliyevds commented 6 months ago

I cant remember you can start from 1.

пн, 22 апр. 2024 г., 12:54 Rory Lu @.***>:

I have 24 GB GPU but I get this error. I try to use enhance my own audio with yours best checkpoints what is problem or it needs more gpu? OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. Process 2674591 has 16.90 GiB memory in use. Including non-PyTorch memory, this process has 1.26 GiB memory in use.

Hi, this has happened to me too, how did you solve it

Hi what is your audio sample rate?

16 kHz

How many batches did you set on each GPU?

— Reply to this email directly, view it on GitHub https://github.com/yxlu-0102/MP-SENet/issues/13#issuecomment-2068727935, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3QRDA5TGTZXHYRCVS4J2Z3Y6S62FAVCNFSM6AAAAABAY7VA52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRYG4ZDOOJTGU . You are receiving this because you authored the thread.Message ID: @.***>

QQQQQQQQY commented 6 months ago

I cant remember you can start from 1. пн, 22 апр. 2024 г., 12:54 Rory Lu @.>: … I have 24 GB GPU but I get this error. I try to use enhance my own audio with yours best checkpoints what is problem or it needs more gpu? OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. Process 2674591 has 16.90 GiB memory in use. Including non-PyTorch memory, this process has 1.26 GiB memory in use. Hi, this has happened to me too, how did you solve it Hi what is your audio sample rate? 16 kHz How many batches did you set on each GPU? — Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3QRDA5TGTZXHYRCVS4J2Z3Y6S62FAVCNFSM6AAAAABAY7VA52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRYG4ZDOOJTGU . You are receiving this because you authored the thread.Message ID: @.>

Thank you very much for your reply, I will try

Johnsonabuse commented 4 months ago

作者您好 请问您用什么型号什么数量的GPU训练的？我用的两张3090的训练您的模型 每一个Epoch大概要1个小时 batchsize的参数设置为多少呢？谢谢

yxlu-0102 commented 4 months ago

作者您好 请问您用什么型号什么数量的GPU训练的？我用的两张3090的训练您的模型 每一个Epoch大概要1个小时 batchsize的参数设置为多少呢？谢谢

我用的也是两张 3090，batchsize 设的还是 4

vkeep commented 3 months ago

作者您好 请问您用什么型号什么数量的GPU训练的？我用的两张3090的训练您的模型 每一个Epoch大概要1个小时 batchsize的参数设置为多少呢？谢谢

我用的也是两张 3090，batchsize 设的还是 4

I use default config, bs =4, segment size = 32000, 2*4090 24G gpu, also OOM.

will the result degrade when set the bs =2?

Johnsonabuse commented 3 months ago

Yes I set batchsize to 2 when I Use 2×3090

发自我的iPhone

------------------ Original ------------------ From: vkeep @.> Date: Tue,Jul 23,2024 3:11 PM To: yxlu-0102/MP-SENet @.> Cc: Johnsonabuse @.>, Comment @.> Subject: Re: [yxlu-0102/MP-SENet] OutOfMemoryError: CUDA out of memory. Triedto allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50GiB is free. (Issue #13)

作者您好 请问您用什么型号什么数量的GPU训练的？我用的两张3090的训练您的模型 每一个Epoch大概要1个小时 batchsize的参数设置为多少呢？谢谢

我用的也是两张 3090，batchsize 设的还是 4

I use default config, bs =4, segment size = 32000, 2*4090 24G gpu, also OOM.

will the result degrade when set the bs =2?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

Lijingze666 commented 2 months ago

我有 24 GB 的 GPU，但我收到此错误。我尝试使用您最好的检查点来增强自己的音频，这是什么问题或需要更多 gpu？ OutOfMemoryError：CUDA 内存不足。尝试分配 15.25 GiB。GPU 0 的总容量为 23.69 GiB，其中 5.50 GiB 是免费的。进程 2674591 有 16.90 GiB 内存正在使用。包括非 PyTorch 内存在内，此进程使用的内存为 1.26 GiB。

嗨，我也遇到过这种情况，你是怎么解决的

您何时遇到此问题？在训练还是推理中？

Hello, I encountered this issue during the inference process. How can I resolve it?