search

yxlu-0102 / MP-SENet

MP-SENet: A Speech Enhancement Model with Parallel Denoising of Magnitude and Phase Spectra
MIT License
267 stars 40 forks source link

OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. #13

Open risqaliyevds opened 7 months ago

risqaliyevds commented 7 months ago

I have 24 GB GPU but I get this error. I try to use enhance my own audio with yours best checkpoints what is problem or it needs more gpu?

OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. Process 2674591 has 16.90 GiB memory in use. Including non-PyTorch memory, this process has 1.26 GiB memory in use.

yxlu-0102 commented 7 months ago

I used two 24 GB GPUs to train the model, so you just used one or somethings ?

risqaliyevds commented 7 months ago

I solved this problem. I have another question can I train a 48k rate audio model? If yes what I should change or it is impossible? I changed config.json file (to 48k) and trained my own model (get checkpoint) but it wasn't different from your best checkpoint.

yxlu-0102 commented 7 months ago

Our model was trained on a corpus with a 16 kHz sampling rate, where the original data was downsampled from 48 kHz to 16 kHz. It is feasible to train our model with a 48 kHz noisy dataset. You just need to change the sampling_rate in the config.json to 48000.

QQQQQQQQY commented 3 months ago

I have 24 GB GPU but I get this error. I try to use enhance my own audio with yours best checkpoints what is problem or it needs more gpu?

OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. Process 2674591 has 16.90 GiB memory in use. Including non-PyTorch memory, this process has 1.26 GiB memory in use.

Hi, this has happened to me too, how did you solve it

yxlu-0102 commented 3 months ago

I have 24 GB GPU but I get this error. I try to use enhance my own audio with yours best checkpoints what is problem or it needs more gpu? OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. Process 2674591 has 16.90 GiB memory in use. Including non-PyTorch memory, this process has 1.26 GiB memory in use.

Hi, this has happened to me too, how did you solve it

When did you encounter this issue? in training or inference?

QQQQQQQQY commented 3 months ago

I have 24 GB GPU but I get this error. I try to use enhance my own audio with yours best checkpoints what is problem or it needs more gpu? OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. Process 2674591 has 16.90 GiB memory in use. Including non-PyTorch memory, this process has 1.26 GiB memory in use.

Hi, this has happened to me too, how did you solve it

When did you encounter this issue? in training or inference?

Hi, I'm happy to hear from you, I came across it in training

risqaliyevds commented 3 months ago

I have 24 GB GPU but I get this error. I try to use enhance my own audio with yours best checkpoints what is problem or it needs more gpu? OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. Process 2674591 has 16.90 GiB memory in use. Including non-PyTorch memory, this process has 1.26 GiB memory in use.

Hi, this has happened to me too, how did you solve it

Hi what is your audio sample rate?

QQQQQQQQY commented 3 months ago

I have 24 GB GPU but I get this error. I try to use enhance my own audio with yours best checkpoints what is problem or it needs more gpu? OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. Process 2674591 has 16.90 GiB memory in use. Including non-PyTorch memory, this process has 1.26 GiB memory in use.

Hi, this has happened to me too, how did you solve it

Hi what is your audio sample rate?

16 kHz

yxlu-0102 commented 3 months ago

I have 24 GB GPU but I get this error. I try to use enhance my own audio with yours best checkpoints what is problem or it needs more gpu? OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. Process 2674591 has 16.90 GiB memory in use. Including non-PyTorch memory, this process has 1.26 GiB memory in use.

Hi, this has happened to me too, how did you solve it

Hi what is your audio sample rate?

16 kHz

How many batches did you set on each GPU?

risqaliyevds commented 3 months ago

I cant remember you can start from 1.

пн, 22 апр. 2024 г., 12:54 Rory Lu @.***>:

I have 24 GB GPU but I get this error. I try to use enhance my own audio with yours best checkpoints what is problem or it needs more gpu? OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. Process 2674591 has 16.90 GiB memory in use. Including non-PyTorch memory, this process has 1.26 GiB memory in use.

Hi, this has happened to me too, how did you solve it

Hi what is your audio sample rate?

16 kHz

How many batches did you set on each GPU?

— Reply to this email directly, view it on GitHub https://github.com/yxlu-0102/MP-SENet/issues/13#issuecomment-2068727935, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3QRDA5TGTZXHYRCVS4J2Z3Y6S62FAVCNFSM6AAAAABAY7VA52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRYG4ZDOOJTGU . You are receiving this because you authored the thread.Message ID: @.***>

QQQQQQQQY commented 3 months ago

I cant remember you can start from 1. пн, 22 апр. 2024 г., 12:54 Rory Lu @.>: … I have 24 GB GPU but I get this error. I try to use enhance my own audio with yours best checkpoints what is problem or it needs more gpu? OutOfMemoryError: CUDA out of memory. Tried to allocate 15.25 GiB. GPU 0 has a total capacty of 23.69 GiB of which 5.50 GiB is free. Process 2674591 has 16.90 GiB memory in use. Including non-PyTorch memory, this process has 1.26 GiB memory in use. Hi, this has happened to me too, how did you solve it Hi what is your audio sample rate? 16 kHz How many batches did you set on each GPU? — Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3QRDA5TGTZXHYRCVS4J2Z3Y6S62FAVCNFSM6AAAAABAY7VA52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRYG4ZDOOJTGU . You are receiving this because you authored the thread.Message ID: @.>

Thank you very much for your reply, I will try

Johnsonabuse commented 1 month ago

作者您好 请问您用什么型号什么数量的GPU训练的？我用的两张3090的训练您的模型 每一个Epoch大概要1个小时 batchsize的参数设置为多少呢？谢谢

yxlu-0102 commented 1 month ago

作者您好 请问您用什么型号什么数量的GPU训练的？我用的两张3090的训练您的模型 每一个Epoch大概要1个小时 batchsize的参数设置为多少呢？谢谢

我用的也是两张 3090，batchsize 设的还是 4

vkeep commented 4 days ago

作者您好 请问您用什么型号什么数量的GPU训练的？我用的两张3090的训练您的模型 每一个Epoch大概要1个小时 batchsize的参数设置为多少呢？谢谢

我用的也是两张 3090，batchsize 设的还是 4

I use default config, bs =4, segment size = 32000, 2*4090 24G gpu, also OOM.

will the result degrade when set the bs =2?