bytedance / music_source_separation

Other
1.24k stars 192 forks source link

CUDA out of memory #27

Closed sophia1488 closed 2 years ago

sophia1488 commented 2 years ago

Hi! First, thanks for your contribution to music source separation! Since I'd like to separate vocals & accompaniment using only 1 model, and the pre-trained model you provided has only one target, I decided to train it from scratch. I also modify the code so that a validation set is added.

However, I cannot train it on 3 GTX 1080 with small batch size (<12), and I thought that the ResUNet_ismir doesn't require that much memory? Or does it really need a lot of GPU memory? Could you provide some insight, like how long it takes for you to train a model and how many GPU memory it requires?

Thank you a lot!

qiuqiangkong commented 2 years ago

Hi, please use ResUNet_subband model, it is 4x smaller and the performance is close to the ResUNet system.

On Tue, 9 Nov 2021 at 14:30, Sophia Y. Chou @.***> wrote:

Hi! First, thanks for your contribution to music source separation! Since I'd like to separate vocals & accompaniment using only 1 model, and the pre-trained model you provided has only one target, I decided to train it from scratch. I also modify the code so that a validation set is added.

However, I cannot train it on 3 GTX 1080 with small batch size (<12), and I thought that the ResUNet_ismir doesn't require that much memory? Or does it really need a lot of GPU memory? Could you provide some insight, like how long it takes for you to train a model and how many GPU memory it requires?

Thank you a lot!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bytedance/music_source_separation/issues/27, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADFXTSLSQPEESZFFCH7T6X3ULC5YTANCNFSM5HUPDLSA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

sophia1488 commented 2 years ago

Thank you so much, though I later found that it was my preprocessing problem... One other thing I'd like to ask is that is it normal to have negative SDR at epoch 8?

qiuqiangkong commented 2 years ago

Usually the SDR is positive after 1 epochs.

On Sat, 13 Nov 2021 at 21:00, Sophia Y. Chou @.***> wrote:

Thank you so much, though I later found that it was my preprocessing problem... One other thing I'd like to ask is that is it normal to have negative SDR at epoch 8?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bytedance/music_source_separation/issues/27#issuecomment-968064629, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADFXTSJRBROSYV4X4I3A6NLULZOPJANCNFSM5HUPDLSA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

sophia1488 commented 2 years ago

Got it, thanks!