search

facebookresearch / demucs

Code for the paper Hybrid Spectrogram and Waveform Source Separation
MIT License
8.35k stars 1.06k forks source link

Videogame Stems as possible model training source #386

Open Dyslexicon opened 2 years ago

Dyslexicon commented 2 years ago

❓ Questions

Perhaps the massive amount of videogame stems ie Guitar Hero, Rockband, etc, could be utilized to create improved separation models. Partial mixdowns would likely be required first as many of these are already separated into individual files for kick/snare/overheads (and would need to be mixed down simply to "drums").

If more training data is desired for future separation models or versions of Demucs I want to bring this to the attention of either the Demucs team, or anyone who would be interested in undertaking such a task as a freelance effort.

More ambitiously - I would like to suggest a highly desirable innovation that Demucs could aspire to in the future would be separating individual percussive elements (kick/snare/toms/cymbals), as seen on software such as RipX (which does a very poor job), but I believe Demucs could excel at if fed large amounts of separated stem data, in the context of future versions of Demucs or advanced separation models.

I would enjoy seeing Demucs outclass any commercial software at this task in future versions. Videogame stems may be the largest available source for training the AI on this feature, and could provide an expanded data set for the current version of Demucs to train improved models.

dmr-songquito commented 1 year ago

For separating the different drum instruments, you can use this open-source software: https://www.audiolabs-erlangen.de/content/resources/MIR/NMFtoolbox/demoDrumSoundSeparationNMF.html

This methods works very well together with drum stems extracted by demucs.

Dyslexicon commented 1 year ago

For separating the different drum instruments, you can use this open-source software: https://www.audiolabs-erlangen.de/content/resources/MIR/NMFtoolbox/demoDrumSoundSeparationNMF.html

This methods works very well together with drum stems extracted by demucs.

These examples of individual Drum element separation are stunningly good. Would be extremely valuable if Demucs could incorporate this technology! "demucs -n drum_ElementsNMF" yielding kick.wav, snare.wav, cymbals.wav

Dyslexicon commented 1 year ago

Struggling on how to actually install and utilize this NMFtoolbox. Anyone willing to write up some actual clear and concise (even pedantic) directions for both installation and usage via Python/Anaconda?

Many thanks!

CarlGao4 commented 1 year ago

Can you provide a link to the data that could be used to train demucs? I'd like to have a try if I can download them