search

facebookresearch / demucs

Code for the paper Hybrid Spectrogram and Waveform Source Separation
MIT License
8.1k stars 1.01k forks source link

Demucs Output Doesn't Null Against Input #587

Open hockinsk opened 6 months ago

hockinsk commented 6 months ago

Hi, is it expected the summed Demucs 4 stem output doesn't entirely null with the Source? I've tried various prediction shifts and overlaps but always an 'Unseparated' amount of audio left. Hope the video shows, source being nulled against total Demucs output and leaving the unseparated residue audible / visible. https://github.com/facebookresearch/demucs/assets/1609513/2b7c24ed-7c18-4385-aba4-e7be18143b80

CarlGao4 commented 6 months ago

Stem "other" is also a normal stem like "vocals" and other stem. Or, demucs will not only separate three stems and calculate "other".

hockinsk commented 6 months ago

Yes, but vocal + bass + drums + other when summed together do not equal the stereo input signal. There is an 'unmixed' amount of audio. In order to get the unmixed audio, I merge demucs output, then phase invert that against the input. The audio not phase cancelled = unmixed. I do notice one other product that used Demucs 4 Stem actually output the Unmixed stem so it's a 5 stem model.

CarlGao4 commented 6 months ago

I remember this has been explained in another issue... demucs will see "other" same as other stems like "vocals" and "drums", though it is trained with dataset whose mixture is calculated by adding up all stems... So you can calculate the "other" stem yourself using mixture - bass - drums - vocals if you are using API. You can also try Demucs-GUI and see how I handle this problem

hockinsk commented 6 months ago

I have a process that creates the 'Missing' stem of audio. I then separate that missing audio with SpectraLayers which is 100% separation and put the missing audio back into Vocal, Bass, Drums & Other. Then the Demucs will 100% phase invert to silence against the original input.

https://github.com/facebookresearch/demucs/assets/1609513/22f5891b-64c9-46a1-a684-515df26a81eb