search

mixxxdj / mixxx

Mixxx is Free DJ software that gives you everything you need to perform live mixes.
http://mixxx.org
Other
4.53k stars 1.28k forks source link

AI stems separation #11391

Open napaalm opened 1 year ago

napaalm commented 1 year ago

Feature Description

VirtualDJ and Serato now offer an AI-based real-time stem separation feature. Since there are many open source separation models with GPL-compatible licenses (such as demucs), a similar feature could be included in Mixxx.

I don't know if demucs or other models are capable of real-time processing, so my proposal is to allow the user to precompute these stems and save them in the library, similarly to track analysis. This solution would obviously require to support multi-track decks and possibly a specific file format, which has been discussed on the forum and in #7935.

daschuer commented 1 year ago

Do you have interest to implement this? We need first describe how the routing through effect and master Mixxx should work. Something like a graph that indicates which part of Mixxx should made stems aware.

JoergAtGithub commented 1 year ago

Real-Time-Processing seems to be difficult: https://github.com/mixxxdj/mixxx/pull/4760#issuecomment-1145281433

timewasternl commented 1 year ago

Perhaps a first step can be to load already separated stems for a track and leave the actual AI separation itself outside of Mixxx (for now).

fsalomon commented 7 months ago

In case anybody is interested: the .serato-stems file format is fairly simple to decode. It's just some basic TLV with the actual mp3 payload for the four stems xored by 0x26. See this code for better understanding. It will take a .serato-stems file and dump out all four stems into separate mp3 files.

JoergAtGithub commented 7 months ago

That's interesting. Currently @acolombier is working on support for the Native Instruments STEMS format ( #13044 ). As far as I can see, the Serato format is just an internal stems cache, only written and read by SeratoDJ itself. While the NI STEMS format is intended for stems exchange between different applications.