search

facebookresearch / audiocraft

Audiocraft is a library for audio processing and generation with deep learning. It features the state-of-the-art EnCodec audio compressor / tokenizer, along with MusicGen, a simple and controllable music generation LM with textual and melodic conditioning.
MIT License
21.06k stars 2.17k forks source link

How to fuse several `WaveformConditioner`s #460

Open javanasse opened 6 months ago

javanasse commented 6 months ago

Conditioners are defined in config/conditioner like chroma2music.yaml. The chroma model's parameters are populated under self_wav:.

I have defined another conditioner, and would like to train MusicGen conditioned using two WaveformConditioners, namely the ChromaStemConditioner and the new conditioner that I defined. What is the best way to do that?

Is there a syntax for doing this in the conditioner YAML or is it better to create a new WaveformConditioner that simply returns a stacked tensor formed from both conditions?

adefossez commented 3 months ago

you could hack around by create a self_wav2 attribute, then you can set it to some other conditionner class: https://github.com/facebookresearch/audiocraft/blob/main/audiocraft/data/music_dataset.py#L64 add a line out.wav[key + '2'] = value just under