search

facebookresearch / svoice

We provide a PyTorch implementation of the paper Voice Separation with an Unknown Number of Multiple Speakers In which, we present a new method for separating a mixed audio sequence, in which multiple voices speak simultaneously. The new method employs gated neural networks that are trained to separate the voices at multiple processing steps, while maintaining the speaker in each output channel fixed. A different model is trained for every number of possible speakers, and the model with the largest number of speakers is employed to select the actual number of speakers in a given sample. Our method greatly outperforms the current state of the art, which, as we show, is not competitive for more than two speakers.
Other
1.23k stars 178 forks source link

fixed speaker #75

Closed srdfjy closed 1 year ago

srdfjy commented 2 years ago

Hi @adiyoss ,I want to pin certain speakers to s1 when separating audio, does this model support it?

adiyoss commented 2 years ago

The model is invariant to the speakers and speaker order. So we don't have support for it, but it should be straight forward to include it and train the model that like that. You would probably need to change the objective function for that.

srdfjy commented 2 years ago

If the number of speakers of the audio to be separated is unknown, select 5 speakers to generate the training sample, and modify config.yaml->swave->C=5. Is the generalization performance of the trained model optimal?