FastSVC implementation improvements

lesterphillip / SVCC23_FastSVC

Singing Voice Conversion Challenge 2023 Starter Kit: FastSVC Reimplementation

111 stars 9 forks source link

FastSVC implementation improvements #2

Open lesterphillip opened 1 year ago

lesterphillip commented 1 year ago

Adding here some implementation improvements that I need to do courtesy of comments from @r9y9

[XX] Change F0 to log-F0 (and continuous)
[] Use original speaker embedding during training,
[XX] Fix v/uv extraction
[XX] Normalize features: F0, loudness, and speaker embeddings

I can probably add some samples to see how much improvement these make.

lesterphillip commented 1 year ago

Some updates:

I think the F0 normalization really doesn't work and it results in noisy/muffled sounds. I'm not exactly sure why? I'll try to investigate another case where the F0 isn't normalized.
Just a note, but I've been encountering issues when using 24kHz, and it significantly degrades in quality. Not sure why, or if using individually extracted speaker embeddings instead of the averaged speaker embeddings could improve this.

It's a little hard to retrain since it takes several days, so it takes some time to get some findings.

lesterphillip commented 9 months ago

I've tried more variations on improving this reimplementation (as in the checklist above), and the original version is still the best despite the unusual practices. So, it would just be better to keep it as it is right now.

On the other hand, the current issue is that compared to the original version, the synthesis has muffled-like sounds, as attached below. fastsvc_comparison.zip

More samples found here.

I don't think I can spend a lot of time on this project as it takes too long to train and I think it would just be better to try newer architectures.