search

PlayVoice / whisper-vits-svc

Core Engine of Singing Voice Conversion & Singing Voice Clone
https://huggingface.co/spaces/maxmax20160403/sovits5.0
MIT License
2.6k stars 919 forks source link

index retrieval implementation from RVC #88

Closed futorio closed 9 months ago

futorio commented 1 year ago

Hi! I'm currently attempting to implement the "Feature Retrieval" method from the original RVC repository. However, I'm facing an issue where increasing the number of n-retrieval-vectors to 4 results in complete silence at the output audio file. Prior to this, with values below 4, the timbre correction works well. I'm seeking assistance in understanding and resolving this problem.

For start train indexes run command: python train_retrieval.py Or run with debug information python train_retrieval.py --debug

For enable feature retrieval in inference add to standard svc_inference.py script flags: "--enable-retrieval" and "--debug" if you want display retrieval debug information.

futorio commented 1 year ago

I can just fix the count of nearest vectors (to 1, 3 or 2), but I don't really want to do that

thepowerfuldeez commented 9 months ago

Hi! Maybe we could re-open?

futorio commented 9 months ago

Hi! Maybe we could re-open?

Hi, yes i can. I need to fix conflicts

thepowerfuldeez commented 9 months ago

Did it help to improve robustness and timbre leakage in your experiments?

futorio commented 9 months ago

Yes it improved timbre leakage. But in my experiments, the best result was achieved with different numbers of nearest vectors. I think this is due to the vector counting method from the original RVC repository. Now I'm at the stage of code refactoring, I think I'll be able to push an updated version soon.

thepowerfuldeez commented 9 months ago

You mean by using distance threshold instead of top-k vectors?

futorio commented 9 months ago

No, I mean the number of vectors from the faiss index that are closer to the vector from the inference audio. In the code it will be possible to write your own implementation of calculating the averaged vector from the faiss index

futorio commented 9 months ago

@thepowerfuldeez I have tested the index training stage on synthetic data, but I have not yet had the opportunity to test the inference stage. If it’s not difficult for you, you can try retrieval on your data or wait for me.