Closed hesz94 closed 1 year ago
Sure thing, I'll post it tomorrow
Redacted comment, found the problem.
tbh your fork seems to be doing well, i'm not sure if i ought to continue rather than archiving this and letting you do your thing
as for 1. if tortoise was trained on LJSpeech dataset, it may be an artefact of that: LJSpeech Dataset - all samples within the dataset are prepared as 22.05kHz
nevertheless I've asked the author about this here
Added the expanded latent loading/averaging in #13
can be closed
What I think @hesz94 meant in his first point was:
voice_samples
voice_samples
are then used by tts()
in get_conditioning_latents()
at two lines: [here](), where they're required to be 22.050kHz, and here, where every sample is reconverted to 24kHz. This probably degrades the quality of the voice samples passed to the diffuser, because the inputs are going from original -> 22.05 -> 24
.so what i ought to do is to accept voice samples of any frequency, and specifically convert them to the right values for get_conditioning_latents, which I have implemented in this commit.
If any of this sounds wrong, please reopen the issue.
1) According to the readme the voice samples for voice cloning HAVE TO be at 22.05kHz, but then the first thing done to them after loading is resampling them to 24kHz - should prolly switch up to inputting non-default sampling rate if we're resampling anyway
2) After reading the voice samples and resampling them, only first 102400 samples (~4.27s at 24kHz fs) are used to generate the latents - as far as I'm aware nothing stops us from either taking more samples, or looping the process over existing samples and averaging latents for the specific voice. Tried both approaches locally and they seemed to have increased voice stability (which makes sense since such averaged latents should be more representative of a voice profile).