Emilia dataset

[kunibald413]

have you seen this dataset? maybe it's better suited for zero-shot task, more natural speech than audiobook

https://github.com/open-mmlab/Amphion/blob/main/preprocessors/Emilia/README.md

[e-c-k-e-r]

101k hours

Hot damn. I think my own collection caps out at ~14K hours between pieces of LibriSpeech and audiobooks, and a very, very small smattering from videogames. Even the smaller languages being >1K hours is a huge help, since the biggest hurdle for me was trying to even find a large enough corpus to piece together for my own dataset for just Japanese.

2.4TB

Daunting, but I'll see if I have some spare spinning rust I can store it and pick at it. If anything I might just start with the smaller languages to squeeze out some multi-lingual-ness that I keep meaning to go about doing.

There being transcriptions already help a ton, as a half of the pain with audio processing is the time waiting to transcribe. The other half having to quantize it all through EnCodec is a bit of a boon still, but I think my recent setup of being able to split across GPUs should help.

---

Having more audio in general, and especially non-monotonous utterances, should help a ton. I'm pretty sure I already hit immense diminishing returns after an epoch.

Appreciated. I'll see what I can pick at over the next few days while my spark hasn't waned yet once again.

[kunibald413]

doing large preps and trains can be soul-crushing. passion and a little bit of compulsion can keep you going, but i hope you don't burn yourself out.

[e-c-k-e-r]

doing large preps and trains can be soul-crushing.

It's not so bad this go-around. It used to be agonizing with system instability (segfaults or hard reboots with anything under PyTorch) on my original training system with my 4070Ti. Swapping to my 7900XTX almost entirely resolved the problems for dataset preparation and non-important training.

The estimated week-and-a-half wait for the dataset to process is always a good time for any last-minute checks or ideas to get added; for example: something to solve my dillema of subpar zero-shot performance that may stem from my naive prompt sampling (for months I've been entertaining the idea of using a vector store for input prompts to sample the closest utterance for a given training sample).

passion and a little bit of compulsion can keep you going, but i hope you don't burn yourself out.

I think I'm beyond my hyperfixate-then-burnout-then-hyperfixate cycles I've kept getting myself under, just seems to be lulls in progress until I grow a wild hair and fiddle with a new idea (for example, the STT task being quite promising for reducing the loss overall, so I hope putting emphasis on more languages would in turn help the model overall). The urgency and dread of trying to push out a decent model went away by the time I pushed out a quasi-decent ar+nar-llama-8 model with my 4xV100 system: it's rather quick to churn out a model with good results, and I don't need to keep bruteforcing additional epochs for very little gains like I did with the ar+nar-retnet-8.

---

Although again that 50K hours of English audio is still daunting. I think the best approach is to download the N-smallest .tars to prioritize more speakers over speakers with a lot of utterances (which unfortunately a lot of datasets seem to prioritize the latter over the former).

[e-c-k-e-r]

Should have everything prepared for the next training session. Detailing the below mostly for my sake to have it in writing somewhere:

• I added both the French and a portion of the English (tar files under I think 300MiB for 312 speakers) datasets, just to make sure I have enough data.
  • I would have added the Korean one and some of the Chinese dataset too, but I didn't want to have to wrestle with the phonemizer to get it to work like I've had to with Japanese (although the more I think about it, Korean shouldn't have issues, and Chinese I can probably find a better phonemizer/procedure for). There's also the issue of updating the tokenizer, since Japanese gave some unique IPAs that I had to remap the last time I tried working with it.
• (Theoretically) I have my "use the most similar utterance for a given sample" system added to hopefully better help zero-shot voice cloning. Theoretically, since I need to:
  • Verify it works when integrated. It passes my mental model, and separate testing shows that it does "work".
  • Verify that what it thinks is similar is adequate enough, since it's primarily just relying on cosine similarities of audio passed through the resp embedding (similar to inputs for STT).
  • Verify that my tokenizer does contain all the phonemes (and not have a repeat of missing phonemes with the small Japanese portion of my dataset).
  • Actually compute the similarity metadata, since I have yet to actually integrate it into the metadata generation procedure (I'm not looking forward to seeing how long it would take to process).
  • Do the final processed data => HDF5 and shove it to my 4xV100 system (but there's been some weird Linux instability after a while and the entire file gets tainted when it's not closed properly, so I'll see what I can do about it).

As for the actual dataset to train against, I think I'm going to:

• drop the donated audiobooks I've had to supplement my dataset, and LibriSpeech (small+medium+duplicate)
  • The model was already trained a majority of the time against this, and I don't want a majority of this training session dedicated to a control dataset.
  • I feel it would be a huge pain to compute the similarity metadata on speakers with a xboxhueg amount of utterances.
  • Both of these pieces of my dataset are concerningly transcribed and trimmed from whole audio files, so I don't really trust how whisperX decides to slice it at.
  • They're the most audiobook-y.
• retain LibriTTS-R, as it's not sliced, already transcribed, and I feel is the cleanest besides Emilia.
• retain all my other handpicked vidya-derived audio, since they're still rather small compared to the rest of the dataset, and they're also not audiobook-y.

I'll just resume training from the existing ar+nar-tts+stt-llama-8 weights since I don't think I need to restart from scratch (as the model is still rather malleable from all the other tweaks I've glued on), but have the same dataset sampling method of "sort by duration with a fix batch size, and let it go from smallest to largest utterances" (as I still do not trust my batch-by-duration-size method to be stable).

My hope is that the Emilia dataset is the answer to my problems. As time goes on I'm more pilled on having a very, very good dataset rather than a large one, and I feel the big crux of these TTS systems are having a sloppy dataset. If results looks promising then that'll be my push to deal with processing the rest of the dataset.