Can I train a English 1 Speaker Voice?

[CypherpunkSamurai]

Hello 👋🏼

I want to train a vits model on pretrained model using english voice I have. How do I make the data and which model to use?

my current data is "metadata.csv", "voice_1.wav"... metadata.csv has:

voice_1|Transcript
voice_2|Transcript...

also do I need to use the official ljspeech model for english?

[AnyaCoder]

You can use your prepare wav files only to train, follow the instructions in DATA.md and LOCAL.md

[CypherpunkSamurai]

Which model for English?

[AnyaCoder]

Which model for English?

CJE model

[CypherpunkSamurai]

Just a note the file https://drive.google.com/file/d/132l97zjanpoPY4daLgqXoM7HKXPRbS84/view?usp=sharing is not found. Can you please reupload it to github releases / huggingface or mediafire etc?

[CypherpunkSamurai]

What is sampled_audio4ft_v2.zip ?

[AnyaCoder]

What is sampled_audio4ft_v2.zip ?

aux train data, if you don't have enough qualified audio files

[CypherpunkSamurai]

@AnyaCoder how can I use manual transcript like ljspeech and not run whisper?

[CypherpunkSamurai]

I checked final_annotation_train buts its all unknown ascii

[AnyaCoder]

I checked final_annotation_train buts its all unknown ascii

The latest transcriptions are the files named short_character_anno.txt and long_character_anno.txt, which are written in utf-8. You can run whisper first, get the two files mentioned before, and manually replace the transcrption using your own one, like ljspeech or vctk. The file final_annotation_train.txt contains only phonetic symbols, that perplex humans. If you are really reluctant, you can manually write the two files above, referring to the format wav_path|speaker_id|[lang]transcription[lang].Then run the preprocessv2.py, get the final_annotation_train.txt and final_annotation_val.txt.

[CypherpunkSamurai]

If you are really reluctant, you can manually write the two files above, referring to the format

Like...

list_of_folders = os.listdir("custom_voices")

for voice_n, voice in enumerate(list_of_folders):
     for metadata_line in open(os.path.join("custom_voices", voice, "metadata.csv")):
         line = metadata_line.split("|")
         transcript = line[len(line)-1]
         wav_file_p = os.path.join("custom_voices", voice, line[0])
         open("short_character_anno.txt", "a").write(f"{wav_file_p}|{voice_n}|{transcript}")

[AnyaCoder]

Examples: ZH -> EN

[CypherpunkSamurai]

Ah, ok. So speaker name instead of id. And language tagging.

Like

./wav/ljspeech/wav1.wav|ljspeech_v2|[EN]Printing in the only...[EN]

[CypherpunkSamurai]

I'm using a complete replica of ljspeech, i just wanna use this repo for training.

The TTS from mozilla is really weird.

[AnyaCoder]

Author

This repo is great. I'm sure it will satisfy you.

[CypherpunkSamurai]

Just a question, can i use sentences like this:

379 chars

There was nothing so _very_ remarkable in that; nor did Alice think it so _very_ much out of the way to hear the Rabbit say to itself, “Oh dear! Oh dear! I shall be late!” (when she thought it over afterwards, it occurred to her that she ought to have wondered at this, but at the time it all seemed quite natural); but when the Rabbit actually _took a watch out of its waistcoat-pocket_, and looked at it, and then hurried on, Alice started to her feet, for it flashed across her mind that she had never before seen a rabbit with either a waistcoat-pocket, or a watch to take out of it, and burning with curiosity, she ran across the field after it, and fortunately was just in time to see it pop down a large rabbit-hole under the hedge.

The sentence is continuation with ,. The annotations like quote and ; give effect to the voice. Would be great to retain the annotations.

Can we use long sentences? Can we retain annotations while training?

[AnyaCoder]

1. Punctuation issues: I haven't tested all the punctuation, but it is known that commas, full stops, apostrophes and exclamation points can produce pauses. It's best to replace them all with one of these four if you can.
2. It's best not to use long sentences, which can cause a lot of video memory usage and potentially make the video memory low. 2s to 10s is appropriate.
3. This depends on the corresponding sentence in the final_annotation_train.txt file, whether the punctuation you want.

[CypherpunkSamurai]

Alright...

1. Ok. Will try.
2. I'm using a P100 / T4 (Online GPU) for training, is it ok to keep long sentences?

[AnyaCoder]

Alright...

1. Ok. Will try.
2. I'm using a P100 / T4 (Online GPU) for training, is it ok to keep long sentences?

For Q.2, try a try. If the problem appears, shorten the length of wav files~

[CypherpunkSamurai]

I think I missed a point somewhere. can you please let me know how to use train the original ljspeech using this code? I can then change it accordingly.

[AnyaCoder]

I think I missed a point somewhere. can you please let me know how to use train the original ljspeech using this code? I can then change it accordingly.

What problems occured?

[CypherpunkSamurai]

I think I missed a point somewhere. can you please let me know how to use train the original ljspeech using this code? I can then change it accordingly.

What problems occured?

I don't want to run whisper. I want to use the metadata from the zip only. There are like 1k wav, and running whisper will take a lot of time. I already have the metadata, I want to use it instead.

[AnyaCoder]

That's ok, here is the python code.

with open('metadata.csv', 'r', encoding='utf-8') as source_file:
    # 打开或创建short_character_anno.txt进行写入
    with open('short_character_anno.txt', 'w', encoding='utf-8') as target_file:
        # 逐行读取source_file
        for line in source_file:
            # 删除行尾的换行符并按“|”分隔行
            parts = line.strip().split('|')
            if len(parts) != 2:
                # 如果该行不是两个部分，跳过此行
                continue
            voice_name, transcript = parts
            # 转换为期望的格式
            new_line = f"{voice_name}.wav|ljspeech|[EN]{transcription}[EN]\n"
            # 写入到short_character_anno.txt
            target_file.write(new_line)

make sure that the file metadata.csv is in the same folder with your *.wav files.

[vic-yes]

I think I missed a point somewhere. can you please let me know how to use train the original ljspeech using this code? I can then change it accordingly.

@CypherpunkSamurai , have you tried to train the original ljspeech?

@AnyaCoder , I also would like to train the ljspeech using this code. https://huggingface.co/espnet/kan-bayashi_ljspeech_vits/tree/main/exp/tts_train_vits_raw_phn_tacotron_g2p_en_no_space, here is the ljspeech model I'd like to train, but the config file is a ymal file, could you know how to convert the file to the "finetune_speaker.json"?

I also found that there are two models "G" and "D", what differences are they? Do I have to split the ljspeech model to G and D model?

[CypherpunkSamurai]

@vic-yes no I haven't. I tried on my custom voice, it turned out robotic. The tone is weird.

[AnyaCoder]

@CypherpunkSamurai @vic-yes come to my repository Bert-VITS2, it is very good in 3 languages ZH, JP, EN

[vic-yes]

@AnyaCoder , thanks, I just noticed this project too, and I am currently studying it.

[vic-yes]

@AnyaCoder , do you have a pre-trained English model used in Bert-VITS2 for finetuning?

[AnyaCoder]

yes, you can get them from here: pretrained_models

[CypherpunkSamurai]

@AnyaCoder any colab scripts?

[vic-yes]

yes, you can get them from here: pretrained_models

@AnyaCoder , it seems to be a CN pre-trained model, not EN.

[AnyaCoder]

yes, you can get them from here: pretrained_models

@AnyaCoder , it seems to be a CN pre-trained model, not EN. Oh, sorry, here are the models for EN: https://huggingface.co/SpicyqSama007/Bert-VITS2-v2.0.1-ZH-JP-EN/tree/main

[vic-yes]

yes, you can get them from here: pretrained_models

@AnyaCoder , it seems to be a CN pre-trained model, not EN. Oh, sorry, here are the models for EN: https://huggingface.co/SpicyqSama007/Bert-VITS2-v2.0.1-ZH-JP-EN/tree/main

@AnyaCoder grateful for your help! could you also provide the config.json file?

[AnyaCoder]

@AnyaCoder any colab scripts?

my repo Bert-VITS2 script, you can modify the path to your data: base.ipynb

or these links(translation required, I think you will know): 【腾讯文档】2.0-轻量部署方法： https://docs.qq.com/doc/DS3V5d3dnZXlxVUlL 【bilibili】使用方法： https://docs.qq.com/doc/DS3V3QnFLQVJYZHdn 【腾讯文档】2.0的维护: https://docs.qq.com/doc/DS3V3QnFLQVJYZHdn 【腾讯文档】Bert-VITS2-v2.0报错自查 https://docs.qq.com/doc/DS0FqQ0VGZ2dnVWlP 嫌麻烦，可以用整合包(Intergration Package)： 「Bert-VITS2-v2.0.1」 链接：https://pan.quark.cn/s/f18e81404a63 提取码：SepH

[AnyaCoder]

yes, you can get them from here: pretrained_models

@AnyaCoder , it seems to be a CN pre-trained model, not EN. Oh, sorry, here are the models for EN: https://huggingface.co/SpicyqSama007/Bert-VITS2-v2.0.1-ZH-JP-EN/tree/main

@AnyaCoder grateful for your help! could you also provide the config.json file?

config.json can be found in the repo Bert-VITS2, in the folderconfigs