search

mozilla / TTS

:robot: :speech_balloon: Deep learning for Text to Speech (Discussion forum: https://discourse.mozilla.org/c/tts)
Mozilla Public License 2.0
9.3k stars 1.24k forks source link

Running compute_embeddings.py fails with "TypeError: type object argument after ** must be a mapping, not str" #356

Closed thorstenMueller closed 4 years ago

thorstenMueller commented 4 years ago

Hello dear community.

Thanks to the great support by @nmstoker i try to run compute_embeddings.py (master branch) on a ljspeech dataset in a venv environment:

python3 ./compute_embeddings.py path/best_model.pth.tar path/speaker_encoder/config.json path/LJSpeech-1.1 path/output

The process fails directly with following error:

 > Setting up Audio Processor...
 | > sample_rate:22050
 | > num_mels:80
 | > min_level_db:-100
 | > frame_shift_ms:12.5
 | > frame_length_ms:50
 | > ref_level_db:20
 | > num_freq:1025
 | > power:1.5
 | > preemphasis:0.98
 | > griffin_lim_iters:60
 | > signal_norm:True
 | > symmetric_norm:True
 | > mel_fmin:0
 | > mel_fmax:8000.0
 | > max_norm:4.0
 | > clip_norm:True
 | > do_trim_silence:True
 | > sound_norm:False
 | > n_fft:2048
 | > hop_length:275
 | > win_length:1100
Traceback (most recent call last):
  File "./compute_embeddings.py", line 76, in <module>
    model = SpeakerEncoder(**c.model)
TypeError: type object argument after ** must be a mapping, not str

dataset and config source

Split metadata.csv (even it shouldn't be needed for compute embeddings):

shuf metadata.csv > metadata_shuf.csv
head -n 12000 metadata_shuf.csv > metadata_train.csv
tail -n 1100 metadata_shuf.csv > metadata_val.csv

config.json:

{
"github_branch":"* dev",
"restore_path":"/home/thorsten/___dev/tts/datasets/mozilla-pretrained-ljspeech/best_model.pth.tar",
"github_branch":"* dev",
    "model": "Tacotron2",          // one of the model in models/
    "run_name": "ljspeech-bn",
    "run_description": "tacotron2 basline finetuned with BN prenet",

    // AUDIO PARAMETERS
    "audio":{
        // Audio processing parameters
        "num_mels": 80,         // size of the mel spec frame.
        "num_freq": 1025,       // number of stft frequency levels. Size of the linear spectogram frame.
        "sample_rate": 22050,   // DATASET-RELATED: wav sample-rate. If different than the original data, it is resampled.
        "frame_length_ms": 50,  // stft window length in ms.
        "frame_shift_ms": 12.5, // stft window hop-lengh in ms.
        "preemphasis": 0.98,    // pre-emphasis to reduce spec noise and make it more structured. If 0.0, no -pre-emphasis.
        "min_level_db": -100,   // normalization range
        "ref_level_db": 20,     // reference level db, theoretically 20db is the sound of air.
        "power": 1.5,           // value to sharpen wav signals after GL algorithm.
        "griffin_lim_iters": 60,// #griffin-lim iterations. 30-60 is a good range. Larger the value, slower the generation.
        // Normalization parameters
        "signal_norm": true,    // normalize the spec values in range [0, 1]
        "symmetric_norm": true, // move normalization to range [-1, 1]
        "max_norm": 4,          // scale normalization to range [-max_norm, max_norm] or [0, max_norm]
        "clip_norm": true,      // clip normalized values into the range.
        "mel_fmin": 0.0,         // minimum freq level for mel-spec. ~50 for male and ~95 for female voices. Tune for dataset!!
        "mel_fmax": 8000.0,        // maximum freq level for mel-spec. Tune for dataset!!
        "do_trim_silence": true  // enable trimming of slience of audio as you load it. LJspeech (false), TWEB (false), Nancy (true)
    },

    // DISTRIBUTED TRAINING
    "distributed":{
        "backend": "nccl",
        "url": "tcp:\/\/localhost:54321"
    },

    "reinit_layers": [],    // give a list of layer names to restore from the given checkpoint. If not defined, it reloads all heuristically matching layers.

    // TRAINING
    "batch_size": 32,       // Batch size for training. Lower values than 32 might cause hard to learn attention. It is overwritten by 'gradual_training'.
    "eval_batch_size":16,
    "r": 7,                 // Number of decoder frames to predict per iteration. Set the initial values if gradual training is enabled.
    "gradual_training": [[0, 7, 64], [1, 5, 64], [50000, 3, 32], [130000, 2, 16], [290000, 1, 32]], // ONLY TACOTRON - set gradual training steps [first_step, r, batch_size]. If it is null, gradual training is disabled.
    "loss_masking": true,         // enable / disable loss masking against the sequence padding.

    // VALIDATION
    "run_eval": true,
    "test_delay_epochs": 10,  //Until attention is aligned, testing only wastes computation time.
    "test_sentences_file": null,  // set a file to load sentences to be used for testing. If it is null then we use default english sentences.

    // OPTIMIZER
    "grad_clip": 1,                // upper limit for gradients for clipping.
    "epochs": 1000,                // total number of epochs to train.
    "lr": 0.0001,                  // Initial learning rate. If Noam decay is active, maximum learning rate.
    "lr_decay": false,             // if true, Noam learning rate decaying is applied through training.
    "wd": 0.000001,         // Weight decay weight.
    "warmup_steps": 4000,          // Noam decay steps to increase the learning rate from 0 to "lr"

    // TACOTRON PRENET
    "memory_size": -1,              // ONLY TACOTRON - size of the memory queue used fro storing last decoder predictions for auto-regression. If < 0, memory queue is disabled and decoder only uses the last prediction frame.
    "prenet_type": "bn",     // "original" or "bn".
    "prenet_dropout": false,        // enable/disable dropout at prenet.

    // ATTENTION
    "attention_type": "original",  // 'original' or 'graves'
    "attention_heads": 5,          // number of attention heads (only for 'graves')
    "attention_norm": "sigmoid",   // softmax or sigmoid. Suggested to use softmax for Tacotron2 and sigmoid for Tacotron.
    "windowing": false,            // Enables attention windowing. Used only in eval mode.
    "use_forward_attn": false,      // if it uses forward attention. In general, it aligns faster.
    "forward_attn_mask": false,    // Additional masking forcing monotonicity only in eval mode.
    "transition_agent": false,     // enable/disable transition agent of forward attention.
    "location_attn": true,        // enable_disable location sensitive attention. It is enabled for TACOTRON by default.
    "bidirectional_decoder": false,  // use https://arxiv.org/abs/1907.09006. Use it, if attention does not work well with your dataset.

    // STOPNET
    "stopnet": true,               // Train stopnet predicting the end of synthesis.
    "separate_stopnet": true,     // Train stopnet seperately if 'stopnet==true'. It prevents stopnet loss to influence the rest of the model. It causes a better model, but it trains SLOWER.

    // TENSORBOARD and LOGGING
    "print_step": 25,       // Number of steps to log traning on console.
    "save_step": 10000,      // Number of training steps expected to save traninpg stats and checkpoints.
    "checkpoint": true,     // If true, it saves checkpoints per "save_step"
    "tb_model_param_stats": false,     // true, plots param stats per layer on tensorboard. Might be memory consuming, but good for debugging.

    // DATA LOADING
    "text_cleaner": "phoneme_cleaners",
    "enable_eos_bos_chars": false, // enable/disable beginning of sentence and end of sentence chars.
    "num_loader_workers": 4,        // number of training data loader processes. Don't set it too big. 4-8 are good values.
    "num_val_loader_workers": 4,    // number of evaluation data loader processes.
    "batch_group_size": 0,  //Number of batches to shuffle after bucketing.
    "min_seq_len": 6,       // DATASET-RELATED: minimum text length to use in training
    "max_seq_len": 150,     // DATASET-RELATED: maximum text length

    // PATHS
    "output_path": "/home/thorsten/___dev/tts/datasets/mozilla-pretrained-ljspeech/keep/",      // DATASET-RELATED: output path for all training outputs.

    // PHONEMES
    "phoneme_cache_path": "ljspeech_ph_cache",  // phoneme computation is slow, therefore, it caches results in the given folder.
    "use_phonemes": true,           // use phonemes instead of raw characters. It is suggested for better pronounciation.
    "phoneme_language": "en-us",     // depending on your target language, pick one from  https://github.com/bootphon/phonemizer#languages
    // MULTI-SPEAKER and GST
    "use_speaker_embedding": false,     // use speaker embedding to enable multi-speaker learning.
    "style_wav_for_test": null,          // path to style wav file to be used in TacotronGST inference.
    "use_gst": false,       // TACOTRON ONLY: use global style tokens

    // DATASETS
    "datasets":   // List of datasets. They all merged and they get different speaker_ids.
        [
            {
                "name": "ljspeech",
                //"path": "/data/ro/shared/data/keithito/LJSpeech-1.1/",
                "path": "/home/thorsten/___dev/tts/datasets/mozilla-pretrained-ljspeech/LJSpeech-1.1/",
                //"path": "/home/erogol/Data/LJSpeech-1.1",
                "meta_file_train": "metadata_train.csv",
                "meta_file_val": "metadata_val.csv"
            }
        ]

}

General information:

Output from pip3 list:

DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
absl-py (0.9.0)
attrdict (2.0.1)
attrs (19.3.0)
audioread (2.1.8)
bokeh (1.4.0)
cachetools (4.0.0)
certifi (2019.11.28)
cffi (1.14.0)
chardet (3.0.4)
Click (7.0)
clldutils (3.5.0)
colorlog (4.1.0)
csvw (1.7.0)
cycler (0.10.0)
decorator (4.4.1)
Flask (1.1.1)
google-auth (1.11.2)
google-auth-oauthlib (0.4.1)
grpcio (1.27.2)
idna (2.8)
isodate (0.6.0)
itsdangerous (1.1.0)
Jinja2 (2.11.1)
joblib (0.14.1)
kiwisolver (1.1.0)
librosa (0.7.2)
llvmlite (0.31.0)
Markdown (3.2.1)
MarkupSafe (1.1.1)
matplotlib (3.1.3)
numba (0.48.0)
numpy (1.18.1)
oauthlib (3.1.0)
packaging (20.1)
phonemizer (2.1)
Pillow (7.0.0)
pip (9.0.1)
pkg-resources (0.0.0)
protobuf (3.11.3)
pyasn1 (0.4.8)
pyasn1-modules (0.2.8)
pycparser (2.19)
pyparsing (2.4.6)
python-dateutil (2.8.1)
PyYAML (5.3)
regex (2020.1.8)
requests (2.22.0)
requests-oauthlib (1.3.0)
resampy (0.2.2)
rfc3986 (1.3.2)
rsa (4.0)
scikit-learn (0.22.1)
scipy (1.4.1)
segments (2.1.3)
setuptools (45.2.0)
six (1.14.0)
SoundFile (0.10.3.post1)
tabulate (0.8.6)
tensorboard (2.1.0)
tensorboardX (2.0)
torch (1.4.0)
tornado (6.0.3)
tqdm (4.42.1)
tts (1.1)
Unidecode (1.1.1)
uritemplate (3.0.1)
urllib3 (1.25.8)
Werkzeug (1.0.0)
wheel (0.34.2)
thorstenMueller commented 4 years ago

After chatting with Neil (@nmstoker) and experimenting on my local environment i learned the following lessons which might be helpful for others.

Using the following prepared datasets/files combination worked for me for compute embeddings:

If you see a growing number of .npy files you're on a good way.

After process has finished it's time to take a look to the jupyter notebook "PlotUmapLibriTTS.ipynb" under "/TTS/speaker_encoder/notebooks/"

At the end of notebook execution the cluster-graph was successfully generated.

Finally Neil told me a look to Resemblyzer (https://github.com/resemble-ai/Resemblyzer#what-can-i-do-with-this-package) might be helpful for analyzing quality of dataset.

manuel3265 commented 3 years ago

Hello @thorstenMueller I am trying to run the compute_embeddings.py but, I get the following error

Traceback (most recent call last): File "TTS/bin/compute_embeddings.py", line 100, in <module> model.load_state_dict(torch.load(args.model_path)['model']) File "/home/manuel_servex/tmp/tts-train-venv/lib/python3.7/site-packages/torch/serialization.py", line 579, in load with _open_file_like(f, 'rb') as opened_file: File "/home/manuel_servex/tmp/tts-train-venv/lib/python3.7/site-packages/torch/serialization.py", line 230, in _open_file_like return _open_file(name_or_buffer, mode) File "/home/manuel_servex/tmp/tts-train-venv/lib/python3.7/site-packages/torch/serialization.py", line 211, in __init__ super(_open_file, self).__init__(open(name, mode)) IsADirectoryError: [Errno 21] Is a directory:

From what I understand it is the model_path parameter, which must be a model, if so, is it necessary to do a previous step? or how can I execute the compute_embeddings. I would appreciate it if you could provide a quick guide.

erogol commented 3 years ago

This project is not maintained anymore. Feel free to move over Coqui-TTS https://github.com/coqui-ai/TTS which is the actively developed hard fork.