high stop loss on LJSpeech dataset with TTS

[OswaldoBornemann]

I am trying Tacotron2 based on LJSpeech dataset and try to reproduced the result that erogol did. But both two experiments did not work well, both of two seem to have high stop loss.

The first experiment main config is below

{
"github_branch":"* master",
        "run_name": "tacotron2_test11",
        "run_description": "using forward attention, with original prenet, loss masking,separate stopnet, sigmoid. Compare this with 4817. Pytorch DPP",

        "audio":{
            // Audio processing parameters
            "num_mels": 80,         // size of the mel spec frame. 
            "num_freq": 1025,       // number of stft frequency levels. Size of the linear spectogram frame.
            "sample_rate": 22050,   // DATASET-RELATED: wav sample-rate. If different than the original data, it is resampled.
            "frame_length_ms": 50,  // stft window length in ms.
            "frame_shift_ms": 12.5, // stft window hop-lengh in ms.
            "preemphasis": 0.98,    // pre-emphasis to reduce spec noise and make it more structured. If 0.0, no -pre-emphasis.
            "min_level_db": -100,   // normalization range
            "ref_level_db": 20,     // reference level db, theoretically 20db is the sound of air.
            "power": 1.5,           // value to sharpen wav signals after GL algorithm.
            "griffin_lim_iters": 60,// #griffin-lim iterations. 30-60 is a good range. Larger the value, slower the generation.
            // Normalization parameters
            "signal_norm": true,    // normalize the spec values in range [0, 1]
            "symmetric_norm": false, // move normalization to range [-1, 1]
            "max_norm": 1,          // scale normalization to range [-max_norm, max_norm] or [0, max_norm]
            "clip_norm": true,      // clip normalized values into the range.
            "mel_fmin": 0.0,         // minimum freq level for mel-spec. ~50 for male and ~95 for female voices. Tune for dataset!!
            "mel_fmax": 8000.0,        // maximum freq level for mel-spec. Tune for dataset!!
            "do_trim_silence": true  // enable trimming of slience of audio as you load it. LJspeech (false), TWEB (false), Nancy (true)
        },

        "distributed":{
            "backend": "nccl",
            "url": "tcp:\/\/localhost:54321"
        },

        "reinit_layers": [],

        "model": "Tacotron2",          // one of the model in models/    
        "grad_clip": 1,                // upper limit for gradients for clipping.
        "epochs": 1000,                // total number of epochs to train.
        "lr": 0.0001,                  // Initial learning rate. If Noam decay is active, maximum learning rate.
        "lr_decay": false,             // if true, Noam learning rate decaying is applied through training.
        "warmup_steps": 4000,          // Noam decay steps to increase the learning rate from 0 to "lr"
        "memory_size": 5,              // ONLY TACOTRON - memory queue size used to queue network predictions to feed autoregressive connection. Useful if r < 5. 
        "attention_norm": "sigmoid",   // softmax or sigmoid. Suggested to use softmax for Tacotron2 and sigmoid for Tacotron.
        "prenet_type": "original",     // "original" or "bn".
        "prenet_dropout": true,        // enable/disable dropout at prenet.
        "use_forward_attn": true,      // enable/disable forward attention. In general, it aligns faster.
        "forward_attn_mask": false,    // Apply forward attention mask af inference to prevent bad modes. Try it if your model does not align well.
        "transition_agent": true,     // enable/disable transition agent of forward attention.
        "location_attn": false,        // enable_disable location sensitive attention. It is enabled for TACOTRON by default.
        "loss_masking": true,          // enable / disable loss masking against the sequence padding.
        "enable_eos_bos_chars": true, // enable/disable beginning of sentence and end of sentence chars.
        "stopnet": true,               // Train stopnet predicting the end of synthesis. 
        "separate_stopnet": true,      // Train stopnet seperately if 'stopnet==true'. It prevents stopnet loss to influence the rest of the model. It causes a better model, but it trains SLOWER.
        "tb_model_param_stats": false,     // true, plots param stats per layer on tensorboard. Might be memory consuming, but good for debugging. 

        "windowing": false,             // Enables attention windowing. Used only in eval mode.
        "forward_attn_masking": false,  // Enable forward attention masking which improves attention stability. Use it if network does not work as you like when it is off.        

        "batch_size": 32,       // Batch size for training. Lower values than 32 might cause hard to learn attention.
        "eval_batch_size":16,   
        "r": 1,                 // Number of frames to predict for step.
    "gradual_training": null,
        "wd": 0.000001,         // Weight decay weight.
        "checkpoint": true,     // If true, it saves checkpoints per "save_step"
        "save_step": 1000,      // Number of training steps expected to save traning stats and checkpoints.
        "print_step": 10,       // Number of steps to log traning on console.
        "batch_group_size": 0,  //Number of batches to shuffle after bucketing.

        "run_eval": true,
        "test_delay_epochs": 5,  //Until attention is aligned, testing only wastes computation time.
        "test_sentences_file": null,  // set a file to load sentences to be used for testing. If it is null then we use default english sentences.
        //"data_path": "/home/jovyan/zengruihong-volume-1/peter/tts_data/LJSpeech-1.1/",  // DATASET-RELATED: can overwritten from command argument
        //"meta_file_train": "metadata_train.csv",      // DATASET-RELATED: metafile for training dataloader.
        //"meta_file_val": "metadata_val.csv",    // DATASET-RELATED: metafile for evaluation dataloader.
        //"dataset": "ljspeech",      // DATASET-RELATED: one of TTS.dataset.preprocessors depending on your target dataset. Use "tts_cache" for pre-computed dataset by extract_features.py
        "min_seq_len": 0,       // DATASET-RELATED: minimum text length to use in training
        "max_seq_len": 300,     // DATASET-RELATED: maximum text length
        "output_path": "taco_output/LJ_tacotron2_output/",      // DATASET-RELATED: output path for all training outputs.
        "num_loader_workers": 4,        // number of training data loader processes. Don't set it too big. 4-8 are good values.
        "num_val_loader_workers": 4,    // number of evaluation data loader processes.
        "phoneme_cache_path": "mozilla_us_phonemes",  // phoneme computation is slow, therefore, it caches results in the given folder.
        "use_phonemes": true,           // use phonemes instead of raw characters. It is suggested for better pronounciation.
        "phoneme_language": "en-us",     // depending on your target language, pick one from  https://github.com/bootphon/phonemizer#languages
        "text_cleaner": "phoneme_cleaners",
        "use_speaker_embedding": false, // whether to use additional embeddings for separate speakers
    "use_gst": false,
    "datasets":   // List of datasets. They all merged and they get different speaker_ids.
        [
            {
                "name": "ljspeech",
                "path": "/tts_data/LJSpeech-1.1/",
                "meta_file_train": "metadata_train.csv",
                "meta_file_val": "metadata_val.csv"
            }
        ]

    }

The results are below

The second experiment main config is below

{
"github_branch":"* master",
        "run_name": "tacotron2_test12",
        "run_description": "using forward attention, with original prenet, loss masking,separate stopnet, sigmoid. Compare this with 4817. Pytorch DPP",

        "audio":{
            // Audio processing parameters
            "num_mels": 80,         // size of the mel spec frame. 
            "num_freq": 1025,       // number of stft frequency levels. Size of the linear spectogram frame.
            "sample_rate": 22050,   // DATASET-RELATED: wav sample-rate. If different than the original data, it is resampled.
            "frame_length_ms": 50,  // stft window length in ms.
            "frame_shift_ms": 12.5, // stft window hop-lengh in ms.
            "preemphasis": 0.98,    // pre-emphasis to reduce spec noise and make it more structured. If 0.0, no -pre-emphasis.
            "min_level_db": -100,   // normalization range
            "ref_level_db": 20,     // reference level db, theoretically 20db is the sound of air.
            "power": 1.5,           // value to sharpen wav signals after GL algorithm.
            "griffin_lim_iters": 60,// #griffin-lim iterations. 30-60 is a good range. Larger the value, slower the generation.
            // Normalization parameters
            "signal_norm": true,    // normalize the spec values in range [0, 1]
            "symmetric_norm": false, // move normalization to range [-1, 1]
            "max_norm": 1,          // scale normalization to range [-max_norm, max_norm] or [0, max_norm]
            "clip_norm": true,      // clip normalized values into the range.
            "mel_fmin": 0.0,         // minimum freq level for mel-spec. ~50 for male and ~95 for female voices. Tune for dataset!!
            "mel_fmax": 8000.0,        // maximum freq level for mel-spec. Tune for dataset!!
            "do_trim_silence": true  // enable trimming of slience of audio as you load it. LJspeech (false), TWEB (false), Nancy (true)
        },

        "distributed":{
            "backend": "nccl",
            "url": "tcp:\/\/localhost:54321"
        },

        "reinit_layers": [],

        "model": "Tacotron2",          // one of the model in models/    
        "grad_clip": 1,                // upper limit for gradients for clipping.
        "epochs": 1000,                // total number of epochs to train.
        "lr": 0.0001,                  // Initial learning rate. If Noam decay is active, maximum learning rate.
        "lr_decay": false,             // if true, Noam learning rate decaying is applied through training.
        "warmup_steps": 4000,          // Noam decay steps to increase the learning rate from 0 to "lr"
        "memory_size": 5,              // ONLY TACOTRON - memory queue size used to queue network predictions to feed autoregressive connection. Useful if r < 5. 
        "attention_norm": "softmax",   // softmax or sigmoid. Suggested to use softmax for Tacotron2 and sigmoid for Tacotron.
        "prenet_type": "original",     // "original" or "bn".
        "prenet_dropout": true,        // enable/disable dropout at prenet.
        "use_forward_attn": true,      // enable/disable forward attention. In general, it aligns faster.
        "forward_attn_mask": true,    // Apply forward attention mask af inference to prevent bad modes. Try it if your model does not align well.
        "transition_agent": false,     // enable/disable transition agent of forward attention.
        "location_attn": false,        // enable_disable location sensitive attention. It is enabled for TACOTRON by default.
        "loss_masking": true,          // enable / disable loss masking against the sequence padding.
        "enable_eos_bos_chars": false, // enable/disable beginning of sentence and end of sentence chars.
        "stopnet": true,               // Train stopnet predicting the end of synthesis. 
        "separate_stopnet": true,      // Train stopnet seperately if 'stopnet==true'. It prevents stopnet loss to influence the rest of the model. It causes a better model, but it trains SLOWER.
        "tb_model_param_stats": false,     // true, plots param stats per layer on tensorboard. Might be memory consuming, but good for debugging. 

        "windowing": false,             // Enables attention windowing. Used only in eval mode.
        "forward_attn_masking": false,  // Enable forward attention masking which improves attention stability. Use it if network does not work as you like when it is off.        

        "batch_size": 80,       // Batch size for training. Lower values than 32 might cause hard to learn attention.
        "eval_batch_size":16,   
        "r": 1,                 // Number of frames to predict for step.
    "gradual_training": null,
        "wd": 0.000001,         // Weight decay weight.
        "checkpoint": true,     // If true, it saves checkpoints per "save_step"
        "save_step": 1000,      // Number of training steps expected to save traning stats and checkpoints.
        "print_step": 10,       // Number of steps to log traning on console.
        "batch_group_size": 0,  //Number of batches to shuffle after bucketing.

        "run_eval": true,
        "test_delay_epochs": 5,  //Until attention is aligned, testing only wastes computation time.
        "test_sentences_file": null,  // set a file to load sentences to be used for testing. If it is null then we use default english sentences.
        //"data_path": "/home/jovyan/zengruihong-volume-1/peter/tts_data/LJSpeech-1.1/",  // DATASET-RELATED: can overwritten from command argument
        //"meta_file_train": "metadata_train.csv",      // DATASET-RELATED: metafile for training dataloader.
        //"meta_file_val": "metadata_val.csv",    // DATASET-RELATED: metafile for evaluation dataloader.
        //"dataset": "ljspeech",      // DATASET-RELATED: one of TTS.dataset.preprocessors depending on your target dataset. Use "tts_cache" for pre-computed dataset by extract_features.py
        "min_seq_len": 0,       // DATASET-RELATED: minimum text length to use in training
        "max_seq_len": 300,     // DATASET-RELATED: maximum text length
        "output_path": "taco_output/LJ_tacotron2_output/",      // DATASET-RELATED: output path for all training outputs.
        "num_loader_workers": 3,        // number of training data loader processes. Don't set it too big. 4-8 are good values.
        "num_val_loader_workers": 3,    // number of evaluation data loader processes.
        "phoneme_cache_path": "mozilla_us_phonemes",  // phoneme computation is slow, therefore, it caches results in the given folder.
        "use_phonemes": true,           // use phonemes instead of raw characters. It is suggested for better pronounciation.
        "phoneme_language": "en-us",     // depending on your target language, pick one from  https://github.com/bootphon/phonemizer#languages
        "text_cleaner": "phoneme_cleaners",
        "use_speaker_embedding": false, // whether to use additional embeddings for separate speakers
    "use_gst": false,
    "datasets":   // List of datasets. They all merged and they get different speaker_ids.
        [
            {
                "name": "ljspeech",
                "path": "/home/jovyan/data1/peter/tts_data/LJSpeech-1.1/",
                "meta_file_train": "metadata_train.csv",
                "meta_file_val": "metadata_val.csv"
            }
        ]
    }

Both two experiments have been trained almost 10 days.

[erogol]

try without forward attention but location sensitive attention. 2nd one looks fine btw. Reopen if you need it.

[OswaldoBornemann]

@erogol But how could you successfully used forward attention with LJSpeech dataset ?

[erogol]

I don'rt remember how I trained it before but currently I sue it for inference only.

[OswaldoBornemann]

@erogol thanks a lot ! And happy new year ~