[Bug] YourTTS alignment is weird

Describe the bug

I am training YourTTS model on VCTK using the recipe provided in this repo. But the alignment seems off, and the audio output is not great, as it often skips over words.

Below is the alignment plot. I tried both using VITS checkpoint trained on LJSpeech for 1M steps (like in the paper), and starting from scratch, but the alignment is still the same. Can someone confirm if the current code for training YourTTS is working or not? Thanks.

eval losses:

To Reproduce

install TTS
download checkpoint from https://github.com/coqui-ai/TTS/releases/download/v0.5.0_models/tts_models--multilingual--multi-dataset--your_tts.zip
in recipes/vctk/yourtts/train_yourtts.py, change SPEAKER_ENCODER_CHECKPOINT_PATH and SPEAKER_ENCODER_CONFIG_PATH to your downloaded checkpoint
(optional) change RESTORE_PATH to model_file.pth to start training from pretrained VITS checkpoint
run train_yourtts.py

Expected behavior

good audio, good alignment

Logs

No response

Environment

{
    "CUDA": {
        "GPU": [
            "A100-SXM4-40GB"
        ],
        "available": true,
        "version": "11.8"
    },
    "Packages": {
        "PyTorch_debug": false,
        "PyTorch_version": "2.1.1",
        "TTS": "0.22.0",
        "numpy": "1.22.0"
    },
    "System": {
        "OS": "Linux",
        "architecture": [
            "64bit",
            "ELF"
        ],
        "processor": "x86_64",
        "python": "3.9.18",
        "version": "#152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022"
    }
}

Additional context