coqui-ai / Trainer

🐸 - A general purpose model trainer, as flexible as it gets
196 stars 116 forks source link

[Bug] Cannot restore YourTTS model #97

Closed Ca-ressemble-a-du-fake closed 1 year ago

Ca-ressemble-a-du-fake commented 1 year ago

Describe the bug

Hi,

Following YourTTS vctk recipe I try to restore a model to continue the training.

But I get the following error :

 > Restoring from best_model_22016.pth ...
 > Restoring Model...
 > Restoring Optimizer...
Traceback (most recent call last):
  File "/home/caraduf/Models/train_yourtts_16kHz.py", line 318, in <module>
    trainer = Trainer(
  File "/home/caraduf/CoquiTTS/Trainer/trainer/trainer.py", line 507, in __init__
    (self.model, self.optimizer, self.scaler, self.restore_step, self.restore_epoch) = self.restore_model(
  File "/home/caraduf/CoquiTTS/Trainer/trainer/trainer.py", line 711, in restore_model
    optimizer = _restore_list_objs(checkpoint["optimizer"], optimizer)
  File "/home/caraduf/CoquiTTS/Trainer/trainer/trainer.py", line 701, in _restore_list_objs
    obj.load_state_dict(states)
AttributeError: 'list' object has no attribute 'load_state_dict'

The obj is a list but the else is executed :


def _restore_list_objs(states, obj):           
            if isinstance(obj, list):
                for idx, state in enumerate(states):
                    obj[idx].load_state_dict(state)
            if isinstance(obj, dict):
                for key, state in states.items():
                    obj[key].load_state_dict(state)
            else:
                obj.load_state_dict(states)
            return obj

A workaround is to replace the second if with an elif because if obj is a List it cannot become a Dict. So in my opinion it makes sense to use a elif but I can be wrong.

To Reproduce

Set a restore path to a checkpoint in the recipe and run the recipe.

python3 train_yourtts.py

Expected behavior

THe model is restored and the training goes on.

Logs

No response

Environment

{
    "CUDA": {
        "GPU": [
            "NVIDIA GeForce RTX 3090"
        ],
        "available": true,
        "version": "11.7"
    },
    "Packages": {
        "PyTorch_debug": false,
        "PyTorch_version": "1.13.1+cu117",
        "Trainer": "v0.0.22",
        "numpy": "1.22.4"
    },
    "System": {
        "OS": "Linux",
        "architecture": [
            "64bit",
            "ELF"
        ],
        "processor": "x86_64",
        "python": "3.10.6",
        "version": "#64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023"
    }
}

Additional context

No response

erogol commented 1 year ago

should be fixed by now