rhasspy / piper

A fast, local neural text to speech system
https://rhasspy.github.io/piper-samples/
MIT License
5.17k stars 360 forks source link

LR Scheduler error at training #239

Open gns23 opened 9 months ago

gns23 commented 9 months ago

since i managed to install piper-tts on fresh install of raspberry pi os 64bit i'm trying to train a voice using ljspeech like throsten did in his video https://www.youtube.com/watch?v=b_we_jma220

while executing

(.venv) user@raspi:~/piper/src/python $ python3 -m piper_train --dataset-dir /home/user/train --accelerator 'cpu' --devices 1 --batch-size 24 --validation-split 0.0 --num-test-examples 0 --max_epochs 10000 --resume_from_checkpoint /home/user/train/epoch\=3135-step\=2702056.ckpt --checkpoint-epochs 1 --precision 32 --quality medium

i get following error


DEBUG:piper_train:Namespace(dataset_dir='/home/user/train', checkpoint_epochs=1, quality='medium', resume_from_single_speaker_checkpoint=None, logger=True, enable_checkpointing=True, default_root_dir=None, gradient_clip_val=None, gradient_clip_algorithm=None, num_nodes=1, num_processes=None, devices='1', gpus=None, auto_select_gpus=False, tpu_cores=None, ipus=None, enable_progress_bar=True, overfit_batches=0.0, track_grad_norm=-1, check_val_every_n_epoch=1, fast_dev_run=False, accumulate_grad_batches=None, max_epochs=10000, min_epochs=None, max_steps=-1, min_steps=None, max_time=None, limit_train_batches=None, limit_val_batches=None, limit_test_batches=None, limit_predict_batches=None, val_check_interval=None, log_every_n_steps=50, accelerator='cpu', strategy=None, sync_batchnorm=False, precision=32, enable_model_summary=True, weights_save_path=None, num_sanity_val_steps=2, resume_from_checkpoint='/home/user/train/epoch=3135-step=2702056.ckpt', profiler=None, benchmark=None, deterministic=None, reload_dataloaders_every_n_epochs=0, auto_lr_find=False, replace_sampler_ddp=True, detect_anomaly=False, auto_scale_batch_size=False, plugins=None, amp_backend='native', amp_level=None, move_metrics_to_cpu=False, multiple_trainloader_mode='max_size_cycle', batch_size=24, validation_split=0.0, num_test_examples=0, max_phoneme_ids=None, hidden_channels=192, inter_channels=192, filter_channels=768, n_layers=6, n_heads=2, seed=1234)
/home/user/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/trainer/connectors/checkpoint_connector.py:52: LightningDeprecationWarning: Setting `Trainer(resume_from_checkpoint=)` is deprecated in v1.5 and will be removed in v1.7. Please pass `Trainer.fit(ckpt_path=)` directly instead.
  rank_zero_deprecation(
GPU available: False, used: False
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
DEBUG:piper_train:Checkpoints will be saved every 1 epoch(s)
/home/user/piper/src/python/.venv/lib/python3.11/site-packages/torch/nn/utils/weight_norm.py:30: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
  warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
DEBUG:vits.dataset:Loading dataset: /home/user/train/dataset.jsonl
/home/user/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py:731: LightningDeprecationWarning: `trainer.resume_from_checkpoint` is deprecated in v1.5 and will be removed in v2.0. Specify the fit checkpoint path with `trainer.fit(ckpt_path=)` instead.
  ckpt_path = ckpt_path or self.resume_from_checkpoint
Restoring states from the checkpoint path at /home/user/train/epoch=3135-step=2702056.ckpt
DEBUG:fsspec.local:open file: /home/user/train/epoch=3135-step=2702056.ckpt
/home/user/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/callbacks/model_checkpoint.py:345: UserWarning: The dirpath has changed from '/home/hansenm/larynx2/local/de/thorsten/medium/lightning_logs/version_3/checkpoints' to '/home/user/train/lightning_logs/version_0/checkpoints', therefore `best_model_score`, `kth_best_model_path`, `kth_value`, `last_model_path` and `best_k_models` won't be reloaded. Only `best_model_path` will be reloaded.
  warnings.warn(
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/user/piper/src/python/piper_train/__main__.py", line 147, in <module>
    main()
  File "/home/user/piper/src/python/piper_train/__main__.py", line 124, in main
    trainer.fit(model)
  File "/home/user/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 696, in fit
    self._call_and_handle_interrupt(
  File "/home/user/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 735, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 1147, in _run
    self.strategy.setup(self)
  File "/home/user/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/strategies/single_device.py", line 74, in setup
    super().setup(trainer)
  File "/home/user/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 153, in setup
    self.setup_optimizers(trainer)
  File "/home/user/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/strategies/strategy.py", line 141, in setup_optimizers
    self.optimizers, self.lr_scheduler_configs, self.optimizer_frequencies = _init_optimizers_and_lr_schedulers(
                                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/core/optimizer.py", line 194, in _init_optimizers_and_lr_schedulers
    _validate_scheduler_api(lr_scheduler_configs, model)
  File "/home/user/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/core/optimizer.py", line 351, in _validate_scheduler_api
    raise MisconfigurationException(
pytorch_lightning.utilities.exceptions.MisconfigurationException: The provided lr scheduler `ExponentialLR` doesn't follow PyTorch's LRScheduler API. You should override the `LightningModule.lr_scheduler_step` hook with your own logic if you are using a custom LR scheduler.

can anyone provide some help? wish i could take a look at the output of 'pip list' from someone where its working..

thx!

Bigbynth commented 8 months ago

hey I got the dependency that was working for me here, check it out: torch==1.13.1 torchaudio==0.13.1 torchvision==0.14.1 torchtext==0.14.0 torchdata==0.5.1