search

thorstenMueller / Thorsten-Voice

Thorsten-Voice: A free to use, offline working, high quality german TTS voice should be available for every project without any license struggling.
http://www.thorsten-voice.de
Creative Commons Zero v1.0 Universal
554 stars 52 forks source link

Errors training with newer python #64

Closed popey closed 6 months ago

popey commented 6 months ago

Hi Thorsten! I'm just starting out with training. (A bad version of) my voice (apope UK) is in a bunch of open-source tools (originating in Mycroft AI) and I'd like to redo it to make it better. Your tutorial is excellent. I'm going through it now to learn the process, thanks for making it.

I'm running in a venv on Ubuntu 23.10 on my workstation, and found the training part failed. Ever seen this?

(I did not specify to use the GPU because I'm remote from the machine and the GPU is unplugged. I just wanted to test the process. I will put a GPU in it at some point, but I can't even run the training on the CPU yet.)

 python3 -m piper_train --dataset-dir ./out-train/  --batch-size 4 --validation-split 0.0 --num-test-examples 0 --max_epochs 10000 --resume_from_checkpoint ./out-train/epoch\=2218-step\=838782.ckpt --checkpoint-epochs 1 --precision 32 --quality high
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/alan/Source/rhasspy/piper/src/python/piper_train/__main__.py", line 7, in <module>
    from pytorch_lightning import Trainer
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/__init__.py", line 34, in <module>
    from pytorch_lightning.callbacks import Callback  # noqa: E402
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/callbacks/__init__.py", line 26, in <module>
    from pytorch_lightning.callbacks.pruning import ModelPruning
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/callbacks/pruning.py", line 30, in <module>
    from pytorch_lightning.core.module import LightningModule
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/core/__init__.py", line 16, in <module>
    from pytorch_lightning.core.module import LightningModule
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/core/module.py", line 42, in <module>
    from pytorch_lightning.trainer.connectors.logger_connector.fx_validator import _FxValidator
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/trainer/__init__.py", line 16, in <module>
    from pytorch_lightning.trainer.trainer import Trainer
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/trainer/trainer.py", line 39, in <module>
    from pytorch_lightning.accelerators import (
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/accelerators/__init__.py", line 14, in <module>
    from pytorch_lightning.accelerators.cpu import CPUAccelerator  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/accelerators/cpu.py", line 19, in <module>
    from pytorch_lightning.utilities.device_parser import parse_cpu_cores
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/utilities/device_parser.py", line 20, in <module>
    from pytorch_lightning.plugins.environments import TorchElasticEnvironment
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/plugins/__init__.py", line 10, in <module>
    from pytorch_lightning.plugins.precision.apex_amp import ApexMixedPrecisionPlugin
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/plugins/precision/__init__.py", line 17, in <module>
    from pytorch_lightning.plugins.precision.fsdp_native_native_amp import FullyShardedNativeNativeMixedPrecisionPlugin
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/pytorch_lightning/plugins/precision/fsdp_native_native_amp.py", line 24, in <module>
    from torch.distributed.fsdp.fully_sharded_data_parallel import MixedPrecision
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/torch/distributed/fsdp/__init__.py", line 1, in <module>
    from .flat_param import FlatParameter
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/torch/distributed/fsdp/flat_param.py", line 26, in <module>
    from ._fsdp_extensions import _ext_post_unflatten_transform, _ext_pre_flatten_transform
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/torch/distributed/fsdp/_fsdp_extensions.py", line 7, in <module>
    from torch.distributed.fsdp._shard_utils import _create_chunk_sharded_tensor
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/torch/distributed/fsdp/_shard_utils.py", line 10, in <module>
    from torch.distributed._shard.sharded_tensor import (
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/torch/distributed/_shard/__init__.py", line 1, in <module>
    from .api import (
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/torch/distributed/_shard/api.py", line 6, in <module>
    from torch.distributed._shard.sharded_tensor import (
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/torch/distributed/_shard/sharded_tensor/__init__.py", line 8, in <module>
    import torch.distributed._shard.sharding_spec as shard_spec
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/torch/distributed/_shard/sharding_spec/__init__.py", line 1, in <module>
    from .api import (
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/torch/distributed/_shard/sharding_spec/api.py", line 16, in <module>
    import torch.distributed._shard.sharded_tensor.metadata as sharded_tensor_meta
  File "/home/alan/Source/rhasspy/piper/src/python/.venv/lib/python3.11/site-packages/torch/distributed/_shard/sharded_tensor/metadata.py", line 70, in <module>
    @dataclass
     ^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 1230, in dataclass
    return wrap(cls)
           ^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 1220, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 958, in _process_class
    cls_fields.append(_get_field(cls, name, type, kw_only))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 815, in _get_field
    raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'torch.distributed._shard.sharded_tensor.metadata.TensorProperties'> for field tensor_properties is not allowed: use default_factory
pip list
Package                  Version      Editable project location
------------------------ ------------ ------------------------------------------
absl-py                  2.1.0
aiohttp                  3.9.5
aiosignal                1.3.1
attrs                    23.2.0
audioread                3.0.1
certifi                  2024.2.2
cffi                     1.16.0
charset-normalizer       3.3.2
coloredlogs              15.0.1
Cython                   0.29.37
decorator                5.1.1
flatbuffers              24.3.25
frozenlist               1.4.1
fsspec                   2024.5.0
grpcio                   1.64.0
humanfriendly            10.0
idna                     3.7
joblib                   1.4.2
lazy_loader              0.4
librosa                  0.10.2.post1
lightning-utilities      0.11.2
llvmlite                 0.42.0
Markdown                 3.6
MarkupSafe               2.1.5
mpmath                   1.3.0
msgpack                  1.0.8
multidict                6.0.5
numba                    0.59.1
numpy                    1.26.4
nvidia-cublas-cu11       11.10.3.66
nvidia-cuda-nvrtc-cu11   11.7.99
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cudnn-cu11        8.5.0.96
onnxruntime              1.18.0
packaging                24.0
pip                      24.0
piper-phonemize          1.1.0
piper_train              1.0.0        /home/alan/Source/rhasspy/piper/src/python
platformdirs             4.2.2
pooch                    1.8.1
protobuf                 5.26.1
pycparser                2.22
pyDeprecate              0.3.2
pytorch-lightning        1.7.7
PyYAML                   6.0.1
requests                 2.32.1
scikit-learn             1.4.2
scipy                    1.13.0
setuptools               69.5.1
six                      1.16.0
soundfile                0.12.1
soxr                     0.3.7
sympy                    1.12
tensorboard              2.16.2
tensorboard-data-server  0.7.2
threadpoolctl            3.5.0
torch                    1.13.1
torchmetrics             0.11.4
tqdm                     4.66.4
typing_extensions        4.11.0
urllib3                  2.2.1
Werkzeug                 3.0.3
wheel                    0.43.0
yarl                     1.9.4
domcross commented 6 months ago

Hi Alan,

Given the technical limitations I always liked your voice on my Mycroft Mark-I.

(Py)Torch 1.13 is not compatible with Python 3.11, you may try Python 3.10 instead: https://github.com/Lightning-AI/pytorch-lightning/issues/15614

popey commented 6 months ago

Thanks for the kind words, and help. I'll try this on a 22.04 machine with Python 3.10.

thorstenMueller commented 6 months ago

I feel honored, that the great open source voice contributor @popey uses my video tutorial to train a more up to date voice version 🥰.