Open langyaya opened 1 year ago
-- Process 0 terminated with the following error: Traceback (most recent call last): File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\multiprocessing\spawn.py", line 69, in _wrap fn(i, args) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\train_ms.py", line 193, in run train_and_evaluate(rank, epoch, hps, [net_g, net_d, net_dur_disc], [optim_g, optim_d, optim_dur_disc], [scheduler_g, scheduler_d, scheduler_dur_disc], scaler, [train_loader, eval_loader], logger, [writer, writer_eval]) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\train_ms.py", line 231, in train_and_evaluate (z, z_p, m_p, logs_p, m_q, logs_q), (hiddenx, logw, logw) = net_g(x, x_lengths, spec, spec_lengths, speakers, tone, language, bert) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(input, kwargs) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\nn\parallel\distributed.py", line 1040, in forward output = self._run_ddp_forward(*inputs, *kwargs) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\nn\parallel\distributed.py", line 1000, in _run_ddp_forward return module_to_run(inputs[0], kwargs[0]) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(*input, kwargs) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\models.py", line 667, in forward l_length_sdp = self.sdp(x, x_mask, w, g=g) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(*input, *kwargs) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\models.py", line 177, in forward z_q, logdet_q = flow(z_q, x_mask, g=(x + h_w)) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(input, kwargs) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\modules.py", line 374, in forward x1, logabsdet = piecewise_rational_quadratic_transform(x1, File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\transforms.py", line 33, in piecewise_rational_quadratic_transform outputs, logabsdet = spline_fn( File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\transforms.py", line 82, in unconstrained_rational_quadratic_spline outputs[inside_interval_mask], logabsdet[inside_interval_mask] = rational_quadratic_spline( File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\transforms.py", line 105, in rational_quadratic_spline if torch.min(inputs) < left or torch.max(inputs) > right: RuntimeError: min(): Expected reduction dim to be specified for input.numel() == 0. Specify the reduction dim with the 'dim' argument.
-- Process 0 terminated with the following error: Traceback (most recent call last): File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\multiprocessing\spawn.py", line 69, in _wrap fn(i, args) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\train_ms.py", line 193, in run train_and_evaluate(rank, epoch, hps, [net_g, net_d, net_dur_disc], [optim_g, optim_d, optim_dur_disc], [scheduler_g, scheduler_d, scheduler_dur_disc], scaler, [train_loader, eval_loader], logger, [writer, writer_eval]) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\train_ms.py", line 231, in train_and_evaluate (z, z_p, m_p, logs_p, m_q, logs_q), (hiddenx, logw, logw) = net_g(x, x_lengths, spec, spec_lengths, speakers, tone, language, bert) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(input, kwargs) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\nn\parallel\distributed.py", line 1040, in forward output = self._run_ddp_forward(*inputs, *kwargs) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\nn\parallel\distributed.py", line 1000, in _run_ddp_forward return module_to_run(inputs[0], kwargs[0]) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(*input, kwargs) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\models.py", line 667, in forward l_length_sdp = self.sdp(x, x_mask, w, g=g) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(*input, *kwargs) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\models.py", line 177, in forward z_q, logdet_q = flow(z_q, x_mask, g=(x + h_w)) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl return forward_call(input, kwargs) File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\modules.py", line 374, in forward x1, logabsdet = piecewise_rational_quadratic_transform(x1, File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\transforms.py", line 33, in piecewise_rational_quadratic_transform outputs, logabsdet = spline_fn( File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\transforms.py", line 82, in unconstrained_rational_quadratic_spline outputs[inside_interval_mask], logabsdet[inside_interval_mask] = rational_quadratic_spline( File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\transforms.py", line 105, in rational_quadratic_spline if torch.min(inputs) < left or torch.max(inputs) > right: RuntimeError: min(): Expected reduction dim to be specified for input.numel() == 0. Specify the reduction dim with the 'dim' argument.
可以发一下上面的加载信息吗?
=========================
INFO:OUTPUT_MODEL:{'train': {'log_interval': 10, 'eval_interval': 30, 'seed': 52, 'epochs': 2000, 'learning_rate': 0.0002, 'betas': [0.8, 0.99], 'eps': 1e-09, 'batch_size': 5, 'fp16_run': False, 'lr_decay': 0.999875, 'segment_size': 16384, 'init_lr_ratio': 1, 'warmup_epochs': 0, 'c_mel': 45, 'c_kl': 1.0}, 'data': {'use_mel_posterior_encoder': False, 'training_files': 'filelists/train.list', 'validation_files': 'filelists/val.list', 'max_wav_value': 32768.0, 'sampling_rate': 44100, 'filter_length': 2048, 'hop_length': 512, 'win_length': 2048, 'n_mel_channels': 128, 'mel_fmin': 0.0, 'mel_fmax': None, 'add_blank': True, 'n_speakers': 1, 'cleaned_text': True, 'spk2id': {'zxcbert': 0}}, 'model': {'use_spk_conditioned_encoder': True, 'use_noise_scaled_mas': True, 'use_mel_posterior_encoder': False, 'use_duration_discriminator': True, 'inter_channels': 192, 'hidden_channels': 192, 'filter_channels': 768, 'n_heads': 2, 'n_layers': 6, 'kernel_size': 3, 'p_dropout': 0.1, 'resblock': '1', 'resblock_kernel_sizes': [3, 7, 11], 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'upsample_rates': [8, 8, 2, 2, 2], 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 8, 2, 2], 'n_layers_q': 3, 'use_spectral_norm': False, 'gin_channels': 256}, 'model_dir': './logs\./OUTPUT_MODEL', 'cont': True}
WARNING:OUTPUT_MODEL:D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package is not a git repository, therefore hash value comparison will be ignored.
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
skipped: 0 , total: 198
skipped: 0 , total: 4
Using noise scaled MAS for VITS2
Using duration discriminator for VITS2
256 2
256 2
256 2
256 2
256 2
./logs./OUTPUT_MODEL\DUR_600.pth
load
INFO:OUTPUT_MODEL:Loaded checkpoint './logs./OUTPUT_MODEL\DUR_600.pth' (iteration 15)
./logs./OUTPUT_MODEL\G_600.pth
load
INFO:OUTPUT_MODEL:Loaded checkpoint './logs./OUTPUT_MODEL\G_600.pth' (iteration 15)
./logs./OUTPUT_MODEL\D_600.pth
load
INFO:OUTPUT_MODEL:Loaded checkpoint './logs./OUTPUT_MODEL\D_600.pth' (iteration 15)
0it [00:00, ?it/s]D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\mel_processing.py:78: FutureWarning: Pass sr=44100, n_fft=2048, n_mels=128, fmin=0.0, fmax=None as keyword args. From version 0.10 passing these as positional arguments will result in an error
mel = librosa_mel_fn(sampling_rate, n_fft, num_mels, fmin, fmax)
[W C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\distributed\c10d\reducer.cpp:1305] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator ())
D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\autograd__init__.py:197: UserWarning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed. This is not an error, but may impair performance.
grad.sizes() = [1, 9, 96], strides() = [25632, 96, 1]
bucket_view.sizes() = [1, 9, 96], strides() = [864, 96, 1] (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\distributed\c10d\reducer.cpp:339.)
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
1it [00:06, 6.75s/it]
Traceback (most recent call last):
File "train_ms.py", line 402, in
-- Process 0 terminated with the following error: Traceback (most recent call last): File "D:\BaiduNetdiskDownload\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\multiprocessing\spawn.py", line 69, in _wrap 这是上面的信息
很奇怪的问题,但是我没有遇到过。搜索了一下这个错误,也是有人使用vits(1)过程中产生的,他通过降级torch解决了。但按理来说首次训练没报这个错 继续训练也不应该啊
那我降级试试能否解决
遇到同样的问题,报错信息如下:
加载config中的配置0
加载config中的配置localhost
加载config中的配置10086
加载config中的配置0
加载config中的配置1
加载环境变量
MASTER_ADDR: localhost,
MASTER_PORT: 10086,
WORLD_SIZE: 1,
RANK: 0,
LOCAL_RANK: 0
12-31 07:04:37 INFO | data_utils.py:63 | Init dataset...
100%|███████████████████████████████████████████████████████████████████████████| 1401/1401 [00:00<00:00, 17513.46it/s]
12-31 07:04:37 INFO | data_utils.py:78 | skipped: 0, total: 1401
12-31 07:04:37 INFO | data_utils.py:63 | Init dataset...
100%|████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<?, ?it/s]
12-31 07:04:37 INFO | data_utils.py:78 | skipped: 0, total: 7
Using noise scaled MAS for VITS2
Using duration discriminator for VITS2
INFO:models:Loaded checkpoint 'Data\ichika\models\DUR_600.pth' (iteration 3)
INFO:models:Loaded checkpoint 'Data\ichika\models\G_700.pth' (iteration 3)
PytorchStreamReader failed reading zip archive: failed finding central directory
INFO:models:Loaded checkpoint 'Data\ichika\models\WD_600.pth' (iteration 3)
Traceback (most recent call last):
File "train_ms.py", line 840, in <module>
run()
File "train_ms.py", line 338, in run
scheduler_d = torch.optim.lr_scheduler.ExponentialLR(
File "G:\AI\voice\train\Bert-VITS2-2.3一键包\venv\lib\site-packages\torch\optim\lr_scheduler.py", line 586, in __init__
super().__init__(optimizer, last_epoch, verbose)
File "G:\AI\voice\train\Bert-VITS2-2.3一键包\venv\lib\site-packages\torch\optim\lr_scheduler.py", line 42, in __init__
raise KeyError("param 'initial_lr' is not specified "
KeyError: "param 'initial_lr' is not specified in param_groups[0] when resuming an optimizer"
遇到同样的问题,报错信息如下:
加载config中的配置0 加载config中的配置localhost 加载config中的配置10086 加载config中的配置0 加载config中的配置1 加载环境变量 MASTER_ADDR: localhost, MASTER_PORT: 10086, WORLD_SIZE: 1, RANK: 0, LOCAL_RANK: 0 12-31 07:04:37 INFO | data_utils.py:63 | Init dataset... 100%|███████████████████████████████████████████████████████████████████████████| 1401/1401 [00:00<00:00, 17513.46it/s] 12-31 07:04:37 INFO | data_utils.py:78 | skipped: 0, total: 1401 12-31 07:04:37 INFO | data_utils.py:63 | Init dataset... 100%|████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:00<?, ?it/s] 12-31 07:04:37 INFO | data_utils.py:78 | skipped: 0, total: 7 Using noise scaled MAS for VITS2 Using duration discriminator for VITS2 INFO:models:Loaded checkpoint 'Data\ichika\models\DUR_600.pth' (iteration 3) INFO:models:Loaded checkpoint 'Data\ichika\models\G_700.pth' (iteration 3) PytorchStreamReader failed reading zip archive: failed finding central directory INFO:models:Loaded checkpoint 'Data\ichika\models\WD_600.pth' (iteration 3) Traceback (most recent call last): File "train_ms.py", line 840, in <module> run() File "train_ms.py", line 338, in run scheduler_d = torch.optim.lr_scheduler.ExponentialLR( File "G:\AI\voice\train\Bert-VITS2-2.3一键包\venv\lib\site-packages\torch\optim\lr_scheduler.py", line 586, in __init__ super().__init__(optimizer, last_epoch, verbose) File "G:\AI\voice\train\Bert-VITS2-2.3一键包\venv\lib\site-packages\torch\optim\lr_scheduler.py", line 42, in __init__ raise KeyError("param 'initial_lr' is not specified " KeyError: "param 'initial_lr' is not specified in param_groups[0] when resuming an optimizer"
问题已解决,是因为首次训练后直接把窗口关闭导致数据文件被损坏,把最新的文件删掉就好了
RuntimeError: min(): Expected reduction dim to be specified for input.numel() == 0. Specify the reduction dim with the 'dim' argument.