Closed sanshaoyeyang closed 1 year ago
我也是这样,error, emb_g.weight is not in the checkpoint
同问qwq
经过痛苦的反复尝试,包括不限于更换pytorch、cuda、代码修改,都不起作用 最终换了个小数据集实验,居然跑通了,问题应该是某段数据被跳过导致读取了空数据引起的 换了个数据集之后重新跑一遍,训练代码终于正常跑起来了,问题完结!
感谢,我在减小数据集到十几条的时候也跑起来了。不过正常训练不能只有这么小的数据集,接下来我还要再看看到底是哪段数据的缺失,给代码补上跳过的机制
我也是这样,error, emb_g.weight is not in the checkpoint
这不是错误,首次训练你的说话人和底膜对不上就会这样,是正常的
经过痛苦的反复尝试,包括不限于更换pytorch、cuda、代码修改,都不起作用 最终换了个小数据集实验,居然跑通了,问题应该是某段数据被跳过导致读取了空数据引起的 换了个数据集之后重新跑一遍,训练代码终于正常跑起来了,问题完结!
我怀疑是num_workers的数量问题,我设为4了。你的显卡是桌面4090对吗?那么方便说一下内存多大,数据集长度有没有过长呢,有没有爆显存呢?我自己16G显存+64G内存在我这个设定下是能训没分割的纳西妲语音的(1577条,最长的有30秒)你把它设为别的数,比如2或者0试试看?
经过痛苦的反复尝试,包括不限于更换pytorch、cuda、代码修改,都不起作用 最终换了个小数据集实验,居然跑通了,问题应该是某段数据被跳过导致读取了空数据引起的 换了个数据集之后重新跑一遍,训练代码终于正常跑起来了,问题完结!
AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute '_shutdown'这个错误搜了一下似乎是内存不足导致的,如果可以的话能不能复现一下错误再记录一下内存使用啊?如果物理内存没满是不是虚拟内存不够?原作者设置的24我在第一次跑的时候已提交内存已经超过128G了
一模一样的报错,已经搞了三天了,还是没解决,3090 24G显存,内存64G,虚拟内存300G,10s一段的音频共50段,总时长8分多钟,只要输入训练命令就是一模一样的报错,环境已更新到了最新
补充: 用的是最新的,2023年9月3日晚上九点传的那个版本 采样率44.1
E:\VITS_Bert>%PYTHON% train_first.py -c ./configs\config.json
INFO:OUTPUT_MODEL:{'train': {'log_interval': 10, 'eval_interval': 100, 'seed': 52, 'epochs': 1000, 'learning_rate': 0.0002, 'betas': [0.8, 0.99], 'eps': 1e-09, 'batch_size': 10, 'fp16_run': False, 'lr_decay': 0.999875, 'segment_size': 16384, 'init_lr_ratio': 1, 'warmup_epochs': 0, 'c_mel': 45, 'c_kl': 1.0}, 'data': {'use_mel_posterior_encoder': False, 'training_files': 'filelists/train.list', 'validation_files': 'filelists/val.list', 'max_wav_value': 32768.0, 'sampling_rate': 44100, 'filter_length': 2048, 'hop_length': 512, 'win_length': 2048, 'n_mel_channels': 128, 'mel_fmin': 0.0, 'mel_fmax': None, 'add_blank': True, 'n_speakers': 1, 'cleaned_text': True, 'spk2id': {'Warma': 0}}, 'model': {'use_spk_conditioned_encoder': True, 'use_noise_scaled_mas': True, 'use_mel_posterior_encoder': False, 'use_duration_discriminator': True, 'inter_channels': 192, 'hidden_channels': 192, 'filter_channels': 768, 'n_heads': 2, 'n_layers': 6, 'kernel_size': 3, 'p_dropout': 0.1, 'resblock': '1', 'resblock_kernel_sizes': [3, 7, 11], 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'upsample_rates': [8, 8, 2, 2, 2], 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 8, 2, 2], 'n_layers_q': 3, 'use_spectral_norm': False, 'gin_channels': 256}, 'model_dir': './logs\./OUTPUT_MODEL'}
WARNING:OUTPUT_MODEL:E:\VITS_Bert is not a git repository, therefore hash value comparison will be ignored.
INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
skipped: 11 , total: 230
skipped: 0 , total: 4
Using noise scaled MAS for VITS2
Using duration discriminator for VITS2
256 2
256 2
256 2
256 2
256 2
./logs./OUTPUT_MODEL\DUR_0.pth
load
INFO:OUTPUT_MODEL:Loaded checkpoint './logs./OUTPUT_MODEL\DUR_0.pth' (iteration 694)
./logs./OUTPUT_MODEL\G_0.pth
error, emb_g.weight is not in the checkpoint
load
INFO:OUTPUT_MODEL:Loaded checkpoint './logs./OUTPUT_MODEL\G_0.pth' (iteration 0)
./logs./OUTPUT_MODEL\D_0.pth
load
INFO:OUTPUT_MODEL:Loaded checkpoint './logs./OUTPUT_MODEL\D_0.pth' (iteration 0)
Exception ignored in: <function _MultiProcessingDataLoaderIter.del at 0x0000023BDCBB5AF0>
Traceback (most recent call last):
File "E:\VITS_Bert\venv\lib\site-packages\torch\utils\data\dataloader.py", line 1466, in del
self._shutdown_workers()
File "E:\VITS_Bert\venv\lib\site-packages\torch\utils\data\dataloader.py", line 1397, in _shutdown_workers
if not self._shutdown:
AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute '_shutdown'
Traceback (most recent call last):
File "train_first.py", line 409, in
-- Process 0 terminated with the following error: Traceback (most recent call last): File "E:\VITS_Bert\venv\lib\site-packages\torch\multiprocessing\spawn.py", line 69, in _wrap fn(i, args) File "E:\VITS_Bert\train_first.py", line 191, in run train_and_evaluate(rank, epoch, hps, [net_g, net_d, net_dur_disc], [optim_g, optim_d, optim_dur_disc], [scheduler_g, scheduler_d, scheduler_dur_disc], scaler, [train_loader, eval_loader], logger, [writer, writer_eval]) File "E:\VITS_Bert\train_first.py", line 215, in train_and_evaluate for batch_idx, (x, x_lengths, spec, spec_lengths, y, y_lengths, speakers, tone, language, bert) in tqdm(enumerate(train_loader)): File "E:\VITS_Bert\venv\lib\site-packages\torch\utils\data\dataloader.py", line 430, in iter self._iterator = self._get_iterator() File "E:\VITS_Bert\venv\lib\site-packages\torch\utils\data\dataloader.py", line 381, in _get_iterator return _MultiProcessingDataLoaderIter(self) File "E:\VITS_Bert\venv\lib\site-packages\torch\utils\data\dataloader.py", line 988, in init super(_MultiProcessingDataLoaderIter, self).init(loader) File "E:\VITS_Bert\venv\lib\site-packages\torch\utils\data\dataloader.py", line 598, in init self._sampler_iter = iter(self._index_sampler) File "E:\VITS_Bert\data_utils.py", line 309, in iter ids_bucket = ids_bucket + ids_bucket (rem // len_bucket) + ids_bucket[:(rem % len_bucket)] ZeroDivisionError: integer division or modulo by zero
从输入指令到结尾报错全都在这,3090卡24G显存,win10 128G内存,数据集10s以下包括10s一共有246条,标注的时候模型用的medium,config下的batch_size是10 没有明白为什么报错。
尝试过其他TTS项目:VITS fast-fine-tuning 也有很严重的问题,同上的数据集跑一会就torch.multiprocessing.spawn.ProcessExitedException: process 0 terminated with exit code 3221225477,改小过batch_size为2、4、8都不行,模型也是medium。但至少还能练一点才炸,这个完全不知道什么情况,直接跑不起来
补充: 用的是最新的,2023年9月3日晚上九点传的那个版本 采样率44.1
E:\VITS_Bert>%PYTHON% train_first.py -c ./configs\config.json INFO:OUTPUT_MODEL:{'train': {'log_interval': 10, 'eval_interval': 100, 'seed': 52, 'epochs': 1000, 'learning_rate': 0.0002, 'betas': [0.8, 0.99], 'eps': 1e-09, 'batch_size': 10, 'fp16_run': False, 'lr_decay': 0.999875, 'segment_size': 16384, 'init_lr_ratio': 1, 'warmup_epochs': 0, 'c_mel': 45, 'c_kl': 1.0}, 'data': {'use_mel_posterior_encoder': False, 'training_files': 'filelists/train.list', 'validation_files': 'filelists/val.list', 'max_wav_value': 32768.0, 'sampling_rate': 44100, 'filter_length': 2048, 'hop_length': 512, 'win_length': 2048, 'n_mel_channels': 128, 'mel_fmin': 0.0, 'mel_fmax': None, 'add_blank': True, 'n_speakers': 1, 'cleaned_text': True, 'spk2id': {'Warma': 0}}, 'model': {'use_spk_conditioned_encoder': True, 'use_noise_scaled_mas': True, 'use_mel_posterior_encoder': False, 'use_duration_discriminator': True, 'inter_channels': 192, 'hidden_channels': 192, 'filter_channels': 768, 'n_heads': 2, 'n_layers': 6, 'kernel_size': 3, 'p_dropout': 0.1, 'resblock': '1', 'resblock_kernel_sizes': [3, 7, 11], 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'upsample_rates': [8, 8, 2, 2, 2], 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 8, 2, 2], 'n_layers_q': 3, 'use_spectral_norm': False, 'gin_channels': 256}, 'model_dir': './logs./OUTPUT_MODEL'} WARNING:OUTPUT_MODEL:E:\VITS_Bert is not a git repository, therefore hash value comparison will be ignored. INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. skipped: 11 , total: 230 skipped: 0 , total: 4 Using noise scaled MAS for VITS2 Using duration discriminator for VITS2 256 2 256 2 256 2 256 2 256 2 ./logs./OUTPUT_MODEL\DUR_0.pth load INFO:OUTPUT_MODEL:Loaded checkpoint './logs./OUTPUT_MODEL\DUR_0.pth' (iteration 694) ./logs./OUTPUT_MODEL\G_0.pth error, emb_g.weight is not in the checkpoint load INFO:OUTPUT_MODEL:Loaded checkpoint './logs./OUTPUT_MODEL\G_0.pth' (iteration 0) ./logs./OUTPUT_MODEL\D_0.pth load INFO:OUTPUT_MODEL:Loaded checkpoint './logs./OUTPUT_MODEL\D_0.pth' (iteration 0) Exception ignored in: <function _MultiProcessingDataLoaderIter.del at 0x0000023BDCBB5AF0> Traceback (most recent call last): File "E:\VITS_Bert\venv\lib\site-packages\torch\utils\data\dataloader.py", line 1466, in del self._shutdown_workers() File "E:\VITS_Bert\venv\lib\site-packages\torch\utils\data\dataloader.py", line 1397, in _shutdown_workers if not self._shutdown: AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute '_shutdown' Traceback (most recent call last): File "train_first.py", line 409, in main() File "train_first.py", line 58, in main mp.spawn(run, nprocs=n_gpus, args=(n_gpus, hps,)) File "E:\VITS_Bert\venv\lib\site-packages\torch\multiprocessing\spawn.py", line 240, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') File "E:\VITS_Bert\venv\lib\site-packages\torch\multiprocessing\spawn.py", line 198, in start_processes while not context.join(): File "E:\VITS_Bert\venv\lib\site-packages\torch\multiprocessing\spawn.py", line 160, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 0 terminated with the following error: Traceback (most recent call last): File "E:\VITS_Bert\venv\lib\site-packages\torch\multiprocessing\spawn.py", line 69, in _wrap fn(i, args) File "E:\VITS_Bert\train_first.py", line 191, in run train_and_evaluate(rank, epoch, hps, [net_g, net_d, net_dur_disc], [optim_g, optim_d, optim_dur_disc], [scheduler_g, scheduler_d, scheduler_dur_disc], scaler, [train_loader, eval_loader], logger, [writer, writer_eval]) File "E:\VITS_Bert\train_first.py", line 215, in train_and_evaluate for batch_idx, (x, x_lengths, spec, spec_lengths, y, y_lengths, speakers, tone, language, bert) in tqdm(enumerate(train_loader)): File "E:\VITS_Bert\venv\lib\site-packages\torch\utils\data\dataloader.py", line 430, in iter self._iterator = self._get_iterator() File "E:\VITS_Bert\venv\lib\site-packages\torch\utils\data\dataloader.py", line 381, in _get_iterator return _MultiProcessingDataLoaderIter(self) File "E:\VITS_Bert\venv\lib\site-packages\torch\utils\data\dataloader.py", line 988, in init super(_MultiProcessingDataLoaderIter, self).init(loader) File "E:\VITS_Bert\venv\lib\site-packages\torch\utils\data\dataloader.py", line 598, in init self._sampler_iter = iter(self._index_sampler) File "E:\VITS_Bert\data_utils.py", line 309, in iter ids_bucket = ids_bucket + ids_bucket (rem // len_bucket) + ids_bucket[:(rem % len_bucket)] ZeroDivisionError: integer division or modulo by zero
从输入指令到结尾报错全都在这,3090卡24G显存,win10 128G内存,数据集10s以下包括10s一共有246条,标注的时候模型用的medium,config下的batch_size是10 没有明白为什么报错。
尝试过其他TTS项目:VITS fast-fine-tuning 也有很严重的问题,同上的数据集跑一会就torch.multiprocessing.spawn.ProcessExitedException: process 0 terminated with exit code 3221225477,改小过batch_size为2、4、8都不行,模型也是medium。但至少还能练一点才炸,这个完全不知道什么情况,直接跑不起来
9.3日的版本训练已经改用train_ms.py了,其他文件也请更新一下,不再使用的模块记得删掉
经过痛苦的反复尝试,包括不限于更换pytorch、cuda、代码修改,都不起作用 最终换了个小数据集实验,居然跑通了,问题应该是某段数据被跳过导致读取了空数据引起的 换了个数据集之后重新跑一遍,训练代码终于正常跑起来了,问题完结!
一模一样的经历, 我也是把数据集改小了就好了, 之前一直报错channel不匹配, 真的烦死我了, 要是找点看到这个就好了
WARNING:OUTPUT_MODEL:D:\AI\Bert-VITS2-Integration-Package is not a git repository, therefore hash value comparison will be ignored. INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0 INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes. skipped: 1 , total: 500 skipped: 0 , total: 4 Using noise scaled MAS for VITS2 Using duration discriminator for VITS2 256 2 256 2 256 2 256 2 256 2 ./logs./OUTPUT_MODEL\DUR_0.pth load INFO:OUTPUT_MODEL:Loaded checkpoint './logs./OUTPUT_MODEL\DUR_0.pth' (iteration 694) ./logs./OUTPUT_MODEL\G_0.pth error, emb_g.weight is not in the checkpoint load INFO:OUTPUT_MODEL:Loaded checkpoint './logs./OUTPUT_MODEL\G_0.pth' (iteration 0) ./logs./OUTPUT_MODEL\D_0.pth load INFO:OUTPUT_MODEL:Loaded checkpoint './logs./OUTPUT_MODEL\D_0.pth' (iteration 0) Exception ignored in: <function _MultiProcessingDataLoaderIter.del at 0x000002FBBA6BA790> Traceback (most recent call last): File "D:\AI\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\utils\data\dataloader.py", line 1466, in del self._shutdown_workers() File "D:\AI\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\utils\data\dataloader.py", line 1397, in _shutdown_workers if not self._shutdown: AttributeError: '_MultiProcessingDataLoaderIter' object has no attribute '_shutdown' Traceback (most recent call last): File "train_first.py", line 409, in
main()
File "train_first.py", line 58, in main
mp.spawn(run, nprocs=n_gpus, args=(n_gpus, hps,))
File "D:\AI\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\multiprocessing\spawn.py", line 240, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "D:\AI\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\multiprocessing\spawn.py", line 198, in start_processes
while not context.join():
File "D:\AI\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\multiprocessing\spawn.py", line 160, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 0 terminated with the following error: Traceback (most recent call last): File "D:\AI\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\multiprocessing\spawn.py", line 69, in _wrap fn(i, args) File "D:\AI\Bert-VITS2-Integration-Package\train_first.py", line 191, in run train_and_evaluate(rank, epoch, hps, [net_g, net_d, net_dur_disc], [optim_g, optim_d, optim_dur_disc], [scheduler_g, scheduler_d, scheduler_dur_disc], scaler, [train_loader, eval_loader], logger, [writer, writer_eval]) File "D:\AI\Bert-VITS2-Integration-Package\train_first.py", line 215, in train_and_evaluate for batch_idx, (x, x_lengths, spec, spec_lengths, y, y_lengths, speakers, tone, language, bert) in tqdm(enumerate(train_loader)): File "D:\AI\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\utils\data\dataloader.py", line 430, in iter self._iterator = self._get_iterator() File "D:\AI\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\utils\data\dataloader.py", line 381, in _get_iterator return _MultiProcessingDataLoaderIter(self) File "D:\AI\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\utils\data\dataloader.py", line 988, in init super(_MultiProcessingDataLoaderIter, self).init(loader) File "D:\AI\Bert-VITS2-Integration-Package\venv\lib\site-packages\torch\utils\data\dataloader.py", line 598, in init self._sampler_iter = iter(self._index_sampler) File "D:\AI\Bert-VITS2-Integration-Package\data_utils.py", line 309, in iter ids_bucket = ids_bucket + ids_bucket (rem // len_bucket) + ids_bucket[:(rem % len_bucket)] ZeroDivisionError: integer division or modulo by zero