Hi.
When training LibriTTS VALL-E with prefix mode 4, it causes pickle-load issue.
Can you solve this problem?
The error is in the below.
I think the error is caused from defaultdict.
Traceback (most recent call last):
File "/home/work/tts/user/Model/TTS/VALL-E/vall-e.kt/egs/LibriTTS/bin/trainer.py", line 1214, in <module>
main()
File "/home/work/tts/user/Model/TTS/VALL-E/vall-e.kt/egs/LibriTTS/bin/trainer.py", line 1205, in main
mp.spawn(run, args=(world_size, args), nprocs=world_size, join=True)
File "/root/miniconda3/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/root/miniconda3/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
while not context.join():
File "/root/miniconda3/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:
-- Process 1 terminated with the following error:
Traceback (most recent call last):
File "/root/miniconda3/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
fn(i, *args)
File "/home/work/tts/user/Model/TTS/VALL-E/vall-e.kt/egs/LibriTTS/bin/trainer.py", line 1093, in run
train_one_epoch(
File "/home/work/tts/user/Model/TTS/VALL-E/vall-e.kt/egs/LibriTTS/bin/trainer.py", line 673, in train_one_epoch
iter_dl = iter(train_dl)
File "/root/miniconda3/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 435, in __iter__
return self._get_iterator()
File "/root/miniconda3/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 381, in _get_iterator
return _MultiProcessingDataLoaderIter(self)
File "/root/miniconda3/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1034, in __init__
w.start()
File "/root/miniconda3/lib/python3.10/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/root/miniconda3/lib/python3.10/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/root/miniconda3/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
return Popen(process_obj)
File "/root/miniconda3/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in __init__
super().__init__(process_obj)
File "/root/miniconda3/lib/python3.10/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/root/miniconda3/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 47, in _launch
reduction.dump(process_obj, fp)
File "/root/miniconda3/lib/python3.10/multiprocessing/reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'PromptedPrecomputedFeatures.__init__.<locals>.<lambda>'
Hi. When training LibriTTS VALL-E with prefix mode 4, it causes pickle-load issue. Can you solve this problem? The error is in the below. I think the error is caused from defaultdict.