maum-ai / univnet

Unofficial PyTorch Implementation of UnivNet Vocoder (https://arxiv.org/abs/2106.07889)
https://mindslab-ai.github.io/univnet/
BSD 3-Clause "New" or "Revised" License
264 stars 46 forks source link

Training fail: EOFError: Ran out of input #6

Open icklerly1 opened 2 years ago

icklerly1 commented 2 years ago

Hi there, i am getting an error after 1 iteration of training and I cannot figure out the reason.

Do you have any idea what is causing the error EOFError: Ran out of input ? Thanks in advance!

The error looks like this: Loading train data: 0%| | 0/2732 [00:00<?, ?it/s] Traceback (most recent call last): File "trainer.py", line 44, in <module> train(0, args, args.checkpoint_path, hp, hp_str) File "/opt/3tbdrive1/products/voicesurfer/02.VoiceTraining/univnet/utils/train.py", line 125, in train for mel, audio in loader: File "/opt/3tbdrive1/products/voicesurfer/venv/lib/python3.6/site-packages/tqdm/std.py", line 1185, in __iter__ for obj in iterable: File "/opt/3tbdrive1/products/voicesurfer/venv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 521, in __next__ data = self._next_data() File "/opt/3tbdrive1/products/voicesurfer/venv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data return self._process_data(data) File "/opt/3tbdrive1/products/voicesurfer/venv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data data.reraise() File "/opt/3tbdrive1/products/voicesurfer/venv/lib/python3.6/site-packages/torch/_utils.py", line 434, in reraise raise exception EOFError: Caught EOFError in DataLoader worker process 0. Original Traceback (most recent call last): File "/opt/3tbdrive1/products/voicesurfer/venv/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop data = fetcher.fetch(index) File "/opt/3tbdrive1/products/voicesurfer/venv/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "/opt/3tbdrive1/products/voicesurfer/venv/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp> data = [self.dataset[idx] for idx in possibly_batched_index] File "/opt/3tbdrive1/products/voicesurfer/02.VoiceTraining/univnet/datasets/dataloader.py", line 61, in __getitem__ return self.my_getitem(idx) File "/opt/3tbdrive1/products/voicesurfer/02.VoiceTraining/univnet/datasets/dataloader.py", line 78, in my_getitem mel = self.get_mel(wavpath) File "/opt/3tbdrive1/products/voicesurfer/02.VoiceTraining/univnet/datasets/dataloader.py", line 96, in get_mel mel = torch.load(melpath, map_location='cpu') File "/opt/3tbdrive1/products/voicesurfer/venv/lib/python3.6/site-packages/torch/serialization.py", line 608, in load return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args) File "/opt/3tbdrive1/products/voicesurfer/venv/lib/python3.6/site-packages/torch/serialization.py", line 777, in _legacy_load magic_number = pickle_module.load(f, **pickle_load_args) EOFError: Ran out of input

Datasciensyash commented 2 years ago

It seems to be that you haven't prepared files, containing mel-spectrograms, but the dataset trying to load it from disk.

In this line https://github.com/mindslab-ai/univnet/blob/9bb2b54838bb6d7ce767131cc7b8b61198bc7558/datasets/dataloader.py#L100 Authors generate mel-spectrogram in runtime if loading fails, but it seems they have the wrong try-except statement.

Change this line to:

except (FileNotFoundError, RuntimeError, TypeError, AssertionError, EOFError):

I think it should work.