search

Plachtaa / VITS-fast-fine-tuning

This repo is a pipeline of VITS finetuning for fast speaker adaptation TTS, and many-to-many voice conversion
Apache License 2.0
4.76k stars 718 forks source link

STEP 4報錯「找不到音檔」：No such file or directory: '\ufeff./custom_character_voice/Reiya/processed_0.wav' #583

Open CyberSaga opened 8 months ago

CyberSaga commented 8 months ago

跑到 「STEP 4 开始训练 」就會報錯說找不到處理好的音檔，TensorBoard雖然有跳出來但一直是不能用的 INACTIVE。

錯誤訊息如下：

Traceback (most recent call last):
  File "/content/VITS-fast-fine-tuning/scripts/rearrange_speaker.py", line 11, in <module>
    model_sd = torch.load(args.model_dir, map_location='cpu')
  File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 998, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 445, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 426, in __init__
    super().__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: './OUTPUT_MODEL/G_latest.pth'
cp: cannot stat './finetune_speaker.json': No such file or directory
cp: cannot stat './moegoe_config.json': No such file or directory
ERROR: Failed to launch TensorBoard (exited with 2).
Contents of stderr:
2024-03-19 09:01:22.430077: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-03-19 09:01:22.430153: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-03-19 09:01:22.431434: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-03-19 09:01:22.438849: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-19 09:01:23.463861: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
usage: tensorboard [-h] [--helpfull] {serve,dev} ...
tensorboard: error: argument {serve,dev}: invalid choice: 'tf.debugging.experimental.enable_dump_debug_info(logdir,' (choose from 'serve', 'dev')
2024-03-19 09:01:27.732144: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-03-19 09:01:27.732203: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-03-19 09:01:27.733495: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-03-19 09:01:27.740731: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-03-19 09:01:28.835411: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
DEBUG:torio._extension.utils:Loading FFmpeg6
DEBUG:torio._extension.utils:Failed to load FFmpeg6 extension.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torio/_extension/utils.py", line 116, in _find_ffmpeg_extension
    ext = _find_versionsed_ffmpeg_extension(ffmpeg_ver)
  File "/usr/local/lib/python3.10/dist-packages/torio/_extension/utils.py", line 108, in _find_versionsed_ffmpeg_extension
    _load_lib(lib)
  File "/usr/local/lib/python3.10/dist-packages/torio/_extension/utils.py", line 94, in _load_lib
    torch.ops.load_library(path)
  File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 933, in load_library
    ctypes.CDLL(path)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libavutil.so.58: cannot open shared object file: No such file or directory
DEBUG:torio._extension.utils:Loading FFmpeg5
DEBUG:torio._extension.utils:Failed to load FFmpeg5 extension.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torio/_extension/utils.py", line 116, in _find_ffmpeg_extension
    ext = _find_versionsed_ffmpeg_extension(ffmpeg_ver)
  File "/usr/local/lib/python3.10/dist-packages/torio/_extension/utils.py", line 108, in _find_versionsed_ffmpeg_extension
    _load_lib(lib)
  File "/usr/local/lib/python3.10/dist-packages/torio/_extension/utils.py", line 94, in _load_lib
    torch.ops.load_library(path)
  File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 933, in load_library
    ctypes.CDLL(path)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libavutil.so.57: cannot open shared object file: No such file or directory
DEBUG:torio._extension.utils:Loading FFmpeg4
DEBUG:torio._extension.utils:Successfully loaded FFmpeg4
2024-03-19 09:01:35.987515: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-03-19 09:01:35.987561: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-03-19 09:01:35.988836: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-03-19 09:01:37.153974: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
DEBUG:torio._extension.utils:Loading FFmpeg6
DEBUG:torio._extension.utils:Failed to load FFmpeg6 extension.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torio/_extension/utils.py", line 116, in _find_ffmpeg_extension
    ext = _find_versionsed_ffmpeg_extension(ffmpeg_ver)
  File "/usr/local/lib/python3.10/dist-packages/torio/_extension/utils.py", line 108, in _find_versionsed_ffmpeg_extension
    _load_lib(lib)
  File "/usr/local/lib/python3.10/dist-packages/torio/_extension/utils.py", line 94, in _load_lib
    torch.ops.load_library(path)
  File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 933, in load_library
    ctypes.CDLL(path)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libavutil.so.58: cannot open shared object file: No such file or directory
DEBUG:torio._extension.utils:Loading FFmpeg5
DEBUG:torio._extension.utils:Failed to load FFmpeg5 extension.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torio/_extension/utils.py", line 116, in _find_ffmpeg_extension
    ext = _find_versionsed_ffmpeg_extension(ffmpeg_ver)
  File "/usr/local/lib/python3.10/dist-packages/torio/_extension/utils.py", line 108, in _find_versionsed_ffmpeg_extension
    _load_lib(lib)
  File "/usr/local/lib/python3.10/dist-packages/torio/_extension/utils.py", line 94, in _load_lib
    torch.ops.load_library(path)
  File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 933, in load_library
    ctypes.CDLL(path)
  File "/usr/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libavutil.so.57: cannot open shared object file: No such file or directory
DEBUG:torio._extension.utils:Loading FFmpeg4
DEBUG:torio._extension.utils:Successfully loaded FFmpeg4
INFO:OUTPUT_MODEL:{'train': {'log_interval': 10, 'eval_interval': 100, 'seed': 1234, 'epochs': 10000, 'learning_rate': 0.0002, 'betas': [0.8, 0.99], 'eps': 1e-09, 'batch_size': 16, 'fp16_run': True, 'lr_decay': 0.999875, 'segment_size': 8192, 'init_lr_ratio': 1, 'warmup_epochs': 0, 'c_mel': 45, 'c_kl': 1.0}, 'data': {'training_files': 'final_annotation_train.txt', 'validation_files': 'final_annotation_val.txt', 'text_cleaners': ['zh_ja_mixture_cleaners'], 'max_wav_value': 32768.0, 'sampling_rate': 22050, 'filter_length': 1024, 'hop_length': 256, 'win_length': 1024, 'n_mel_channels': 80, 'mel_fmin': 0.0, 'mel_fmax': None, 'add_blank': True, 'n_speakers': 1, 'cleaned_text': True}, 'model': {'inter_channels': 192, 'hidden_channels': 192, 'filter_channels': 768, 'n_heads': 2, 'n_layers': 6, 'kernel_size': 3, 'p_dropout': 0.1, 'resblock': '1', 'resblock_kernel_sizes': [3, 7, 11], 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'upsample_rates': [8, 8, 2, 2], 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4], 'n_layers_q': 3, 'use_spectral_norm': False, 'gin_channels': 256}, 'speakers': {'Reiya': 0}, 'symbols': ['_', ',', '.', '!', '?', '-', '~', '…', 'A', 'E', 'I', 'N', 'O', 'Q', 'U', 'a', 'b', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'r', 's', 't', 'u', 'v', 'w', 'y', 'z', 'ʃ', 'ʧ', 'ʦ', 'ɯ', 'ɹ', 'ə', 'ɥ', '⁼', 'ʰ', '`', '→', '↓', '↑', ' '], 'model_dir': '././OUTPUT_MODEL', 'max_epochs': 100, 'cont': False, 'drop_speaker_embed': True, 'train_with_pretrained_model': True, 'preserved': 4}
Traceback (most recent call last):
  File "/content/VITS-fast-fine-tuning/finetune_speaker_v2.py", line 372, in <module>
    main()
  File "/content/VITS-fast-fine-tuning/finetune_speaker_v2.py", line 55, in main
    mp.spawn(run, nprocs=n_gpus, args=(n_gpus, hps,))
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 241, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 197, in start_processes
    while not context.join():
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 158, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/spawn.py", line 68, in _wrap
    fn(i, *args)
  File "/content/VITS-fast-fine-tuning/finetune_speaker_v2.py", line 73, in run
    train_dataset = TextAudioSpeakerLoader(hps.data.training_files, hps.data, symbols)
  File "/content/VITS-fast-fine-tuning/data_utils.py", line 42, in __init__
    self._filter()
  File "/content/VITS-fast-fine-tuning/data_utils.py", line 59, in _filter
    lengths.append(os.path.getsize(audiopath) // (2 * self.hop_length))
  File "/usr/lib/python3.10/genericpath.py", line 50, in getsize
    return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '\ufeff./custom_character_voice/Reiya/processed_0.wav'

我確認過檔案有存在，如下圖：

colab檔案結構

另外，錯誤訊息偶爾會改成報

No such file or directory: './custom_character_voice/Reiya/processed_72.wav'

(開頭沒有\ufeff也還是找不到) 所以我想問題可能並不是在於一兩個檔案或是路徑壞掉吧 有沒有大神願意幫我解惑呢

fesolla commented 6 months ago

之前有類似的狀況，我發現如果STEP3時Whisper有抓到音檔的內容並且寫出台詞，那麼STEP4就不會發生問題