トレーニングに失敗する：torch.distributed.DistNetworkError: Unknown error

liruk commented 10 months ago

表題の通り、データセット作成及び自動前処理を行ったあと、「学習を開始する」を押下すると、トレーニングが行われない問題が発生しています。

手順

zipファイル内、Install-Style-Bert-VITS2.batでインストールを行う
Dataset.batから起動するウィンドウで、スライス及び書き起こしを行う。
Train.batから起動するウィンドウで、自動前処理を行う。
「学習を開始する」を押下すると、エラーが発生する。

以下、Train.batのログ出力です。

Running webui_train.py...
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
02-02 08:44:47 |  INFO  | webui_train.py:44 | Step 1: start initialization...
model_name: maid-gentle, batch_size: 4, epochs: 100, save_every_steps: 1000, bf16_run: True
02-02 08:44:48 |SUCCESS | webui_train.py:91 | Step 1: initialization finished.
02-02 08:44:48 |  INFO  | webui_train.py:96 | Step 2: start resampling...
02-02 08:44:48 |  INFO  | subprocess_utils.py:11 | Running: resample.py --in_dir Data\maid-gentle\raw --out_dir Data\maid-gentle\wavs --num_processes 8 --sr 44100
100%|████████████████████████████████████████████████████████████████████████████████| 152/152 [00:11<00:00, 13.65it/s]
02-02 08:45:00 |  INFO  | resample.py:122 | Resampling Done!
02-02 08:45:00 |SUCCESS | subprocess_utils.py:24 | Success: resample.py --in_dir Data\maid-gentle\raw --out_dir Data\maid-gentle\wavs --num_processes 8 --sr 44100
02-02 08:45:00 |SUCCESS | webui_train.py:122 | Step 2: resampling finished.
02-02 08:45:00 |  INFO  | webui_train.py:127 | Step 3: start preprocessing text...
02-02 08:45:00 |  INFO  | subprocess_utils.py:11 | Running: preprocess_text.py --config-path Data\maid-gentle\config.json --transcription-path Data\maid-gentle\esd.list --train-path Data\maid-gentle\train.list --val-path Data\maid-gentle\val.list
0it [00:00, ?it/s]Downloading: "https://github.com/r9y9/open_jtalk/releases/download/v1.11.1/open_jtalk_dic_utf_8-1.11.tar.gz"
Extracting tar file H:\Style-Bert-VITS2\Style-Bert-VITS2\venv\lib\site-packages\pyopenjtalk\dic.tar.gz
152it [00:02, 62.64it/s]
02-02 08:45:07 |  INFO  | preprocess_text.py:100 | Total repeated audios: 0, Total number of audio not found: 0
02-02 08:45:07 |  INFO  | preprocess_text.py:138 | Training set and validation set generation from texts is complete!
02-02 08:45:08 |WARNING | subprocess_utils.py:22 | Warning: preprocess_text.py --config-path Data\maid-gentle\config.json --transcription-path Data\maid-gentle\esd.list --train-path Data\maid-gentle\train.list --val-path Data\maid-gentle\val.list

dic.tar.gz: 0.00B [00:00, ?B/s]
dic.tar.gz:   0%|          | 8.00k/22.6M [00:01<57:25, 6.86kB/s]
dic.tar.gz:  11%|#         | 2.43M/22.6M [00:01<00:07, 2.72MB/s]
dic.tar.gz:  37%|###6      | 8.25M/22.6M [00:01<00:01, 10.4MB/s]
dic.tar.gz: 100%|##########| 22.6M/22.6M [00:01<00:00, 14.3MB/s]

02-02 08:45:08 |WARNING | webui_train.py:158 | Step 3: preprocessing text finished with stderr.
02-02 08:45:08 |  INFO  | webui_train.py:165 | Step 4: start bert_gen...
02-02 08:45:08 |  INFO  | subprocess_utils.py:11 | Running: bert_gen.py --config Data\maid-gentle\config.json
100%|████████████████████████████████████████████████████████████████████████████████| 152/152 [00:08<00:00, 17.94it/s]
02-02 08:45:18 |  INFO  | bert_gen.py:85 | bert.pt is generated! total: 152 bert.pt files.
02-02 08:45:19 |SUCCESS | subprocess_utils.py:24 | Success: bert_gen.py --config Data\maid-gentle\config.json
02-02 08:45:19 |SUCCESS | webui_train.py:182 | Step 4: bert_gen finished.
02-02 08:45:19 |  INFO  | webui_train.py:187 | Step 5: start style_gen...
02-02 08:45:19 |  INFO  | subprocess_utils.py:11 | Running: style_gen.py --config Data\maid-gentle\config.json --num_processes 8
100%|████████████████████████████████████████████████████████████████████████████████| 152/152 [00:02<00:00, 72.48it/s]
02-02 08:45:27 |  INFO  | style_gen.py:77 | Finished generating style vectors! total: 152 npy files.
02-02 08:45:27 |SUCCESS | subprocess_utils.py:24 | Success: style_gen.py --config Data\maid-gentle\config.json --num_processes 8
02-02 08:45:27 |SUCCESS | webui_train.py:204 | Step 5: style_gen finished.
02-02 08:45:27 |SUCCESS | webui_train.py:237 | Success: All preprocess finished!
02-02 08:46:36 |  INFO  | subprocess_utils.py:11 | Running: train_ms.py --config Data\maid-gentle\config.json --model Data\maid-gentle
02-02 08:46:39 |  INFO  | train_ms.py:93 | Loading configuration from config 0
02-02 08:46:39 |  INFO  | train_ms.py:93 | Loading configuration from config localhost
02-02 08:46:39 |  INFO  | train_ms.py:93 | Loading configuration from config 10086
02-02 08:46:39 |  INFO  | train_ms.py:93 | Loading configuration from config 0
02-02 08:46:39 |  INFO  | train_ms.py:93 | Loading configuration from config 1
02-02 08:46:39 |  INFO  | train_ms.py:95 | Loading environment variables
MASTER_ADDR: localhost,
MASTER_PORT: 10086,
WORLD_SIZE: 1,
RANK: 0,
LOCAL_RANK: 0
02-02 08:46:40 | ERROR  | subprocess_utils.py:19 | Error: train_ms.py --config Data\maid-gentle\config.json --model Data\maid-gentle
Traceback (most recent call last):
  File "H:\Style-Bert-VITS2\Style-Bert-VITS2\train_ms.py", line 867, in <module>
    run()
  File "H:\Style-Bert-VITS2\Style-Bert-VITS2\train_ms.py", line 108, in run
    dist.init_process_group(
  File "H:\Style-Bert-VITS2\Style-Bert-VITS2\venv\lib\site-packages\torch\distributed\c10d_logger.py", line 86, in wrapper
    func_return = func(*args, **kwargs)
  File "H:\Style-Bert-VITS2\Style-Bert-VITS2\venv\lib\site-packages\torch\distributed\distributed_c10d.py", line 1177, in init_process_group
    store, rank, world_size = next(rendezvous_iterator)
  File "H:\Style-Bert-VITS2\Style-Bert-VITS2\venv\lib\site-packages\torch\distributed\rendezvous.py", line 246, in _env_rendezvous_handler
    store = _create_c10d_store(master_addr, master_port, rank, world_size, timeout, use_libuv)
  File "H:\Style-Bert-VITS2\Style-Bert-VITS2\venv\lib\site-packages\torch\distributed\rendezvous.py", line 174, in _create_c10d_store
    return TCPStore(
torch.distributed.DistNetworkError: Unknown error

02-02 08:46:40 | ERROR  | webui_train.py:254 | Train failed.

litagin02 commented 10 months ago

ありがとうございます。同様のエラーはこちらでも確認しております。PyTorchのバージョンが2.2.0に上がったことが原因っぽいので、どう修正するかを検討中です。

litagin02 commented 10 months ago

お手数をおかけしますが、コマンドプロンプトでStyle-Bert-VITS2のフォルダを開いて、以下のコマンドを順に入力してみて終了してから再び学習できるか確かめていただけませんでしょうか。

venv\Scripts\activate
pip uninstall torch torchaudio torchvision
pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu118

liruk commented 10 months ago

上記コマンドでTrain及び推論できました、ありがとうございます！

litagin02 / Style-Bert-VITS2

トレーニングに失敗する：torch.distributed.DistNetworkError: Unknown error #38