search

JarodMica / ai-voice-cloning

GNU General Public License v3.0
655 stars 144 forks source link

Voice Training Error - Need Help #68

Open Ramdasta opened 8 months ago

Ramdasta commented 8 months ago

Traying to train a voice but unable to do it here is the error -

D:\Tools\ai-voice-cloning>set PYTHONUTF8=1

D:\Tools\ai-voice-cloning>runtime\python.exe .\src\main.py 2024-03-25 02:38:21 | INFO | rvc.configs.config | Found GPU NVIDIA GeForce RTX 4060 Whisper detected Traceback (most recent call last): File "D:\Tools\ai-voice-cloning\src\utils.py", line 98, in from vall_e.emb.qnt import encode as valle_quantize ModuleNotFoundError: No module named 'vall_e'

Traceback (most recent call last): File "D:\Tools\ai-voice-cloning\src\utils.py", line 118, in import bark ModuleNotFoundError: No module named 'bark'

[textbox, textbox, radio, textbox, dropdown, audio, number, slider, number, slider, slider, slider, radio, slider, slider, slider, slider, slider, slider, slider, checkboxgroup, checkbox, checkbox] [dropdown, slider, dropdown, slider, slider, slider, slider, slider] Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch(). Loading TorToiSe... (AR: D:\Tools\ai-voice-cloning\models\tortoise\autoregressive.pth, diffusion: ./models/tortoise/diffusion_decoder.pth, vocoder: bigvgan_24khz_100band) Hardware acceleration found: cuda use_deepspeed api_debug True D:\Tools\ai-voice-cloning\runtime\lib\site-packages\torch\nn\utils\weight_norm.py:30: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm. warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.") Loading tokenizer JSON: ./modules/tortoise-tts/tortoise/data/tokenizer.json Loaded tokenizer Loading autoregressive model: D:\Tools\ai-voice-cloning\models\tortoise\autoregressive.pth [2024-03-25 02:38:37,684] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed info: version=0.8.3+6eca037c, git-hash=6eca037c, git-branch=HEAD [2024-03-25 02:38:37,685] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter mp_size is deprecated use tensor_parallel.tp_size instead [2024-03-25 02:38:37,685] [INFO] [logging.py:96:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1 WARNING! Setting BLOOMLayerPolicy._orig_layer_class to None due to Exception: module 'transformers.models' has no attribute 'bloom' [2024-03-25 02:38:37,728] [INFO] [logging.py:96:log_dist] [Rank -1] DeepSpeed-Inference config: {'layer_id': 0, 'hidden_size': 1024, 'intermediate_size': 4096, 'heads': 16, 'num_hidden_layers': -1, 'fp16': False, 'pre_layer_norm': True, 'local_rank': -1, 'stochastic_mode': False, 'epsilon': 1e-05, 'mp_size': 1, 'q_int8': False, 'scale_attention': True, 'triangular_masking': True, 'local_attention': False, 'window_size': 1, 'rotary_dim': -1, 'rotate_half': False, 'rotate_every_two': True, 'return_tuple': True, 'mlp_after_attn': True, 'mlp_act_func_type': <ActivationFuncType.GELU: 1>, 'specialized_mode': False, 'training_mp_size': 1, 'bigscience_bloom': False, 'max_out_tokens': 1024, 'scale_attn_by_inverse_layer_idx': False, 'enable_qkv_quantization': False, 'use_mup': False, 'return_single_tuple': False} Loaded autoregressive model Loaded diffusion model Loading vocoder model: bigvgan_24khz_100band Loading vocoder model: bigvgan_24khz_100band.pth Removing weight norm... Loaded vocoder model Loaded TTS, ready for generation. 2024-03-25 02:39:02 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:39:02 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/reset "HTTP/1.1 200 OK" 2024-03-25 02:39:02 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:39:02 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:39:02 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/reset "HTTP/1.1 200 OK" 2024-03-25 02:39:02 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/reset "HTTP/1.1 200 OK" 2024-03-25 02:39:02 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:39:02 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/reset "HTTP/1.1 200 OK" 2024-03-25 02:39:28 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:39:28 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/reset "HTTP/1.1 200 OK" 2024-03-25 02:39:34 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:39:34 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/reset "HTTP/1.1 200 OK" 2024-03-25 02:39:50 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:39:50 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/reset "HTTP/1.1 200 OK" 2024-03-25 02:39:53 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:39:53 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/reset "HTTP/1.1 200 OK" 2024-03-25 02:39:57 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:39:57 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/reset "HTTP/1.1 200 OK" Unloaded TTS Spawning process: train.bat ./training/killgame/train.yaml [Training] [2024-03-25T02:40:04.535690] 2024-03-25 02:40:04 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:04.538690] D:\Tools\ai-voice-cloning>set PYTHONUTF8=1 2024-03-25 02:40:04 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:04.542690] 2024-03-25 02:40:04 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:04.546690] D:\Tools\ai-voice-cloning>.\runtime\python.exe .\src\train.py --yaml "./training/killgame/train.yaml" 2024-03-25 02:40:04 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:40:04 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:40:04 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/reset "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:10.354849] [2024-03-25 02:40:10,354] torch.distributed.elastic.multiprocessing.redirects: [WARNING] NOTE: Redirects are currently not supported in Windows or MacOs. 2024-03-25 02:40:10 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:40:10 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:40:10 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/reset "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.446219] 24-03-25 02:40:13.446 - INFO: name: killgame 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.450220] model: extensibletrainer 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.452219] scale: 1 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.455219] gpu_ids: [0] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.460220] start_step: 0 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.464219] checkpointing_enabled: True 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.467220] fp16: False 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.471219] bitsandbytes: True 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.474220] gpus: 1 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.480220] datasets:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.483221] train:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.485221] name: training 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.487220] n_workers: 2 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.490221] batch_size: 80 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.493221] mode: paired_voice_audio 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.496220] path: ./training/killgame/train.txt 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.498221] fetcher_mode: ['lj'] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.500220] phase: train 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.503221] max_wav_length: 255995 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.505220] max_text_length: 200 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.508221] sample_rate: 22050 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.511221] load_conditioning: True 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.513220] num_conditioning_candidates: 2 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.516221] conditioning_length: 44000 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.518222] use_bpe_tokenizer: True 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.521221] tokenizer_vocab: ./modules/tortoise-tts/tortoise/data/tokenizer.json 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.525222] load_aligned_codes: False 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.528220] data_type: img 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.530221] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.533221] val:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.535221] name: validation 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.537221] n_workers: 2 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.540220] batch_size: 0 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.543221] mode: paired_voice_audio 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.546221] path: ./training/killgame/validation.txt 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.548220] fetcher_mode: ['lj'] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.551265] phase: val 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.554220] max_wav_length: 255995 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.556220] max_text_length: 200 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.559219] sample_rate: 22050 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.562220] load_conditioning: True 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.564220] num_conditioning_candidates: 2 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.567220] conditioning_length: 44000 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.569219] use_bpe_tokenizer: True 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.571220] tokenizer_vocab: ./modules/tortoise-tts/tortoise/data/tokenizer.json 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.574220] load_aligned_codes: False 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.577220] data_type: img 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.580219] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.582220] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.584219] steps:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.587220] gpt_train:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.589219] training: gpt 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.592219] loss_log_buffer: 500 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.594219] optimizer: adamw 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.597221] optimizer_params:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.600220] lr: 1e-05 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.603219] weight_decay: 0.01 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.605219] beta1: 0.9 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.607219] beta2: 0.96 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.610220] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.612220] clip_grad_eps: 4 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.615221] injectors:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.617220] paired_to_mel:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.619221] type: torch_mel_spectrogram 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.621220] mel_norm_file: ./modules/tortoise-tts/tortoise/data/mel_norms.pth 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.623220] in: wav 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.626219] out: paired_mel 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.629221] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.631221] paired_cond_to_mel:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.634219] type: for_each 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.636219] subtype: torch_mel_spectrogram 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.638219] mel_norm_file: ./modules/tortoise-tts/tortoise/data/mel_norms.pth 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.640220] in: conditioning 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.643220] out: paired_conditioning_mel 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.645221] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.648220] to_codes:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.651220] type: discrete_token 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.653220] in: paired_mel 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.655220] out: paired_mel_codes 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.657220] dvae_config: ./models/tortoise/train_diffusion_vocoder_22k_level.yml 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.660220] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.663219] paired_fwd_text:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.665219] type: generator 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.669219] generator: gpt 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.672219] in: ['paired_conditioning_mel', 'padded_text', 'text_lengths', 'paired_mel_codes', 'wav_lengths'] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.677220] out: ['loss_text_ce', 'loss_mel_ce', 'logits'] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.680220] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.684221] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.687221] losses:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.690221] text_ce:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.693221] type: direct 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.695220] weight: 0.01 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.698221] key: loss_text_ce 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.700220] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.703221] mel_ce:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.705220] type: direct 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.708220] weight: 1 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.711220] key: loss_mel_ce 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.713220] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.716219] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.718220] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.721219] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.723219] networks:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.725219] gpt:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.728219] type: generator 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.731219] which_model_G: unified_voice2 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.734221] kwargs:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.736220] layers: 30 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.739220] model_dim: 1024 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.742220] heads: 16 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.745220] max_text_tokens: 402 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.747219] max_mel_tokens: 604 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.749219] max_conditioning_inputs: 2 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.752219] mel_length_compression: 1024 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.754220] number_text_tokens: 256 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.756219] number_mel_codes: 8194 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.759219] start_mel_token: 8192 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.762220] stop_mel_token: 8193 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.764220] start_text_token: 255 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.767220] train_solo_embeddings: False 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.769220] use_mel_codes_as_input: True 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.771220] checkpointing: True 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.774220] tortoise_compat: True 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.776220] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.779220] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.782220] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.784219] path:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.787219] strict_load: True 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.789220] pretrain_model_gpt: ./models/tortoise/autoregressive.pth 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.791219] root: ./ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.795219] experiments_root: ./training\killgame\finetune 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.798220] models: ./training\killgame\finetune\models 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.800219] training_state: ./training\killgame\finetune\training_state 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.803219] log: ./training\killgame\finetune 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.805219] val_images: ./training\killgame\finetune\val_images 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.808220] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.811221] train:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.814220] niter: 600 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.816220] warmup_iter: -1 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.819220] mega_batch_factor: 20 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.821220] val_freq: 15 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.824219] ema_enabled: False 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.827219] default_lr_scheme: CosineAnnealingLR_Restart 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.830220] T_period: [150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150, 150] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.832220] warmup: 0 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.835220] eta_min: 1e-08 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.837220] restarts: [150, 300, 450, 600] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.840220] restart_weights: [0.75, 0.5, 0.25, 0.125] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.845219] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.848219] eval:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.850220] pure: False 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.853220] output_state: gen 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.855220] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.858220] logger:[ 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.861219] save_checkpoint_freq: 15 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.863220] visuals: ['gen', 'mel'] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.866219] visual_debug_rate: 15 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.869218] is_mel_spectrogram: True 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.871220] ] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.875220] is_train: True 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.878220] dist: False 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.880219] 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:13.883219] 24-03-25 02:40:13.446 - INFO: Random seed: 906 2024-03-25 02:40:13 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:40:15 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:40:15 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/reset "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:16.443804] 24-03-25 02:40:16.442 - INFO: Number of training data elements: 202, iters: 3 2024-03-25 02:40:16 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:16.446805] 24-03-25 02:40:16.443 - INFO: Total epochs needed: 200 for iters 600 2024-03-25 02:40:16 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:40:16 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:40:16 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/reset "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:17.270470] D:\Tools\ai-voice-cloning\runtime\lib\site-packages\transformers\configuration_utils.py:363: UserWarning: Passing gradient_checkpointing to a config initialization is deprecated and will be removed in v5 Transformers. Using model.gradient_checkpointing_enable() instead, or if you are using the Trainer API, pass gradient_checkpointing=True in your TrainingArguments. 2024-03-25 02:40:17 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:17.274470] warnings.warn( 2024-03-25 02:40:17 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:40:17 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:40:17 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/reset "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:23.625643] 24-03-25 02:40:23.625 - INFO: Loading model for [./models/tortoise/autoregressive.pth] 2024-03-25 02:40:23 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:40:23 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/api/predict "HTTP/1.1 200 OK" 2024-03-25 02:40:23 | INFO | httpx | HTTP Request: POST http://127.0.0.1:7860/reset "HTTP/1.1 200 OK" [Training] [2024-03-25T02:40:24.363644] 24-03-25 02:40:24.358 - INFO: Start training from epoch: 0, iter: 0 [Training] [2024-03-25T02:40:25.567105] [2024-03-25 02:40:25,567] torch.distributed.elastic.multiprocessing.redirects: [WARNING] NOTE: Redirects are currently not supported in Windows or MacOs. [Training] [2024-03-25T02:40:26.822777] [2024-03-25 02:40:26,822] torch.distributed.elastic.multiprocessing.redirects: [WARNING] NOTE: Redirects are currently not supported in Windows or MacOs. [Training] [2024-03-25T02:40:28.221774] D:\Tools\ai-voice-cloning\runtime\lib\site-packages\torch\optim\lr_scheduler.py:136: UserWarning: Detected call of lr_scheduler.step() before optimizer.step(). In PyTorch 1.1.0 and later, you should call them in the opposite order: optimizer.step() before lr_scheduler.step(). Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate [Training] [2024-03-25T02:40:28.221774] warnings.warn("Detected call of lr_scheduler.step() before optimizer.step(). " [Training] [2024-03-25T02:40:28.801775] D:\Tools\ai-voice-cloning\runtime\lib\site-packages\torch\utils\checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants. [Training] [2024-03-25T02:40:28.801775] warnings.warn( image_2024-03-25_024932748

brudjkhalid commented 7 months ago

Yup same exact issue happens to me.

I wonder what it's caused by because Tortoise works fine for some people and gives this for others like us.