Something went wrong

[nizu-alt]

Possible latent mismatch: click the "(Re)Compute Voice Latents" button and then try again. Error: CUDA error: no kernel image is available for execution on the device CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[HaoTang1]

Same error when training with Chinese. Win 10, CUDA 12.1, RTX 3090

[Training] [2024-05-28T17:33:02.323483] C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\Indexing.cu:1289: block: [144,0,0], thread: [95,0,0] Assertion srcIndex < srcSelectDimSize failed. [Training] [2024-05-28T17:33:03.025133] Disabled distributed training. [Training] [2024-05-28T17:33:03.025133] Path already exists. Rename it to [./training\lt\finetune_archived_240528-173133] [Training] [2024-05-28T17:33:03.026133] Loading from ./models/tortoise/dvae.pth [Training] [2024-05-28T17:33:03.027132] Traceback (most recent call last): [Training] [2024-05-28T17:33:03.028131] File "D:\ai-voice-cloning\src\train.py", line 72, in [Training] [2024-05-28T17:33:03.028131] train(config_path, args.launcher) [Training] [2024-05-28T17:33:03.029131] File "D:\ai-voice-cloning\src\train.py", line 39, in train [Training] [2024-05-28T17:33:03.029131] trainer.do_training() [Training] [2024-05-28T17:33:03.030131] File "D:\ai-voice-cloning\modules\dlas\dlas\train.py", line 408, in do_training [Training] [2024-05-28T17:33:03.030131] metric = self.do_step(train_data) [Training] [2024-05-28T17:33:03.031130] ^^^^^^^^^^^^^^^^^^^^^^^^ [Training] [2024-05-28T17:33:03.031130] File "D:\ai-voice-cloning\modules\dlas\dlas\train.py", line 271, in do_step [Training] [2024-05-28T17:33:03.032130] gradient_norms_dict = self.model.optimize_parameters( [Training] [2024-05-28T17:33:03.032130] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [Training] [2024-05-28T17:33:03.034130] File "D:\ai-voice-cloning\modules\dlas\dlas\trainer\ExtensibleTrainer.py", line 321, in optimize_parameters [Training] [2024-05-28T17:33:03.034130] ns = step.do_forward_backward( [Training] [2024-05-28T17:33:03.035130] ^^^^^^^^^^^^^^^^^^^^^^^^^ [Training] [2024-05-28T17:33:03.035130] File "D:\ai-voice-cloning\modules\dlas\dlas\trainer\steps.py", line 274, in do_forward_backward [Training] [2024-05-28T17:33:03.036130] injected = inj(local_state) [Training] [2024-05-28T17:33:03.036130] ^^^^^^^^^^^^^^^^ [Training] [2024-05-28T17:33:03.037129] File "D:\ai-voice-cloning\venv\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl [Training] [2024-05-28T17:33:03.038129] return self._call_impl(*args, kwargs) [Training] [2024-05-28T17:33:03.038129] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [Training] [2024-05-28T17:33:03.039129] File "D:\ai-voice-cloning\venv\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl [Training] [2024-05-28T17:33:03.039129] return forward_call(*args, *kwargs) [Training] [2024-05-28T17:33:03.040129] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [Training] [2024-05-28T17:33:03.040129] File "D:\ai-voice-cloning\modules\dlas\dlas\trainer\injectors\base_injectors.py", line 94, in forward [Training] [2024-05-28T17:33:03.041127] results = method(params, self.args) [Training] [2024-05-28T17:33:03.041127] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [Training] [2024-05-28T17:33:03.042127] File "D:\ai-voice-cloning\venv\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl [Training] [2024-05-28T17:33:03.043127] return self._call_impl(*args, kwargs) [Training] [2024-05-28T17:33:03.043127] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [Training] [2024-05-28T17:33:03.044127] File "D:\ai-voice-cloning\venv\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl [Training] [2024-05-28T17:33:03.045127] return forward_call(*args, *kwargs) [Training] [2024-05-28T17:33:03.045127] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [Training] [2024-05-28T17:33:03.046126] File "D:\ai-voice-cloning\venv\Lib\site-packages\torch\nn\parallel\data_parallel.py", line 183, in forward [Training] [2024-05-28T17:33:03.046126] return self.module(inputs[0], module_kwargs[0]) [Training] [2024-05-28T17:33:03.047126] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [Training] [2024-05-28T17:33:03.047126] File "D:\ai-voice-cloning\venv\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl [Training] [2024-05-28T17:33:03.049125] return self._call_impl(*args, kwargs) [Training] [2024-05-28T17:33:03.052124] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [Training] [2024-05-28T17:33:03.053124] File "D:\ai-voice-cloning\venv\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl [Training] [2024-05-28T17:33:03.053124] return forward_call(*args, *kwargs) [Training] [2024-05-28T17:33:03.054123] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [Training] [2024-05-28T17:33:03.054123] File "D:\ai-voice-cloning\modules\dlas\dlas\models\audio\tts\unified_voice2.py", line 440, in forward [Training] [2024-05-28T17:33:03.055124] text_emb = self.text_embedding( [Training] [2024-05-28T17:33:03.055124] ^^^^^^^^^^^^^^^^^^^^ [Training] [2024-05-28T17:33:03.056122] File "D:\ai-voice-cloning\venv\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl [Training] [2024-05-28T17:33:03.056122] return self._call_impl(args, kwargs) [Training] [2024-05-28T17:33:03.057122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [Training] [2024-05-28T17:33:03.058122] File "D:\ai-voice-cloning\venv\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl [Training] [2024-05-28T17:33:03.058122] return forward_call(*args, **kwargs) [Training] [2024-05-28T17:33:03.059122] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [Training] [2024-05-28T17:33:03.059122] File "D:\ai-voice-cloning\venv\Lib\site-packages\torch\nn\modules\sparse.py", line 163, in forward [Training] [2024-05-28T17:33:03.060121] return F.embedding( [Training] [2024-05-28T17:33:03.060121] ^^^^^^^^^^^^ [Training] [2024-05-28T17:33:03.061121] File "D:\ai-voice-cloning\venv\Lib\site-packages\torch\nn\functional.py", line 2264, in embedding [Training] [2024-05-28T17:33:03.062121] return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) [Training] [2024-05-28T17:33:03.062121] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [Training] [2024-05-28T17:33:03.063121] RuntimeError: CUDA error: device-side assert triggered [Training] [2024-05-28T17:33:03.066119] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions. [Training] [2024-05-28T17:33:03.066119]

[nucleiis]

Anyone figured out how to fix this error? I'm training Korean and faced the exact same error message