RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasSdot(handle, n, x, incx, y, incy, result)`

[Bocchi-Chan2023]

File "C:\Users[username]\kohya_ss\sd-scripts\sdxl_train_network.py", line 188, in trainer.train(args) File "C:\Users[username]\kohya_ss\sd-scripts\train_network.py", line 859, in train optimizer.step() File "C:\Users[username]\kohya_ss\venv\lib\site-packages\accelerate\optimizer.py", line 132, in step self.scaler.step(self.optimizer, closure) File "C:\Users[username]\kohya_ss\venv\lib\site-packages\torch\cuda\amp\grad_scaler.py", line 416, in step retval = self._maybe_opt_step(optimizer, optimizer_state, *args, kwargs) File "C:\Users[username]\kohya_ss\venv\lib\site-packages\torch\cuda\amp\grad_scaler.py", line 315, in _maybe_opt_step retval = optimizer.step(*args, *kwargs) File "C:\Users[username]\kohya_ss\venv\lib\site-packages\accelerate\optimizer.py", line 185, in patched_step return method(args, kwargs) File "C:\Users[username]\kohya_ss\venv\lib\site-packages\torch\optim\lr_scheduler.py", line 68, in wrapper return wrapped(*args, *kwargs) File "C:\Users[username]\kohya_ss\venv\lib\site-packages\torch\optim\optimizer.py", line 373, in wrapper out = func(args, *kwargs) File "C:\Users[username]\kohya_ss\venv\lib\site-packages\prodigyopt\prodigy.py", line 179, in step d_numerator += (d / d0) dlr * torch.dot(grad.flatten(), (p0.data - p.data).flatten()).item() RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasSdot(handle, n, x, incx, y, incy, result) steps: 0%| | 1/504 [00:10<1:25:52, 10.24s/it, avr_loss=0.0967] Traceback (most recent call last): File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users[username]\kohya_ss\venv\Scripts\accelerate.exe__main__.py", line 7, in File "C:\Users[username]\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main args.func(args) File "C:\Users[username]\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command simple_launcher(args) File "C:\Users[username]\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['C:\Users\[username]\kohya_ss\venv\Scripts\python.exe', 'C:\Users\[username]\kohya_ss/sd-scripts/sdxl_train_network.py', '--no_half_vae', '--bucket_no_upscale', '--bucket_reso_steps=64', '--cache_latents', '--enable_bucket', '--min_bucket_reso=256', '--max_bucket_reso=2048', '--fp8_base', '--gradient_checkpointing', '--learning_rate=1.0', '--lr_scheduler=cosine_with_restarts', '--lr_scheduler_num_cycles=1', '--lr_warmup_steps=50', '--max_data_loader_n_workers=0', '--max_grad_norm=1', '--resolution=512,512', '--max_train_steps=504', '--mem_eff_attn', '--mixed_precision=fp16', '--network_alpha=1', '--network_dim=8', '--network_module=networks.lora', '--no_half_vae', '--optimizer_type=Prodigy', '--output_dir=C:/Users/[username]/kohya_ss/outputs', '--output_name=nyankone', '--pretrained_model_name_or_path=C:/Users/[username]/kohya_ss/models/animagine-xl-3.1.safetensors', '--save_every_n_epochs=1', '--save_model_as=safetensors', '--save_precision=fp16', '--train_batch_size=1', '--train_data_dir=C:\stable-diffusion-webui-forge-zluda\outputs\train_tools\projects\hnyankone\versions\v1\dataset\processed']' returned non-zero exit status 1.

[lshqqytiger]

Try v3.7-pre5.

[Bocchi-Chan2023]

The problem seems to be that cublasDotEx is not implemented. Please consider this if you have time🙏

C:\kohya_ss\venv\lib\site-packages\torch\utils\checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants. warnings.warn( Traceback (most recent call last): File "C:\kohya_ss\sd-scripts\sdxl_train_network.py", line 185, in trainer.train(args) File "C:\kohya_ss\sd-scripts\train_network.py", line 913, in train optimizer.step() File "C:\kohya_ss\venv\lib\site-packages\accelerate\optimizer.py", line 145, in step self.optimizer.step(closure) File "C:\kohya_ss\venv\lib\site-packages\torch\optim\lr_scheduler.py", line 68, in wrapper return wrapped(*args, *kwargs) File "C:\kohya_ss\venv\lib\site-packages\torch\optim\optimizer.py", line 373, in wrapper out = func(args, *kwargs) File "C:\kohya_ss\venv\lib\site-packages\prodigyopt\prodigy.py", line 179, in step d_numerator += (d / d0) dlr * torch.dot(grad.flatten(), (p0.data - p.data).flatten()).item() RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasDotEx( handle, n, x, CUDA_R_16BF, incx, y, CUDA_R_16BF, incy, result, CUDA_R_16BF, CUDA_R_32F) steps: 0%| | 0/852 [01:57<?, ?it/s] Traceback (most recent call last): File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\kohya_ss\venv\Scripts\accelerate.exe__main__.py", line 7, in File "C:\kohya_ss\venv\lib\site-packages\accelerate\commands\accelerate_cli.py", line 47, in main args.func(args) File "C:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 1017, in launch_command simple_launcher(args) File "C:\kohya_ss\venv\lib\site-packages\accelerate\commands\launch.py", line 637, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['C:\kohya_ss\venv\Scripts\python.exe', 'C:\kohya_ss/sd-scripts/sdxl_train_network.py', '--bucket_no_upscale', '--bucket_reso_steps=32', '--cache_latents', '--cache_latents_to_disk', '--caption_extension=.txt', '--enable_bucket', '--min_bucket_reso=256', '--max_bucket_reso=2048', '--fp8_base', '--full_bf16', '--gradient_checkpointing', '--huber_c=0.1', '--huber_schedule=snr', '--learning_rate=1.0', '--loss_type=l2', '--lr_scheduler=constant', '--lr_scheduler_num_cycles=1', '--max_data_loader_n_workers=0', '--max_grad_norm=1', '--resolution=768,768', '--max_train_steps=852', '--mem_eff_attn', '--min_snr_gamma=5', '--min_timestep=0', '--mixed_precision=bf16', '--network_alpha=16', '--network_dim=32', '--network_module=networks.lora', '--no_half_vae', '--optimizer_args', 'weight_decay=0.01', 'decouple=True', 'd0=0.0001', 'use_bias_correction=True', '--optimizer_type=Prodigy', '--output_dir=C:/kohya_ss/outputs', '--output_name=mayano2', '--pretrained_model_name_or_path=C:/stable-diffusion-webui-forge-on-amd/models/Stable-diffusion/ponyDiffusionV6XL_v6StartWithThisOne.safetensors', '--save_every_n_epochs=1', '--save_model_as=safetensors', '--save_precision=bf16', '--scale_weight_norms=5', '--text_encoder_lr=1.0', '--train_batch_size=4', '--training_comment=trigger: the queen of heart 1a', '--train_data_dir=C:\Users\pclif\Downloads\testing', '--unet_lr=1.0']' returned non-zero exit status 1.

[lshqqytiger]

Try v3.8-pre1

[Bocchi-Chan2023]

It worked finally! Thank you dev so much!