search

kohya-ss / sd-scripts

Apache License 2.0
5.22k stars 867 forks source link

UserWarning: cuDNN SDPA backward got grad_output. #1757

Open kitamo1981 opened 4 days ago

kitamo1981 commented 4 days ago

After updating to the latest version today, I receive this message.

"UserWarning: cuDNN SDPA backward got grad_output. ..."

Even though I use the same settings, the performance has decreased by 2x. Previously, with 3060 12GB and training on FLUX with DIM Rank 128, the performance was about 9.5s/it, now this time has increased to 16-17s/it. Even changing Dim Rank to 4 doesn't change anything.

What I noticed is that the GPU is not constantly loaded at 99% - there are only spikes to 100% and then drops to 4%, which I have not noticed before. Vram usage 4,3-7.9GB and it jupm up and down all the time.

kohya-ss commented 2 days ago

Could you please tell us what version you were using before and if you use that version will it return to normal speed?