Open atimogus opened 9 months ago
i guess i solved my problem, i have generated about 60 pics with great consistency
i went to safe mode and did DDU, after entering in windows i uninstalled any nvidia software i had on my pc (through control pannel) and then i restartred my pc and installed newest gpu drivers, after restart i tried stable diffusion and it is working i hope so
edit1:
after i tought i fixed the problem i downloaded tensorRT and controlnet (while installing tensorRT i got some errors about cudnn) so i reinstalled nvidia drivers again but still the same issue but now sometimes i get black/gray images
i get this error NansException: A tensor with all NaNs was produced in Unet. Use --disable-nan-check commandline argument to disable this check.
edit2:
so my theory is that tensorRT is downlaoding some cursed CUDA and now i downloaded CUDA and DDU drivers and uninstall all NVIDIA sofrware through control pannel i am able to generate 20+ images without any fault https://developer.nvidia.com/cuda-toolkit-archive
args i have used when writing this comment set COMMANDLINE_ARGS= --no-half
edit3:
i put back my old args
set COMMANDLINE_ARGS=--medvram --xformers --force-enable-xformers --always-batch-cond-uncond --opt-channelslast --no-hashing --disable-nan-check --api --xformers-flash-attention --opt-split-attention --no-half-vae set PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.6, max_split_size_mb:32
and everything is working fine, i just realised that xformers doesnt give any more speed while generating images, i guss that new xformers needs newer version of pytorch, i will try to downgrade
Does restarting solve your issue? Because it generally does for me. I personally suspect it's some sort of weird memory glitch. Sometimes models will work fine, I switch from models to model and go back to the original one and I'll get NansException. It's inconsistent, and only appears to go away after a long time or restarting the system.
yes, sometimes restarting did resolve issue, but not for long (maybe 50 genetared images it again appears) but after i installed right cuda and everything it is working perfectly fine i did generate about 1000 images and i am not getting any error
Installing new CUDA didn't fix it for me sadly.
Checklist
OS
Windows 10
What version did you experience this issue on?
SD 1.5
What happened?
i have had this arguments before and SD was working fine
set COMMANDLINE_ARGS=--medvram --xformers --force-enable-xformers --always-batch-cond-uncond --opt-channelslast --no-hashing --disable-nan-check --api --xformers-flash-attention --opt-split-attention --no-half-vae set PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.6, max_split_size_mb:32
i was downloading different versions of CUDA because i needed it for tensorflow project (i realised that tf doesnt work on win anymore) so i started running tf on WSL and now i have newest CUDA on my pc. After 2 months of not using SD i opened it again and tried to generate few picutes, i was getting this error (with args i mentioned above)
RuntimeError: mat1 and mat2 must have the same dtype
NO ARGUMENTS
so i tried removing all arguments then i can see in preview that image is generating but i get this output
NansException: A tensor with all NaNs was produced in Unet. This could be either because there's not enough precision to represent the picture, or because your video card does not support half type. Try setting the "Upcast cross attention layer to float32" option in Settings > Stable Diffusion or using the --no-half commandline argument to fix this. Use --disable-nan-check commandline argument to disable this check. `
ONLY --no-half IN ARGUMENTS
then i added --no-half
after this picture i tried other sampling methods still the same (but with euler sampling i get: NansException: A tensor with all NaNs was produced in Unet. Use --disable-nan-check commandline argument to disable this check.)
--no-half-vae --no-half --disable-nan-check
now i try this arguents
set COMMANDLINE_ARGS= --no-half-vae --no-half --disable-nan-check
i tried changing models, it did not fix problem, did upcast cross attention layer to float32" option in Settings i have the same generated pictures
GPU: RTX 3060 12GB Vram every time i changed arguments i deleted venv i also tried installing new nvidia drivers but now i tink --xformers doesnt work but that is not main problem i also tried updating from 1.6.0 to 1.7.0 but did not help
What browsers do you use to access the UI ?
Brave, i have tried with multiple browsers but i get the same issue