search

AUTOMATIC1111 / stable-diffusion-webui

Stable Diffusion web UI
GNU Affero General Public License v3.0
142.1k stars 26.84k forks source link

[issue]: [user was the issue] xformers not detected, training impossible, errors not making sense #5762

Closed DHG-Dav closed 1 year ago

DHG-Dav commented 1 year ago

Is there an existing issue for this?

What happened?

Hello, First, let's speak about xformers, 1 - tried to do as you said "run webui-user.bat" and let it do its job. - didn't worked 2 - went to follow your xformers tutorial : didn't worked. 3 - went "all in" and i compiled manually xformers in the 'modules' directory, it's detected by conda and pip, but not by webui. image image

I don't understand what's going wrong.

Second, i am able to generate pictures, sadly not using the v-prediction models, but i am not able to train the network. I get the message "No inf checks were recorded for this optimizer." and i'm not able to figure out what this mean, it's been 2 days i'm trying obsessively to run this and tried pretty much everything mentioned here without improvement. 007182 007183 007184 007185 007186

I don't get it, any help would be very welcome. Also i tried to run launch.py but got some more nonsensical errors 007178

Of course i tried to run it outside of conda as well, without result. What am i doing wrong ? i'm lost... I've been able to install & run stable diffusion without problem, but this ui doesn't want to run... i want to finetune SD and thought this would be the most convenient way to do it, but i'm out of idea right now and i find it extremely difficult to find explanations about how all this run, and i don't find my way into the code.

Any help is very welcome.

Steps to reproduce the problem

  1. launch webui-user.bat
  2. Press train
  3. doesn't train.

What should have happened?

trained

Commit where the problem happens

685f9631b56ff8bd43bce24ff5ce0f9a0e9af490

What platforms do you use to access UI ?

Windows

What browsers do you use to access the UI ?

Mozilla Firefox

Command Line Arguments

--xformers

Additional information, context and logs

list : absl-py 1.3.0 pypi_0 pypi accelerate 0.15.0 pypi_0 pypi addict 2.4.0 pypi_0 pypi aiohttp 3.8.3 pypi_0 pypi aiosignal 1.3.1 pypi_0 pypi antlr4-python3-runtime 4.9.3 pypi_0 pypi anyio 3.6.2 pypi_0 pypi async-timeout 4.0.2 pypi_0 pypi attrs 22.1.0 pypi_0 pypi basicsr 1.4.2 pypi_0 pypi bcrypt 4.0.1 pypi_0 pypi boltons 21.0.0 pypi_0 pypi bzip2 1.0.8 he774522_0 ca-certificates 2022.10.11 haa95532_0 cachetools 5.2.0 pypi_0 pypi certifi 2022.9.24 py310haa95532_0 cffi 1.15.1 pypi_0 pypi chardet 4.0.0 pypi_0 pypi charset-normalizer 2.1.1 pypi_0 pypi clean-fid 0.1.34 pypi_0 pypi click 8.1.3 pypi_0 pypi colorama 0.4.6 pypi_0 pypi coloredlogs 15.0.1 pypi_0 pypi contourpy 1.0.6 pypi_0 pypi cryptography 38.0.4 pypi_0 pypi cuda 11.6.1 0 nvidia cuda-cccl 11.6.55 0 nvidia cuda-command-line-tools 11.6.2 0 nvidia cuda-compiler 11.6.2 0 nvidia cuda-cudart 11.6.55 0 nvidia cuda-cudart-dev 11.6.55 0 nvidia cuda-cuobjdump 11.6.124 0 nvidia cuda-cupti 11.6.124 0 nvidia cuda-cuxxfilt 11.6.124 0 nvidia cuda-libraries 11.6.1 0 nvidia cuda-libraries-dev 11.6.1 0 nvidia cuda-memcheck 11.8.86 0 nvidia cuda-nsight-compute 12.0.0 0 nvidia cuda-nvcc 11.6.124 0 nvidia cuda-nvdisasm 12.0.76 0 nvidia cuda-nvml-dev 11.6.55 0 nvidia cuda-nvprof 12.0.90 0 nvidia cuda-nvprune 11.6.124 0 nvidia cuda-nvrtc 11.6.124 0 nvidia cuda-nvrtc-dev 11.6.124 0 nvidia cuda-nvtx 11.6.124 0 nvidia cuda-nvvp 12.0.90 0 nvidia cuda-runtime 11.6.1 0 nvidia cuda-sanitizer-api 12.0.90 0 nvidia cuda-toolkit 11.6.1 0 nvidia cuda-tools 11.6.1 0 nvidia cuda-visual-tools 11.6.1 0 nvidia cycler 0.11.0 pypi_0 pypi einops 0.6.0 pypi_0 pypi facexlib 0.2.5 pypi_0 pypi fairscale 0.4.4 pypi_0 pypi fastapi 0.88.0 pypi_0 pypi ffmpy 0.3.0 pypi_0 pypi filelock 3.8.2 pypi_0 pypi filterpy 1.4.5 pypi_0 pypi flatbuffers 22.12.6 pypi_0 pypi font-roboto 0.0.1 pypi_0 pypi fonts 0.0.3 pypi_0 pypi fonttools 4.38.0 pypi_0 pypi frozenlist 1.3.3 pypi_0 pypi fsspec 2022.11.0 pypi_0 pypi future 0.18.2 pypi_0 pypi gfpgan 1.3.8 pypi_0 pypi gitdb 4.0.10 pypi_0 pypi gitpython 3.1.29 pypi_0 pypi google-auth 2.15.0 pypi_0 pypi google-auth-oauthlib 0.4.6 pypi_0 pypi gradio 3.9 pypi_0 pypi grpcio 1.51.1 pypi_0 pypi h11 0.12.0 pypi_0 pypi httpcore 0.15.0 pypi_0 pypi httpx 0.23.1 pypi_0 pypi huggingface-hub 0.11.1 pypi_0 pypi humanfriendly 10.0 pypi_0 pypi idna 2.10 pypi_0 pypi imageio 2.22.4 pypi_0 pypi inflection 0.5.1 pypi_0 pypi invisible-watermark 0.1.5 pypi_0 pypi jinja2 3.1.2 pypi_0 pypi jsonmerge 1.9.0 pypi_0 pypi jsonschema 4.17.3 pypi_0 pypi kiwisolver 1.4.4 pypi_0 pypi kornia 0.6.8 pypi_0 pypi lark 1.1.5 pypi_0 pypi libcublas 11.9.2.110 0 nvidia libcublas-dev 11.9.2.110 0 nvidia libcufft 10.7.1.112 0 nvidia libcufft-dev 10.7.1.112 0 nvidia libcurand 10.3.1.50 0 nvidia libcurand-dev 10.3.1.50 0 nvidia libcusolver 11.3.4.124 0 nvidia libcusolver-dev 11.3.4.124 0 nvidia libcusparse 11.7.2.124 0 nvidia libcusparse-dev 11.7.2.124 0 nvidia libffi 3.4.2 hd77b12b_6 libnpp 11.6.3.124 0 nvidia libnpp-dev 11.6.3.124 0 nvidia libnvjpeg 11.6.2.124 0 nvidia libnvjpeg-dev 11.6.2.124 0 nvidia linkify-it-py 1.0.3 pypi_0 pypi llvmlite 0.39.1 pypi_0 pypi lmdb 1.4.0 pypi_0 pypi markdown 3.4.1 pypi_0 pypi markdown-it-py 2.1.0 pypi_0 pypi markupsafe 2.1.1 pypi_0 pypi matplotlib 3.6.2 pypi_0 pypi mdit-py-plugins 0.3.3 pypi_0 pypi mdurl 0.1.2 pypi_0 pypi mpmath 1.2.1 pypi_0 pypi multidict 6.0.3 pypi_0 pypi mypy-extensions 0.4.3 pypi_0 pypi networkx 2.8.8 pypi_0 pypi nsight-compute 2022.4.0.15 0 nvidia numba 0.56.4 pypi_0 pypi numpy 1.23.5 pypi_0 pypi oauthlib 3.2.2 pypi_0 pypi omegaconf 2.3.0 pypi_0 pypi onnx 1.13.0 pypi_0 pypi onnxruntime 1.13.1 pypi_0 pypi opencv-python 4.6.0.66 pypi_0 pypi openssl 1.1.1s h2bbff1b_0 orjson 3.8.3 pypi_0 pypi packaging 22.0 pypi_0 pypi pandas 1.5.2 pypi_0 pypi paramiko 2.12.0 pypi_0 pypi piexif 1.1.3 pypi_0 pypi pillow 9.3.0 pypi_0 pypi pip 22.3.1 py310haa95532_0 protobuf 3.20.3 pypi_0 pypi psutil 5.9.4 pypi_0 pypi pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pycparser 2.21 pypi_0 pypi pycryptodome 3.16.0 pypi_0 pypi pydantic 1.10.2 pypi_0 pypi pydeprecate 0.3.2 pypi_0 pypi pydub 0.25.1 pypi_0 pypi pynacl 1.5.0 pypi_0 pypi pyparsing 3.0.9 pypi_0 pypi pyre-extensions 0.0.23 pypi_0 pypi pyreadline3 3.4.1 pypi_0 pypi pyrsistent 0.19.2 pypi_0 pypi python 3.10.6 hbb2ffb3_1 python-dateutil 2.8.2 pypi_0 pypi python-multipart 0.0.5 pypi_0 pypi pytorch-cuda 11.6 h867d48c_1 pytorch pytorch-lightning 1.7.7 pypi_0 pypi pytz 2022.6 pypi_0 pypi pywavelets 1.4.1 pypi_0 pypi pyyaml 6.0 pypi_0 pypi realesrgan 0.3.0 pypi_0 pypi regex 2022.10.31 pypi_0 pypi requests 2.25.1 pypi_0 pypi requests-oauthlib 1.3.1 pypi_0 pypi resize-right 0.0.2 pypi_0 pypi rfc3986 1.5.0 pypi_0 pypi rsa 4.9 pypi_0 pypi safetensors 0.2.6 pypi_0 pypi scikit-image 0.19.3 pypi_0 pypi scipy 1.9.3 pypi_0 pypi setuptools 65.5.0 py310haa95532_0 six 1.16.0 pypi_0 pypi smmap 5.0.0 pypi_0 pypi sniffio 1.3.0 pypi_0 pypi sqlite 3.40.0 h2bbff1b_0 starlette 0.22.0 pypi_0 pypi sympy 1.11.1 pypi_0 pypi tb-nightly 2.12.0a20221214 pypi_0 pypi tensorboard 2.11.0 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.1 pypi_0 pypi tifffile 2022.10.10 pypi_0 pypi timm 0.4.12 pypi_0 pypi tk 8.6.12 h2bbff1b_0 tokenizers 0.12.1 pypi_0 pypi torch 1.13.0 pypi_0 pypi torchdiffeq 0.2.3 pypi_0 pypi torchmetrics 0.11.0 pypi_0 pypi torchsde 0.2.5 pypi_0 pypi torchvision 0.14.0 pypi_0 pypi tqdm 4.64.1 pypi_0 pypi trampoline 0.1.2 pypi_0 pypi transformers 4.19.2 pypi_0 pypi typing-extensions 4.4.0 pypi_0 pypi typing-inspect 0.8.0 pypi_0 pypi tzdata 2022g h04d1e81_0 uc-micro-py 1.0.1 pypi_0 pypi urllib3 1.26.13 pypi_0 pypi uvicorn 0.20.0 pypi_0 pypi vc 14.2 h21ff451_1 vs2015_runtime 14.27.29016 h5e58377_2 websockets 10.4 pypi_0 pypi werkzeug 2.2.2 pypi_0 pypi wheel 0.37.1 pyhd3eb1b0_0 wincertstore 0.2 py310haa95532_2 xformers 0.0.15.dev0+ed99125.d20221215 dev_0 xz 5.2.8 h8cc25b3_0 yapf 0.32.0 pypi_0 pypi yarl 1.8.2 pypi_0 pypi zlib 1.2.13 h8cc25b3_0

wywywywy commented 1 year ago

You need Python 3.10

DHG-Dav commented 1 year ago

You need Python 3.10

as specified in the list : "python 3.10.6" (i realize the first screenshot says python 3.9.9 but i don't get WHY since the environment has only python 3.10.9... Another weird thing there lol)

wywywywy commented 1 year ago

That's not what you're actually using according to your logs.

You're using 3.9 in the venv, and 3.11 outside.

wywywywy commented 1 year ago

Best thing to do is to delete the venv and re-run the batch file. It'll rebuild the venv for you. And make sure you actually use 3.10.

Or if you want to use conda, make sure you specify the python version when creating a venv

DHG-Dav commented 1 year ago

Best thing to do is to delete the venv and re-run the batch file. It'll rebuild the venv for you. And make sure you actually use 3.10.

Or if you want to use conda, make sure you specify the python version when creating a venv

thanks for your help. I deleted the venv entirely and running the .bat again, i'll update here in a few hours since torch/torchvision will need 3-4 hours to re-download. (i have a third world bandwidth ...) My conda env is already created with python 3.10.6 at creation.

wywywywy commented 1 year ago

Make sure to double check version before spending time re-downloading.

DHG-Dav commented 1 year ago

Well, in fact the installer found the pytorch already downloaded and installed it super fast. However the problems remains the same even with the correct python version displayed : image

wywywywy commented 1 year ago

Ok this is good. The next step is to put --xformers into the webui-user.bat in the COMMANDLINE_ARGS block instead of using as an argument. It should install the right Xformers for you.

DHG-Dav commented 1 year ago

Ok great thanks a lot, i'm already super grateful, this was totally my bad (sorry) and it worked first try. Now the last problem remains : "No inf checks were recorded for this optimizer." (closing the training at zero step)

wywywywy commented 1 year ago

Here is a related issue that may help you with this new problem - https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/5280

DHG-Dav commented 1 year ago

Here is a related issue that may help you with this new problem - #5280

Thanks, i already saw this (as well as #5715 and #5141 and #5383 ) and tried a few variations,
I tried my "hypernetwork.txt" with plenty of variations of the "a painting of [filewords], art by [name]" as well as the "a photo of a [name], [filewords]" tried to change the spacing, remove the comma... etc i shortened it to just "[filewords]" and "[filewords] artist:[name]", "[filewords] [name]"... etc Also tried the --no-half (right now into the webui-user.bat in the COMMANDLINE_ARGS since i was doing it wrong previously) and still the same problem appears :/ Oh ok, i just created a new hypernetwork with exactly the same settings (didn't use no-half and have the hypernetwork.txt filled with just "[filewords] [name]"...) And it worked, it's currently training i'm a bit frustrated to don't understand the why and what exactly solved it, but thank you so much for your patience to help me, i'm really grateful !

Thanks a lot ! Wish you the best !