[Bug]: Launching the webui never got further "Loading weights [dcd690123c]"

[palinko91]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What happened?

Hello. I have a Radeon VII graphic card and freshly installed EndeavourOS to try out this amazing software. My problem is the program never going further than this line: Loading weights [dcd690123c] from /home/palinko/Desktop/stable-diffusion-webui/models/Stable-diffusion/v2-1_768-ema-pruned.safetensors

I've tried to load 4 different models both ckpt and safetensors, nothing worked and nothing loaded unfortunatelly. Even if I'm waiting a lot of time, processor remainst at 4% python, and ram is around 5-6GB but nothing happening. I got no error message so hard to figure out what went wrong. I tried to edit the webui-user and play with the arguments but not solved problem.

Steps to reproduce the problem

1. source venv/bin/activate
2. TORCH_COMMAND='pip install torch torchvision --extra-index-url https://download.pytorch.org/whl/rocm5.1.1' python launch.py --precision full --no-half

or

1. ./webui.sh

What should have happened?

Should not stall.

Commit where the problem happens

5c1cb9263f980641007088a37360fcab01761d37

What platforms do you use to access UI ?

Linux

What browsers do you use to access the UI ?

Mozilla Firefox

Command Line Arguments

export COMMANDLINE_ARGS="--no-half --precision full --opt-split-attention --opt-sub-quad-attention --api"

export TORCH_COMMAND='pip install torch torchvision --extra-index-url https://download.pytorch.org/whl/rocm5.1.1' python launch.py --precision full --no-half

Additional information, context and logs

No response

[Liso7150]

i have same problem with webui_user: @echo off

set PYTHON= set GIT= set VENV_DIR= set COMMANDLINE_ARGS=--xformers --ckpt-dir "E:\Important\Models" --vae-path "E:\Important\VAE\novelailatest-pruned.vae.pt" --lowvram --use-cpu esrgan, interrogate --opt-sub-quad-attention --administrator --gradio-img2img-tool color-sketch --enable-console-prompts --api --embeddings-dir "E:\Important\embeddings" --hypernetwork-dir "E:\Important\hyper_network" git pull call webui.bat

[Liso7150]

i have same problem with webui_user: @echo off

set PYTHON= set GIT= set VENV_DIR= set COMMANDLINE_ARGS=--xformers --ckpt-dir "E:\Important\Models" --vae-path "E:\Important\VAE\novelailatest-pruned.vae.pt" --lowvram --use-cpu esrgan, interrogate --opt-sub-quad-attention --administrator --gradio-img2img-tool color-sketch --enable-console-prompts --api --embeddings-dir "E:\Important\embeddings" --hypernetwork-dir "E:\Important\hyper_network" git pull call webui.bat

so eather --api or --opt-sub-quad-attention is a problem

[Liso7150]

gs" --hypernetwork-dir "E:\Important\hyper_network" git pull call webui.bat

so eather --

testet hypotase and it is not this

[Liso7150]

[Liso7150]

it how it look, python 3.10.6, 3.10.7 and 3.10.9 tested

[palinko91]

so eather --api or --opt-sub-quad-attention is a problem

I'm not so sure about that, because I tried also to start with every line in webui-user.sh commented out. Then removed the venv folder and __pycache__ and started the webui.sh. Then the weights not loaded at all:

After I wrote this bug issue, I let my pc on all night and basically time not solving the issue. If it not loads in lets say 2 min ideally, but lets say max 10 min, it will never load :(

Now I git pulled to 602a1864b05075ca4283986e6f5c7d5bce864e11, always hoping a new version might fix the issue but not yet.

[ClashSAN]

@Liso7150 you're on windows. just use --xformers flag first. on a regular sd-1.4 .ckpt model to test

@palinko91 try regular 1.4 model .ckpt too.

[ClashSAN]

@palinko91 https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/6709/files the latest launch file switched to 5.2,

[Liso7150]

@Liso7150 you're on windows. just use --xformers flag first. on a regular sd-1.4 .ckpt model to test

@palinko91 try regular 1.4 model .ckpt too.

But i had it....

[Liso7150]

@Liso7150 you're on windows. just use --xformers flag first. on a regular sd-1.4 .ckpt model to test

@palinko91 try regular 1.4 model .ckpt too.

I want still use my marge....

[Liso7150]

@Liso7150 you're on windows. just use --xformers flag first. on a regular sd-1.4 .ckpt model to test

@palinko91 try regular 1.4 model .ckpt too.

why this now?

[Liso7150]

[palinko91]

@palinko91 https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/6709/files the latest launch file switched to 5.2,

Okay I commented out the TORCH_COMMAND cause I learned meanwhile I don't need the --precision full --no-half because that helping with the old cards.

I downloaded this one: https://huggingface.co/CompVis/stable-diffusion-v-1-4-original

And using commit 28189985e6f56dc725938a3f0e4d2462dad74bc5 for the test. Venv, pycache deleted before the test. The result is no different.

[palinko91]

Meantime I've localised this part from webui.py starting but never finishing:

So in /modules/sd_models.py the load_model() calling a function named load_model_weights(). Everything is fine until the if cache_enabled: arm have a line stating this: model.logvar = model.logvar.to(devices.device) # fix for training

This stucking and not going trough so the sd_vae.delete_base_vae() never reached. I think this might help.

I tried to dissasembe that line and try what could cause the stalling. Maybe this will be helpful:

>>> from modules import devices
>>> print(devices.device)
None
>>> print(devices.get_cuda_device_string())
No module 'xformers'. Proceeding without it.
cuda
>>> print(devices.get_optimal_device())
cuda
>>> 

[ClashSAN]

@Liso7150 do you have at least 16gb of regular ram? And how much vram (gpu ram) do you have? If you successfully run it before a while ago, maybe the ram usage has increased due to new features and additions.

@palinko91 I'm unfamilar with that distro, is this compatible with it? The launch scripts are created for ubuntu primarily i'm guessing. Maybe test the pre-release commit here bef193189500884c2b20605290ac8bef8251a788 on ubuntu first to see if there are some breaking changes for linux

[Aamir3d]

@ClashSAN - yep - low RAM can be an issue

[palinko91]

Haha funny I've changed from 22.10 ubuntu to this distro cause I heard it working fine here. And least didn't had to install drivers here, correct python version was preinstalled etc. I don't have ubuntu installed now and I don't think that would solve the issue since I've tested it there before.

[palinko91]

I've commented out that model.logvar.to part and the script going forward until reaching this in the sd_models.py

if shared.cmd_opts.lowvram or shared.cmd_opts.medvram:
        lowvram.setup_for_low_vram(sd_model, shared.cmd_opts.medvram)
    else:
        sd_model.to(shared.device)

I got the else, so sd_model.to is failing again. model is basically an sd_model also. It's a LatentDiffusion object. So now the script stalling there. Trying to figure out what to do.

I've tried shared.device in venv python3 shell it's return cuda.

If I'm commenting out the else: then I can progress in the shell I'm getting:

Loading weights [dcd690123c] from /home/palinko/Desktop/stable-diffusion-webui/models/Stable-diffusion/v2-1_768-ema-pruned.safetensors
Applying sub-quadratic cross attention optimization.

If I want to progress further I have to comment out the "sd_hijack.model_hijack." things but then okay I can get the UI but nothing gonna work haha.

[palinko91]

I was able to recreate the problem in venv python shell, this ends in forever loading also :)

[Liso7150]

@Liso7150 do you have at least 16gb of regular ram? And how much vram (gpu ram) do you have? If you successfully run it before a while ago, maybe the ram usage has increased due to new features and additions.

@palinko91 I'm unfamilar with that distro, is this compatible with it? The launch scripts are created for ubuntu primarily i'm guessing. Maybe test the pre-release commit here bef193189500884c2b20605290ac8bef8251a788 on ubuntu first to see if there are some breaking changes for linux

16 ram 8 vram

[palinko91]

32GB RAM and 16GB VRAM in my case, the UI even recognise it when I get in before as commentes out every line which stalling, but ofcourse it was unuable haha

Torch active/reserved: 0/0 MiB, Sys VRAM: 0/16368 MiB (0.0%)

[ClashSAN]

@Liso7150 you may have to reinstall webui, your latest log is showing "couldn't install torch" I think your older version is clashing with that.

@palinko91 The state of things for linux installation may be in flux, as I do not have linux installation I didn't test. You can revert to the surrounding commits 1d prior to https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/6709 if sorely needed. Follow the https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Install-and-Run-on-AMD-GPUs#running-natively "running natively section" as it applied to linux+amd users before the PR for the script file.

[palinko91]

@palinko91 The state of things for linux installation may be in flux, as I do not have linux installation I didn't test. You can revert to the surrounding commits 1d prior to #6709 if sorely needed. Follow the https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Install-and-Run-on-AMD-GPUs#running-natively "running natively section" as it applied to linux+amd users before the PR for the script file.

Yep I used that how to before. I'm using the newest commit, and launched with this:

export TORCH_COMMAND='pip install torch torchvision --extra-index-url https://download.pytorch.org/whl/rocm5.2' python launch.py --skip-torch-cuda-test --use-cpu all --precision full --no-half

And now works, and I thought it will using CPU but it's not. Somehow still using my GPU, so maybe it's the solution now, so weird.

[ClashSAN]

ah. what a weird day

[Liso7150]

@Liso7150 you may have to reinstall webui, your latest log is showing "couldn't install torch" I think your older version is clashing with that.

@palinko91 The state of things for linux installation may be in flux, as I do not have linux installation I didn't test. You can revert to the surrounding commits 1d prior to #6709 if sorely needed. Follow the https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Install-and-Run-on-AMD-GPUs#running-natively "running natively section" as it applied to linux+amd users before the PR for the script file.

reinstalled and still not fixed