search

vladmandic / automatic

SD.Next: Advanced Implementation Generative Image Models
https://github.com/vladmandic/automatic
GNU Affero General Public License v3.0
5.77k stars 432 forks source link

[Diffusers]: black images/errors when using Lora, SDXL works fine #1981

Closed freke70 closed 1 year ago

freke70 commented 1 year ago

Issue Description

I am using sd_xl_base_1.0_0.9vae.safetensors and can generate images without issue. However, when I add a LoRA module (created for SDxL), I encounter problems:

With one LoRA module, the generated images are completely black. With another LoRA module (https://civitai.com/models/122359/detail-tweaker-xl), I get errors in the logs.

I have turned on --debug and --safe, but that did not resolve the issues.

When I remove the LoRA module, image generation works fine again.

I am using Ubuntu 22.04 with an NVIDIA RTX 6000 GPU.

Could you please provide some pointers on where I should start troubleshooting? What might be causing the LoRA modules to not work properly with my base model?

Diffusers pipeline used

sd_xl_base_1.0_0.9vae.safetensors, diffusers backend

Version Platform Description

Create and activate python venv Launching launch.py... 12:37:02-368518 INFO Starting SD.Next 12:37:02-373666 INFO Python 3.10.10 on Linux 12:37:02-376916 INFO Version: 417ef540 Tue Aug 8 12:05:30 2023 -0400 12:37:02-690416 INFO nVidia CUDA toolkit detected 12:37:02-705782 INFO Verifying requirements 12:37:02-723784 INFO Verifying packages 12:37:02-726011 INFO Verifying repositories 12:37:05-137065 INFO Verifying submodules 12:37:36-442197 INFO Extensions enabled: ['stable-diffusion-webui-images-browser', 'Lora', 'SwinIR', 'LDSR', 'stable-diffusion-webui-rembg', 'clip-interrogator-ext', 'ScuNET', 'sd-extension-system-info', 'sd-webui-agent-scheduler', 'ultimate-upscale-for-automatic1111', 'Stable-Diffusion-Webui-Civitai-Helper', 'openOutpaint-webUI-extension', 'adetailer', 'sd-webui-infinite-image-browsing']

Relevant log output

13:20:26-856772 ERROR    Diffusers LoRA loading failed: add-detail-xl local variable 'state_dict' referenced before assignment
Calculating model hash: /home/nfrederickx/automatic/models/Lora/add-detail-xl.safetensors ━━━━━━━━━━━━━━━━━━━━━━━━━ 8/569 bytes -:--:--
13:20:26-861034 ERROR    activating extra network lora with arguments [<modules.extra_networks.ExtraNetworkParams object at
                         0x7f978332d780>]: OSError
╭───────────────────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────────────────╮
│ /home/nfrederickx/automatic/modules/extra_networks.py:71 in activate                                                                │
│                                                                                                                                     │
│    70 │   │   try:                                                                                                                  │
│ ❱  71 │   │   │   extra_network.activate(p, extra_network_args)                                                                     │
│    72 │   │   except Exception as e:                                                                                                │
│                                                                                                                                     │
│ /home/nfrederickx/automatic/extensions-builtin/Lora/extra_networks_lora.py:23 in activate                                           │
│                                                                                                                                     │
│   22 │   │                                                                                                                          │
│ ❱ 23 │   │   lora.load_loras(names, multipliers)                                                                                    │
│   24                                                                                                                                │
│                                                                                                                                     │
│                                                       ... 3 frames hidden ...                                                       │
│                                                                                                                                     │
│ /home/nfrederickx/automatic/modules/hashes.py:76 in addnet_hash_safetensors                                                         │
│                                                                                                                                     │
│   75 │   offset = n + 8                                                                                                             │
│ ❱ 76 │   b.seek(offset)                                                                                                             │
│   77 │   for chunk in iter(lambda: b.read(blksize), b""):                                                                           │
│                                                                                                                                     │
│ /home/nfrederickx/automatic/venv/lib/python3.10/site-packages/rich/progress.py:262 in seek                                          │
│                                                                                                                                     │
│    261 │   def seek(self, offset: int, whence: int = 0) -> int:                                                                     │
│ ❱  262 │   │   pos = self.handle.seek(offset, whence)                                                                               │
│    263 │   │   self.progress.update(self.task, completed=pos)                                                                       │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
OSError: [Errno 22] Invalid argument

Acknowledgements

freke70 commented 1 year ago

Hi, sorry, maybe it is very lora dependant. This one does work: https://civitai.com/models/117635/greg-rutkowski-inspired-style-lora-sdxl

No idea how to know which ones will work and which ones won't.

vladmandic commented 1 year ago

Fixed in dev, will be posted to main branch soon.