carson-katri / dream-textures

Stable Diffusion built-in to Blender

GNU General Public License v3.0

7.82k stars 427 forks source link

SDXL with ControlNet does not respond to depth #712

Closed Anarchid closed 12 months ago

Anarchid commented 1 year ago

Description

I managed to get dreamtextures working, and with SD2 it works very nice. But i also would like to run it with SDXL.

It looks like the UI supports using a ControlNet instead of an already depth-conditioned model, and the downloader thing allows me to find and download a depth ControlNet for SDXL.

The dream projection UI then also correctly blocks the use of naked SDXL for projection, but re-allows it after i connect the depth ControlNet.

However, the generated image is not affected by the depth map at all, and is a plain SDXL text-to-image.

The same setup with the same checkpoints, works fine via A1111, e.g. if i obtain a depth image from Blender and paste it manually into webUI.

Of note, i'm running this on ubuntu with a 4090 from latest git checkout.

Steps to Reproduce

Install sdxl-1.0 and sdxl-1.0 depth controlnet
use the local transformers backend
select sdxl as model for generation -- disable half precision optimization - otherwise an unrelated error will occur
select "use controlNet" and sdxl-1.0 depth controlnet as the checkpoint
select an object and some polygons, project dream texture
the projected image does not show any hints of reacting to the polygons

Expected Behavior

SDXL with depth controlnet should behave pretty much similar to how sd-2.0-depth behaves.

Addon Version

Other (Built from source)

GPU

NVIDIA

Anarchid commented 1 year ago

From browsing around the source it looks like there are specific configs for using controlnets with other checkpoints (cldm_v15.yaml, cldm_v21.yaml), but no such for sdxl? Does something like this need to be configured specifically?

NullSenseStudio commented 1 year ago

The primary issue seems to be that using ControlNet for depth projection isn't working in general, it's not a SDXL specific issue.

Those config files are only used for checkpoint conversion when using checkpoint importing or checkpoint linking. It doesn't require a SDXL ControlNet config file because these models so far are only distributed as diffusers weights.

Anarchid commented 1 year ago

Is there any likely place where it could be fruitful for me to look around rather than wait for more competent people to deal with it? :)

carson-katri commented 1 year ago

@Anarchid This should be resolved by #717, a new build is being created here: https://github.com/carson-katri/dream-textures/actions/runs/6678624526?pr=711

Let me know if you still have issues and I can look into further fixes.

Anarchid commented 1 year ago

The fix works, thank you greatly!

Anarchid commented 1 year ago

Hm, looks like it only works for the first time per blender run. Restarting Blender helps, but after the first click on "project dream texture" any following attempts are met with this:

Anarchid commented 1 year ago

Switching to sd2 depth (without contorlnet) and back to sdxl with cn also resets the problem.

carson-katri commented 1 year ago

@Anarchid Good catch. This seems to be a bug in diffusers when using auto pipelines. I've opened an issue there, but in the meantime I'll look for a workaround.

carson-katri commented 1 year ago

720 works around this issue for now. Let me know if you run into any other issues with the latest on `releases/0.3.0`. A new build is being created here: https://github.com/carson-katri/dream-textures/actions/runs/6714616416?pr=711

Anarchid commented 1 year ago

This seems to solve the previous problem, but now i get a Cuda OOM error on second execution - seems that the SDXL gets loaded into memory for a second time, where it doesn't quite fit, sadly.

carson-katri commented 1 year ago

The AutoPipeline bug was fixed in upstream diffusers, so I've updated the dependency in #721 and reverted the workaround. Everything should be back to normal.

Anarchid commented 12 months ago

After updating addon and diffusers, everything flies smooth now.