search

Acly / krita-ai-diffusion

Streamlined interface for generating images with AI in Krita. Inpaint and outpaint with optional text prompt, no tweaking required.
https://www.interstice.cloud
GNU General Public License v3.0
6.47k stars 312 forks source link

Layer Diffusion integration #1150

Open paprik opened 3 weeks ago

paprik commented 3 weeks ago

Basic implementation of ComfyUI Layer Diffusion: https://github.com/huchenlei/ComfyUI-layerdiffuse Uses two new custom nodes and adds a new transparency checkbox for the Generate action.

Showcase: transparency_test

paprik commented 3 weeks ago

Created as draft since I likely missed something. Generate (with upscaling too) works great. Could not manage to make it work with Refine/Inpaint yet, it keeps adding a white background for some reason.

Acly commented 3 weeks ago

For Refine/Inpaint a special VAE encode is needed that understands transparency. The regular VAE only does RGB (ignores alpha channel).

It was added in forge a while ago, but is not implemented in the ComfyUI nodes.

How can the 2-pass generation (with upscale step) work though? It should run into the same problem?

paprik commented 3 weeks ago

You're right, 2-pass upscale doesn't work, I've just generated a 2k in one step. Hopefully it gets added to ComfyUI nodes soon.

I still have to try Attention injection, because the Conv one changes the model a quite a lot.