search

DrCyanide / cyanic-sd-krita

A Stable Diffusion plugin for Krita, compatible with A1111 and SD.Next
68 stars 4 forks source link

Upscaling workflow and adding more detail? #22

Closed rexelbartolome closed 9 months ago

rexelbartolome commented 9 months ago

Now that I can use it more often with SD.Next. I'm not entirely sure how to approach upscaling/adding details in this plugin. Originally in interpause's plugin, I just used SD Upscale. But now we have Controlnet Tile, Tiled Diffusion/VAE, Ultimate SD Upscale, etc. I'm not sure what this plugin is designed to do/utilize in that regard. Do let me know your workflow process in general I'd love to know :)

Currently struggling to make Mixture of Diffusers work and not sure what I'm doing wrong. I looked it up and found that I need to change some compute related settings like no half and fp32 precision etc. It works but it doesn't look good. I enable Controlnet Tile but that pushes it to the limit and I get OOM errors (I'm on an RTX 2060 w 6GB VRAM). Can't even use Lora's as I think that adds to the OOM stuff. So I'm thinking I might be going about it the wrong way...

DrCyanide commented 9 months ago

I don't have a specific or recommended approach to upscaling. I've personally never used Controlnet Tile, Tiled Diffusion, or Ultimate SD Upscale, so I can't speak for what does the best job. Since all of those are extensions, and extensions have to have their UI manually converted into Krita, the only one supported at the moment is ControlNet Tile. There's enough features I want to add that I will use (and enough bugs to fix) that writing UI for other extensions isn't something I'm interested in. If someone else writes an extension UI and does a Pull Request I'd be happy to consider adding it to the repo.

The standard upscalers work, and have their own tab. You can choose if it will resize your entire canvas or not, and if not you can resize the upscaled layer like you would any pasted in image that's too big (Krita 5.1.5 doesn't give me control of Transform Masks, which is why I haven't automatically shrunk it to fit).

As for Diffusers, that's something I never got working with SD.Next, so I have no tips there either.

If you're on 6GB VRAM and going OOM you could try passing the --lowvram parameter into SD.Next when it boots up. I have 8GB, and I use to get OOM errors somewhat frequently before I added that (especially if SD had been running for a long time or switching models a lot). Since adding it, I don't think I've gotten another OOM error.

Personally, I use Stable Diffusion as more of a collage source image generator, so having things like Remove Background automatically make and attach transparency masks to images is a big deal for me. I can generate those source images at full resolution, then scale them to fit where I need them, and finally do a light denoising Img2Img to blend it together better. I want to add features like a ControlNet Segmentation Map color search and a Webcam-as-layer input to help with that compositing workflow, then I might look more at different upscaling options (like adding it to Img2Img)

Anyway... I think other questions related to "How would you upscale in this situation?" might be better suited to the general Stable Diffusion Reddit. Since I'm trying to keep this Issues section focused on Issues and Feature Requests, I'm going to go ahead and mark this as closed.

rexelbartolome commented 9 months ago

Yes of course, that's understandable. Just wondering if you had a system developed already for this plugin :)

I think Controlnet Tile is the one I need to look into the most as it can do a lot as people have said

image

That's interesting how you use it to make collages, I have collage artists/graphic design friends that I also recommended SD to, but they're kind of reluctant on trying it. Not to mention the managing of python installations, the many different confusing settings, and the stigma of using AI in general...

rexelbartolome commented 9 months ago

I forgot to ask, how does the plugin handle resolution? In interpause's it has a minimum and maximum size, afaik that's been removed here in favor of high res fix no?

DrCyanide commented 9 months ago

I forgot to ask, how does the plugin handle resolution? In interpause's it has a minimum and maximum size, afaik that's been removed here in favor of high res fix no?

Right now, it uses your selection/layer/canvas size as the size it passes to Stable Diffusion.

If Auto High Res Fix is enabled, and your selected size is bigger than the HR Min size, then it'll trigger High Res Fix.

High Res Fix scales the selection size down so that the smallest dimension is the SD Min Size, then passes that and your actual size to Stable Diffusion. If I understand correctly, the server will generate at the smaller size, then do an Img2Img to upscale it.

At some point I'd like to implement an overall minimum size, so selecting something smaller than 512x512 will pass scaled up dimensions to Stable Diffusion and the result can be scaled down in Krita again, but I haven't gotten to that yet.

DrCyanide commented 9 months ago

Just added a minimum size check, in the Settings page.

If the area you've selected is smaller on one side than whatever you set as the minimum size (default is 512) the plugin will ask Stable Diffusion to generate an image of the same ratio, where the smallest size is the minimum size. In Krita, the preview and the final image will be automatically rescaled down to the selected size. This should make it significantly easier to add small details that are still high quality.