[Feature Request]: Kwai-Kolors Support

[Vigilence]

https://github.com/Kwai-Kolors/Kolors

[lquesada]

there's nothing I have to do to add support to new models, just use them in the sampler.

[Vigilence]

@lquesada I tried this before making this request, atm kolors is not working with this node. It does work if I instead do in painting with the detailed node of the inspire/impact packs.

If your tests turned out different please let me know!

[lquesada]

you probably did something wrong, as these nodes output images and masks - I'd suggest you to read kolors documentation and check that you're meeting the requirements (image size, specs, etc). Sorry, I do not have time to test these nodes with all other existing nodes, models, or hardware combinations.

Alternatively, you could also provide example screenshots of workflows (e.g. a workflow using kolors that works for you and a workflow using kolors and these nodes that doesn't work for you) and the error, and I could look through them.

[Vigilence]

Hello!

Like I said earlier, kolors inpaint works fine in segs detailer with the impact pack. If I use the same settings with your node then it results in a bad inpaint.

The workflow with your node is the one resulting in a bad image, the workflow with the segs detailer results in a flawless inpaint. I switched to flux in the last workflow image since that is what I am currently using, but it was previously using kolors model and was working great.

Maybe its the settings I am using for your node, but the same settings with the model switched to flux result in a great image.

Workflow with your node that produces bad results:

Impact Segs detailer that produces flawless results:

Here is a Kolors Inpainting1.json of crop and stitch and Segs Detailer combined so you can compare.

[lquesada]

sorry, but I do not understand what is the issue. Is it the quality of the sampling? I do not see the flawless results to compare.

However, different models work better at different resolutions, different models are better for inpainting than others, and my nodes work slightly different to other nodes. Maybe you're hitting a combination of different behavior of nodes that makes the model inpaint look worse.

There's nothing that I can do to adapt that to work better with Kolors though. In the end... my nodes just crop and then stitch the image back - so to Kolors that is just inpainting on an image with a mask - nothing special because of my nodes.

[Vigilence]

Yours on the left, segs detailer on the right. Not sure why your node would produce such a pixelated inpaint compared to segs detailer. If you can take a look at my workflow and point out any flaws or improvements for your node settings I would appreciate it!

Kolors Inpainting1.json

Pictures from same workflow, Left to Right:

[lquesada]

Thanks for providing details!

The main difference is with my nodes you're rendering at a resolution 1024x1024 to 2048x2048, with segs detailer you're doing 1024x1024 to 1280x1280. With my nodes, that is making it as close to 2048x2048 as possible, which is probably too much.

Change min_width and min_height to 768 and max_width and max_height to 1024, or just set them all to 1024. You may also try 1280x1280 but that may still be too much. I believe trying different sizes should produce better results.

Additionally, I'd suggest to increase the context area a bit so more of the face is seen. This will provide more context to the sampler.

[Vigilence]

Hi Luis!

I tried the suggestions you provided and there was a slight improvement but the pixelation was still there. After some further testing I found the issue for the pixelation, it was the rescale algorithim which was set to nearest on the inpaint crop node. Once I switched to bislerp the pixelation is removed.

I noticed that perhaps the segs detailer does something under the hood when it comes to inpainting and the way it treats the model. I am not using it for segs or bbox, (dummy nodes are used for bbox and the segs is just the same mask used for crop and stitch), its only used for inpainting.

I noticed that the segs will create a quality inpaint even at higher denoise while this node will deteriorate badly. My guess is that the detailer node has something that deals with the kolors model better and avoids deformations where as the crop and stitch node is just a basic inpaint.

At 1.0 denoise, same ksampler settings etc. Crop and stitch top, detailer node bottom:

Thanks again for the help. 😊

[lquesada]

I see. Then the only other difference is the scheduler setup you're using for the sampler. I'd suggest to try other samplers in your crop-stitch setup, e.g. euler sampler with sgm_uniform scheduler instead of dpmpp_2m_sde_gpu with karras.

It seems the model is having trouble converging to something reasonable, which may be that Kolors (and I don't really know about that model) requires a different sampler.

Try euler sampler with sgm_uniform scheduler and try other combinations, maybe?