search

ssitu / ComfyUI_UltimateSDUpscale

ComfyUI nodes for the Ultimate Stable Diffusion Upscale script by Coyote-A.
GNU General Public License v3.0
856 stars 58 forks source link

Add color color correction. #103

Open andrej-szontagh opened 2 months ago

andrej-szontagh commented 2 months ago

Hi USDU Developers! I have found out that color correction node is a great way how to reduce tile visibility on higher denoise values, however I can only apply color correction to the whole upscaled image and for that I can only use wavelet method because other modes are far less effective, however on higher denoise values you can get also a lot of structural change so that is causing color bleed and something I would call for a lack of a better word a "color blurriness". I believe that greater results could be achieved if color correction is applied on each tile separately which could also make the other color-correction modes more usable. It seems that Automatic1111 has this. Consider adding this option. Thank you.

andrej-szontagh commented 2 months ago

I have made a comparison. The last one is made manually and put together in photoshop. These images are over the top with a too high denoise value, but they illustrate my point better.

Input raw upscaled image: rawupscale

The USDU output with 32 mask blur usdu

The same image but with a Wavelet color correction afterwards usdu+wavelet

The same image but with Adain color correction per each tile and put together in photoshop. The underlying image is the previous one (Wavelet) and the tiles are blended over it with a 30px radius mask blend. usdu+adain_per_tile

alex-mitov commented 2 months ago

The same image but with Adain color correction per each tile and put together in photoshop. The underlying image is the previous one (Wavelet) and the tiles are blended over it with a 30px radius mask blend.

Hi, @andrej-szontagh . Can you kindly explain a bit better the steps for the last image? What do you mean by "color correction per each tile"? And what exactly do you mean by "blending the tiles over it"? Also, how did you combine everything in the end in Photoshop? I am having the same problems with higher denoise and it's a bit frustrating. The results in your last image look pretty good and seems to solve the problem.

Thanks in advance!

andrej-szontagh commented 2 months ago

Hi Alex, I have actually written a set of custom nodes myself just for this specific thing. Basically, try this, Use USDU with high denoise, offset:0, padding:0 .. this way you will see individual tiles which will be quite different. Then add color correction node ("Image Color Match") at the end and set it to "wavelet" .. set the reference image to the original image before upscale. You will see that the visible tiling is greatly reduced. One of the reasons for this is that even if you impaint with the mask the output image color balance is usually slightly off even on pixels that should stay the same .. so by correcting the color you can greatly reduce the issue of visible tiling.

Sometimes it's not about the seams, but the white on one tile is not exactly the same white on the neighbour tile. This happens even with relatively low denoise values if you do something like a plain smooth plaster wall .. and this type of issue can be greatly improved by color correction. However the "wavelet" mode has a downside and that it basically causes color bleeding especially on higher denoise values. So I have found out that I can use different color correction modes that doesn't have this issue on each individual tile and then put them together. I have made bunch of nodes which allow me to do whatever I want with each tile.

alex-mitov commented 2 months ago

Oh, cool! Are these publicly available nodes? Yes, so far I've been using the wavelet color correction with the color correction node, but as you stated, it tends to cause bleeding.

andrej-szontagh commented 2 months ago

Not, yet. I am quite busy, but I am thinking about it. Basically the idea is that I can generate the masks and split image into tiles as list of images which you can then process as normal images sequentially, then I have another special node that puts all the tiles together and also supports extra masks. So it's like USDU, but instead of a monolithic node you get to build the whole pipeline and can use and process intermediate result however you wish and then put everything together. You can actually do that with the nodes that are already available, but it's going to very rigid and hard to deal with. Basically my first prototype had like at least 100 nodes or more. With custom nodes I have reduced it by an order of magnitude. However my setup is till pretty huge, but 10x less huge :) ..