search

WASasquatch / was-node-suite-comfyui

An extensive node suite for ComfyUI with over 210 new nodes
MIT License
1.15k stars 170 forks source link

[FEATURE REQUEST] WAS_Latent_Noise - latent mask #44

Closed MoonMoon82 closed 1 year ago

MoonMoon82 commented 1 year ago

Could you please make WAS_Latent_Noise aware of samples["noise_mask"] ? So if you already set a latent mask, the Noise Injection Node would only add noise the the masked / not masked area.

WASasquatch commented 1 year ago

I'll think about that. Right now I'm taking a break from ComfyUI as it's just uesless for my needs due to Comfy's "personal choices".

The point of the Latent Noise injector node is for people that aren't using an upscale model, because the latent upscale itself will cause anomalies in the diffusion process. So if you use like 0.4 noise injection, your final HR fix result will be of higher quality and not have weird dots in it or other anomalies from latent upscaling.

WASasquatch commented 1 year ago

I couldn't figure out how to mask the tensor. The masks that ComfyUI is creating is not compatible with the latent samples tensors, and the masks used for inpainting are apparently handled at diffusion time, as all you do is set samples['noise_mask'] = TheMask you don't actually mask anything in the code int he node for masking a image for inpainting.

MoonMoon82 commented 1 year ago

I already saw it in the code.. I tried to figure out how 'noise_mask' is handled in the sampler, but I'm far not skilled enough to find the right part in the source. And the other thing is: I wanted to add noise to the out-masked latent area to make the out-masked area disappear completely. Usually even the out-masked area stays in the result. For example, if the out-masked area is a white background, as a result from the sampler, you get a (nearly) white background again. Instead I think, it would be more accurate to replace the out-masked area with noise instead of adding it. But that's far far away from what I can do :( I tried to replace the out-masked area with noise in the input image - right before the vae-encoder. But the results are terrible.

In case of VAEEncodeForInpaint I really don't understand anything. I thought there could be a way to inject noise, but it also returns 'noise_mask'

WASasquatch commented 1 year ago

LDM doesn't actually do well with random noise like Guided Diffusion (Disco Diffusion). It can really only handle low levels of noise that trick or up to do something different