Anyone know how to use xinsir/controlnet-tile-sdxl-1.0?

[pandayummy]

Your question

I know it might not be the right place to ask this question, but the huggingface forum just no infomation. So I come here to see if someone could share a workflow on https://huggingface.co/xinsir/controlnet-tile-sdxl-1.0

Logs

No response

Other

No response

[raysers]

我推测你看得懂中文视频，因此给一个B站链接，关于xinsir的promax使用，tile版估计也差不多。链接：【Union模型发布ProMax版本，SDXL已成究极形态，xinsir剑指SD3控制之王-哔哩哔哩】 https://b23.tv/pI2aJlb

[ArubaitoDev]

Here's how I use it: tiled-diffusion-example.json

Basic steps:

1. Generate your image
2. Upscale it normally (model, lanczos, etc)
3. Apply TilePreprocessor to the full image before sending to the controlnet
4. Apply tile controlnet to the conditioning and preprocessed image
5. Apply TiledDiffusion (Mixture of Diffusers) to the model before sending it to the resampler
6. Tiled vae encode
7. Resample
8. Tiled vae decode

Things to know:

• This depends on custom nodes
  • Tiled Diffusion & VAE for ComfyUI: VAEEncodeTiled_TiledDiffusion, TiledDiffusion, VAEDecodeTiled_TiledDiffusion
  • ComfyUI's ControlNet Auxiliary Preprocessors: TilePreprocessor
• Adjust tile size on the encoder/decoder (bigger = more VRAM). Recreate the nodes to get defaults for your system (I think?)
• You may need to mess with the Tiled Diffusion parameters too if VRAM is an issue
• MultiDiffusion never worked for me, but Mixture of Diffusers did
• It doesn't really matter if the seeds are the same or not
• For denoise on the resampler, 0.5 is the highest I'll go. Otherwise it changes too much. I use multiple detailer and upscale passes, and for each pass I'll lower the denoise value as the image becomes larger and more refined. You should experiment with this value and find what works for you.

Hope that helps.

Edit: Clarification, forgot to mention tile preprocessor

[pandayummy]

Here's how I use it: tiled-diffusion-example.json

Basic steps:

1. Generate your image
2. Upscale it normally (model, lanczos, etc)
3. Apply TilePreprocessor to the full image before sending to the controlnet
4. Apply tile controlnet to the conditioning and preprocessed image
5. Apply TiledDiffusion (Mixture of Diffusers) to the model before sending it to the resampler
6. Tiled vae encode
7. Resample
8. Tiled vae decode

Things to know:

• This depends on custom nodes

  • Tiled Diffusion & VAE for ComfyUI: VAEEncodeTiled_TiledDiffusion, TiledDiffusion, VAEDecodeTiled_TiledDiffusion
  • ComfyUI's ControlNet Auxiliary Preprocessors: TilePreprocessor
• Adjust tile size on the encoder/decoder (bigger = more VRAM). Recreate the nodes to get defaults for your system (I think?)
• You may need to mess with the Tiled Diffusion parameters too if VRAM is an issue
• MultiDiffusion never worked for me, but Mixture of Diffusers did
• It doesn't really matter if the seeds are the same or not
• For denoise on the resampler, 0.5 is the highest I'll go. Otherwise it changes too much. I use multiple detailer and upscale passes, and for each pass I'll lower the denoise value as the image becomes larger and more refined. You should experiment with this value and find what works for you.

Hope that helps.

Edit: Clarification, forgot to mention tile preprocessor

Thank you for your detailed instructions. I have a new question now, the TilePreprocessor seems like image blur, what’s the point of blurring the input image if we’re going to use it to condition the image generation process?

[ArubaitoDev]

I have a new question now, the TilePreprocessor seems like image blur, what’s the point of blurring the input image if we’re going to use it to condition the image generation process?

I honestly don't know; I think I just copied some nodes from a youtube video. So I went on a quest to find out. The source code for TilePreprocessor has a comment with a link to TTPLanet_SDXL_Controlnet_Tile_Realistic on HuggingFace right above the apply_gaussian_blur method. On this HF model card we can find the explanation for blurring:

If you encounter the edge halo issue with t2i or i2i, particularly with i2i, ensure that the preprocessing provides the controlnet image with sufficient blurring. If the output is too sharp, it may result in a 'halo'—a pronounced shape around the edges with high contrast. In such cases, apply some blur before sending it to the controlnet. If the output is too blurry, this could be due to excessive blurring during preprocessing, or the original picture may be too small.

However, we're not using the TTPLANET_Controlnet_Tile_realistic controlnet model, we're using the model from xinsir. It may not have the same "edge halo" issue.

I did some tests without TilePreprocessor and didn't notice the issue. VRAM usage is the same with or without it too. So TilePreprocessor is probably unnecessary here.

[pandayummy]

So ControlNet tile is meant for detailing low resolution images. We don't need any image preprocessor for the conditioning image unless the output being too sharp. Cheers!