Integrate HiDiffusion

[ShenZhang-Shin]

Is your feature request related to a problem? Please describe. Object duplication and low efficiency happens when generating high-resolution image (2048x2048, 4096x4096) for stable diffusion models. HiDiffusion can solve these problems, increasing the resolution and speed of pretrained diffusion models. paper link; project link; code link

text-to-image: 2048x2048 resolution images generated by HiDiffusion

HiDiffusion also supports controlnet and other tasks. controlnet: 2048x2048 resolution images

Efficiency comparison with other acceleration methods, HiDiffusion can significantly reduce time consumption without compromising image quality.

Describe the solution you'd like. HiDiffusion code is based diffusers and is designed as a plug-and-play implementation. It can be integrated into diffusers pipelines by only adding a single line of code! Users just needs to install hidiffusion

pip3 install hidiffusion

and use it by adding a single line of code

apply_hidiffusion(pipeline)

We want to integrate HiDiffusion into the official code and documentation so that more people can access this feature.

Describe alternatives you've considered. None

Additional context. None

[hidiffusion]

cc @a-r-r-o-w @sayakpaul if you're interested in this work!

[DN6]

Hi @ShenZhang-Shin I think we can start by adding to the community pipelines here since the API design is a bit different. I'm thinking we could do something like

sd_pipe = StableDiffusionPipeline.from_pretrained("...")
pipe = HiDiffusionPipeline.from_pipe(sd_pipe)

The HiDiffusionPipeline could then encapsulate the logic you have in the apply_hi_diffusion function?

[ShenZhang-Shin]

Hi @DN6 Adding to the community pipelines and using HiDiffusionPipeline.from_pipe(sd_pipe) is a great choice. However, this approach differs from the API in the HiDiffusion repository (apply_hidiffusion(pipeline)), which might confuse users about why there are two different APIs. Additionally, adding to the community pipelines requires copying a portion of the HiDiffusion code into the diffusers pipelines. This will be a hassle, as I plan to update HiDiffusion to support more models in the future continually.

I believe it is better and more flexible to add a section about HiDiffusion in the Hugging Face Diffusers documentation so that we don't need to create a new API.

[DN6]

Hi @ShenZhang-Shin yeah we can definitely add a section in the docs to highlight the project. Will open a PR soon for it. cc: @stevhliu

[ShenZhang-Shin]

@DN6 Thanks!

[ShenZhang-Shin]

Hi, @DN6 @stevhliu I am wondering if there have been any updates on the PR

[DN6]

@ShenZhang-Shin Just opened the PR here https://github.com/huggingface/diffusers/pull/9013

[ShenZhang-Shin]

Hi @DN6 Thank you for your support! It looks quite good.

After viewing the community projects, I found many interesting works and can't wait to try them out!

Additionally, I think placing hidiffusion in the INFERENCE TECHNIQUES or ADVANCED INFERENCE sections would be a great choice. I will write a documentation about HiDiffusion. If I want to contribute to the diffusers documentation, can I submit a commit in https://github.com/huggingface/diffusers/pull/9013, or should I create a new PR?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.