Stable Diffusion SD XL support

facebookincubator / AITemplate

AITemplate is a Python framework which renders neural network into high performance CUDA/HIP C++ code. Specialized for FP16 TensorCore (NVIDIA GPU) and MatrixCore (AMD GPU) inference.

Apache License 2.0

4.57k stars 372 forks source link

Stable Diffusion SD XL support #851

Open CyberTimon opened 1 year ago

CyberTimon commented 1 year ago

Hello!

It would be super cool to accelerate the stable diffusion xl models, as they are pretty slow because of the 1024x1024 res.

I think it could work pretty easily, the only problem seems to be the second text encoder.

What do you guys think or has anyone successfully compiled the xl models?

hlky commented 1 year ago

UNet2DConditionModel and UNet blocks need updating for architecture changes and for extra conditionings
CLIP needs to output correct hidden state layer, also for bigG, text projection and pooled output
Official VAE overflows in fp16, either use community fp16 fixed version or modify VAE to run in fp32
Pipeline changes (extra conditioning, concat clip, denoise for refiner etc)

JJHu1993 commented 1 year ago

@hlky is there any plan to support sdxl? If yes, when will it be finished?

hlky commented 1 year ago

@JJHu1993 I have sdxl support in my private projects. When I have time I will bring those changes here.

JJHu1993 commented 1 year ago

@hlky Anticipating your sdxl changes with great excitement, I believe that the integration of sdxl support will significantly enhance the influence of the AITemplate project.

CyberTimon commented 1 year ago

@JJHu1993 I have sdxl support in my private projects. When I have time I will bring those changes here.

Yes it would be great to see these changes. I can't wait to have a faster SDXL. Please share it!

Thank you very much!

BahzBeih commented 1 year ago

@JJHu1993 I have sdxl support in my private projects. When I have time I will bring those changes here.

thanks a lot, this will very great, i have a question, did AMD RX Vega 56 supported by AITemplate ?

CyberTimon commented 1 year ago

Any news? @hlky

hlky commented 1 year ago

@CyberTimon I'll do it today.

CLIP changes are done, but need testing for numerical accuracy, also, there is a workaround for the limitations of dynamic_slice with regards to pooled output.
For UNet the crop conditioning is done outside of UNet, this isn't 100% faithful to Diffusers implementation, but it is slightly faster by ~0.1it/s in my testing (2.1it/s vs 2.2it/s, 3060 12GB, 1024x1024), the crop conditioning is the same every iteration so there's no need to do it inside UNet.
VAE is compatible if you use the fp16 fixed version, I'll include changes that pass dtype everywhere so it can be ran in fp32 if desired, and an option that overrides the pipeline's VAE with the fixed version.
Bringing over the XL pipeline and modifying it to work with AIT is the main thing at this point. PR will be limited to XL text-to-image pipeline for simplicity.

CyberTimon commented 1 year ago

Thank you very much! @hlky

aycaecemgul commented 1 year ago

is there an update on this? the comments are deleted