search

PaddlePaddle / PaddleMIX

Paddle Multimodal Integration and eXploration, supporting mainstream multi-modal tasks, including end-to-end large-scale multi-modal pretrain models and diffusion model toolbox. Equipped with high performance and flexibility.
Apache License 2.0
242 stars 101 forks source link

ToME + ControlNet is supported and how much is the speed ratio? #462

Open bigmover opened 3 months ago

bigmover commented 3 months ago

Have you modified the implementation of controlnet to acheive ToME + ControlNet？

JunnYu commented 3 months ago

you can test speed use this code

# install develop paddlemix
pip install git+https://github.com/PaddlePaddle/PaddleMIX.git
# install paddle
pip install paddlepaddle-gpu==0.0.0.post117 -f https://www.paddlepaddle.org.cn/whl/linux/gpu/develop.html
import paddle
from ppdiffusers import ControlNetModel, StableDiffusionControlNetPipeline
from ppdiffusers.utils import load_image

controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny")
pipe = StableDiffusionControlNetPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", safety_checker=None, controlnet=controlnet, paddle_dtype=paddle.float16
)

# Apply ToMe with a 50% merging ratio
pipe.unet.apply_tome(ratio=0.5) 
pipe.controlnet.apply_tome(ratio=0.5) 

generator = paddle.Generator().manual_seed(0)
prompt = "bird"
image = load_image(
    "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/sd_controlnet/bird_canny.png"
)

image = pipe(prompt, image, generator=generator).images[0]

image.save("bird.png")