huggingface / optimum-intel

🤗 Optimum Intel: Accelerate inference with Intel optimization tools
https://huggingface.co/docs/optimum/main/en/intel/index
Apache License 2.0
355 stars 99 forks source link

Safety checker doesn't work with Latent Consistency Model pipeline #790

Closed adrianboguszewski closed 1 day ago

adrianboguszewski commented 1 week ago

When I run the following code:

from diffusers.pipelines.stable_diffusion import StableDiffusionSafetyChecker
from optimum.intel import OVLatentConsistencyModelPipeline

safety_checker = StableDiffusionSafetyChecker.from_pretrained("CompVis/stable-diffusion-safety-checker")
ov_pipeline = OVLatentConsistencyModelPipeline.from_pretrained("OpenVINO/LCM_Dreamshaper_v7-int8-ov", device="AUTO")
ov_pipeline.safety_checker = safety_checker

result = ov_pipeline(prompt="", num_inference_steps=2, width=256, height=256, guidance_scale=8.0).images

I get: TypeError: Cannot handle this data type: (1, 1, 256), |u1

The problem is with the tensor layout. The safety checker expects HWC and gets CHW.

Also, assigning the checker in from_pretrained doesn't have any effect.