Deploy on Hugging Face

[giuseppecartella]

Hi!. First of all congratulations for your work!

I was wondering if you plan to upload on the diffusers library the instructpix2pix model finetuned on magic brush? Thank you!

[drogozhang]

Hi,

Thanks for your interest and great suggestion! We will look into it and let you know when the checkpoint is uploaded to diffusers library.

Best, Kai

[vinesmsuic]

I made a huggingface diffusers port of official magicbrush weights.

• https://huggingface.co/vinesmsuic/magicbrush-paper
• https://huggingface.co/vinesmsuic/magicbrush-jul7

[giuseppecartella]

Hi @vinesmsuic, Thank you so much for the effort!. However when loading the model I face the following isseu:

Exception has occurred: RuntimeError Error(s) in loading state_dict for StableDiffusionSafetyChecker: size mismatch for vision_model.vision_model.embeddings.class_embedding: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]). size mismatch for vision_model.vision_model.embeddings.position_ids: copying a param with shape torch.Size([1, 257]) from checkpoint, the shape in current model is torch.Size([1, 50]). size mismatch for vision_model.vision_model.embeddings.patch_embedding.weight: copying a param with shape torch.Size([1024, 3, 14, 14]) from checkpoint, the shape in current model is torch.Size([768, 3, 32, 32]). size mismatch for vision_model.vision_model.embeddings.position_embedding.weight: copying a param with shape torch.Size([257, 1024]) from checkpoint, the shape in current model is torch.Size([50, 768]). size mismatch for vision_model.vision_model.pre_layrnorm.weight: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]). size mismatch for vision_model.vision_model.pre_layrnorm.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]). size mismatch for vision_model.vision_model.encoder.layers.0.self_attn.k_proj.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([768, 768]).

Even adding requires_safety_checker=False, safety_checker=None does not solve the problem.

Do you have any hints? Thank you

[vinesmsuic]

Hi @vinesmsuic, Thank you so much for the effort!. However when loading the model I face the following isseu:

Exception has occurred: RuntimeError Error(s) in loading state_dict for StableDiffusionSafetyChecker: size mismatch for vision_model.vision_model.embeddings.class_embedding: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]). size mismatch for vision_model.vision_model.embeddings.position_ids: copying a param with shape torch.Size([1, 257]) from checkpoint, the shape in current model is torch.Size([1, 50]). size mismatch for vision_model.vision_model.embeddings.patch_embedding.weight: copying a param with shape torch.Size([1024, 3, 14, 14]) from checkpoint, the shape in current model is torch.Size([768, 3, 32, 32]). size mismatch for vision_model.vision_model.embeddings.position_embedding.weight: copying a param with shape torch.Size([257, 1024]) from checkpoint, the shape in current model is torch.Size([50, 768]). size mismatch for vision_model.vision_model.pre_layrnorm.weight: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]). size mismatch for vision_model.vision_model.pre_layrnorm.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([768]). size mismatch for vision_model.vision_model.encoder.layers.0.self_attn.k_proj.weight: copying a param with shape torch.Size([1024, 1024]) from checkpoint, the shape in current model is torch.Size([768, 768]).

Even adding requires_safety_checker=False, safety_checker=None does not solve the problem.

Do you have any hints? Thank you

never ran into this error. can you share the code snippet to reproduce this error?

[giuseppecartella]

from PIL import Image, ImageOps import requests import torch from diffusers import StableDiffusionInstructPix2PixPipeline, EulerAncestralDiscreteScheduler from PIL import Image

url = "https://huggingface.co/datasets/diffusers/diffusers-images-docs/resolve/main/mountain.png"

def download_image(url): image = Image.open(requests.get(url, stream=True).raw) image = ImageOps.exif_transpose(image) image = image.convert("RGB") return image

image = download_image(url) prompt = "make the mountains snowy"

class MagicBrush(): def init(self, weight="vinesmsuic/magicbrush-jul7"): self.pipe = StableDiffusionInstructPix2PixPipeline.from_pretrained( weight, torch_dtype=torch.float16, requires_safety_checker=False, safety_checker=None ).to("cuda") self.pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(self.pipe.scheduler.config)

def infer_one_image(self, src_image, instruct_prompt, seed):
    generator = torch.manual_seed(seed)
    image = self.pipe(instruct_prompt, image=src_image, num_inference_steps=20, image_guidance_scale=1.5, guidance_scale=7, generator=generator).images[0]
    return image

model = MagicBrush() image_output = model.infer_one_image(image, prompt, 42) image_output

I run the exact snippet you posted on hugging face page

[giuseppecartella]

Instead when using the checkpoint "vinesmuic/magicbrush-paper" the error is different

'AttentionBlock' object has no attribute 'to_q' File "/homes/gcartella/Projects/fake_detection_saliency/dataset_creation/try_magicbrush.py", line 20, in __init__ self.pipe = StableDiffusionInstructPix2PixPipeline.from_pretrained( File "/homes/gcartella/Projects/fake_detection_saliency/dataset_creation/try_magicbrush.py", line 31, in <module> model = MagicBrush() AttributeError: 'AttentionBlock' object has no attribute 'to_q'

[vinesmsuic]

@beppefolder try updating your diffusers.

[giuseppecartella]

I had an old version of diffusers and transformers libraries. I updated to the latest versions and now it is perfectly working!! Thank you so much for the help!!!! Giuseppe

[drogozhang]

@vinesmsuic Thank you so much for your effort!!

[drogozhang]

Thanks Max again for the effort! Now both MagicBrush checkpoints have been deployed in diffusers and it's time to close the issue.