XD111ds commented 1 month ago

thank for your code,but why is the image I generated completely black, the answer is important for me.looking forward to your reply.the following is my code:

import torch from diffusers import DiffusionPipeline from utils import ( attn_maps, cross_attn_init, register_cross_attention_hook, set_layer_with_name_and_path, save_by_timesteps_and_path, save_by_timesteps )

1. Init modules

cross_attn_init() ########################### import torch from diffusers import StableDiffusionXLPipeline, UNet2DConditionModel, EulerDiscreteScheduler from huggingface_hub import hf_hub_download from safetensors.torch import load_file import pandas as pd

base = "stabilityai/stable-diffusion-xl-base-1.0" repo = "ByteDance/SDXL-Lightning" ckpt = "sdxl_lightning_4step_unet.safetensors" # Use the correct ckpt for your step setting!

Load model.

unet = UNet2DConditionModel.from_config(base, subfolder="unet").to("mps", torch.float16) unet.load_state_dict(load_file(hf_hub_download(repo, ckpt), device="mps")) pipe = StableDiffusionXLPipeline.from_pretrained(base, unet=unet, torch_dtype=torch.float16, variant="fp16").to("mps")

Ensure sampler uses "trailing" timesteps.

pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing")

pipe = DiffusionPipeline.from_pretrained(

"stabilityai/stable-diffusion-xl-base-1.0",

torch_dtype=torch.float16,

)

pipe = pipe.to("mps")

2. Replace modules and Register hook

pipe.unet = set_layer_with_name_and_path(pipe.unet) pipe.unet = register_cross_attention_hook(pipe.unet) ################################################

height = 512 width = 768 prompt = "A portrait photo of a kangaroo wearing an orange hoodie and blue sunglasses standing on the grass in front of the Sydney Opera House holding a sign on the chest that says 'SDXL'!."

image = pipe( prompt, height=height, width=width, num_inference_steps=15, ).images[0] image.save('test.png')

3. Process and Save attention map

print('resizing and saving ...')

3-1. save by timesteps and path (2~3 minutes)

save_by_timesteps_and_path(pipe.tokenizer, prompt, height, width) #########################################################

3-2. save by timesteps (1~2 minutes)

save_by_timesteps(pipe.tokenizer, prompt, height, width)

################################################

wooyeolBaek commented 1 month ago

@XD111ds It seems that the black attention map issue is caused by setting device = 'mps'. I also use macOS, and the same issue occurs when using MPS. Unfortunately, I haven't been able to resolve it yet, so I recommend using the CUDA environment instead, which can be set with device = 'cuda'.