Inquiry to ask hyperparameter tuning of Lcon weight

[john09282922]

Thanks for nice works! and Congratulation on CVPR24 as well. I am trying to run your model using your code. However, I did not get any satisfaction on image editing. For example, when I change prompt < A "CAT" to "Pig" sitting to next mirror >, it is not working well. Looks like using e instead of self-attention. Can you tell me why it is happening? Can you give me an advice?

thanks, John

[HyelinNAM]

Hi @john09282922! Thank you for your interest in my work.

Since there are some randomly determined components in this framework, the quality of results may vary. Nevertheless, I obtained the images below with my code. Here's the command that I used for this image. (I just added num_inference_steps arguments for convenience. Please pull code again.)

python run.py --prompt "cat sitting next to mirror" --trg_prompt "pig sitting next to mirror" --w_cut 2.0 --num_inference_steps 300

Additionally, regarding the feature that I'm using, please refer to the code below.

1. save self-attention features https://github.com/HyelinNAM/ContrastiveDenoisingScore/blob/285d1ccce307e1d8c0298bcdfb29de7776b5d691/pipeline_cds.py#L166-L169

2. get & load self-attention features > calculate cut loss https://github.com/HyelinNAM/ContrastiveDenoisingScore/blob/285d1ccce307e1d8c0298bcdfb29de7776b5d691/pipeline_cds.py#L190-L195

[john09282922]

Thanks for the information and editing code as well!

Results is much better!.

I have a question "what is attn1"? Is there other attn?

Thanks, John

[HyelinNAM]

I'm glad to be able to help you! Basically, stable diffusion architecture has two type of attention, which are self attention and cross attention. Attention blocks that contain 'attn1' in their name are the self-attention block.

[john09282922]

thanks