search

jonasricker / aeroblade

[CVPR2024] AEROBLADE: Training-Free Detection of Latent Diffusion Images Using Autoencoder Reconstruction Error
https://openaccess.thecvf.com/content/CVPR2024/html/Ricker_AEROBLADE_Training-Free_Detection_of_Latent_Diffusion_Images_Using_Autoencoder_Reconstruction_CVPR_2024_paper.html
40 stars 6 forks source link

compute_distances() fails #1

Closed EricJi150 closed 7 months ago

EricJi150 commented 7 months ago

When running python scripts/run_aeroblade.py compute_distance always fails and gets killed. The error is:

You are using a model of type xlm-roberta to instantiate a model of type M-CLIP. This is not supported for all configurations of models and can yield errors. Killed

jonasricker commented 7 months ago

The warning about xlm-roberta is expected and should not be an issue. The "Killed" sounds like you might be running out of memory. Can you check your RAM usage while running the script?

EricJi150 commented 7 months ago

looks like memory usage hits 100% and the program is killed, any suggestions? I am just trying to generate the LPIPS difference between an original and reconstructed image to get a similar difference image as shown in figure 1 of the paper.

jonasricker commented 7 months ago

How much memory do you have and how large is the image?

EricJi150 commented 7 months ago

My system has a total of about 334 GB. I am trying to use an image of size 512x512.

jonasricker commented 7 months ago

Ok, then memory should definitely not be a problem. Could you try to debug where exactly the memory starts to fill up?

Btw, you could also have a look at experiments/00_other_figures.ipynb to recreate Figure 1, since run_aeroblade.py does not directly yield the error maps.

EricJi150 commented 7 months ago

00_other_figures.ipynb is what I was looking for thanks! However, I inputted an image with inpainted objects created from SD 1-2, but did not get the results I expected clearly outlining the mask.

inpainted_dog image

jonasricker commented 7 months ago

If you created the inpainting using SD1.2 you would have to compute the reconstruction using the AE from SD1. So in the notebook, replace repo_id="stabilityai/stable-diffusion-2-base" with repo_id="CompVis/stable-diffusion-v1-1" within the call(s) to compute_reconstructions.

EricJi150 commented 7 months ago

I switched the AE, but doesn't seem like I can achieve the results from the paper. I have the mask for reference.

download download (1) Capture

jonasricker commented 7 months ago

From the last image it looks like the parts outside the mask are also passed through the autoencoder (one can tell by the malformed wooden grid in the back of the bench). In this case, our method cannot locate the inpainted regions, because the entire image has a low reconstruction error. In our experiments, the regions outside the mask were copied from the original image. I guess that different implementations of inpainting might handle this differently.

EricJi150 commented 7 months ago

Makes sense thank you for your help!

jonasricker commented 6 months ago

Please have a look at #2 for more info on how to inpaint without modifying the unmasked area.