search

rohitgandikota / unified-concept-editing

Unified Concept Editing in Diffusion Models
https://unified.baulab.info
MIT License
98 stars 3 forks source link

Reproduce results in Table 2 #19

Open shaswati1 opened 3 months ago

shaswati1 commented 3 months ago

Hi @rohitgandikota, Thanks for this amazing work!

I tried to reproduce the results in table 2 and followed below steps:

  1. erased concept "Cassette Player" and got "erased model erased-cassette_player-towards_uncond-preserve_false-sd_1_4-method_replace.pt"
  2. generated 500 samples per class using the prompts in "unified-concept-editing/data/imagenette.csv"
  3. computed top1 classification score using "eval-scripts/imageclassify.py" using the prompts in "unified-concept-editing/data/imagenette.csv"
  4. aggregated the top1 score based on the class to compute the mean top1 score for each of the 10 classes

However, I got 0.19 as accuracy for the erased class i.e. "Cassette Player" while table 2 in the paper shows 0.0. Am I missing any steps? Can you please help in this regard?

rohitgandikota commented 3 weeks ago

Yes you have it right till step 3. Step 4 is basically we take the classification class rather than the score it self. We classify an image into 1 of the 10 classes and compare the total number before and after. If you see 16% for original model that means 16 percent of the 500 images generated by the original model were actually classified as cassette player by the classification model.