search

HanSolo9682 / CounterCurate

This is the implementation of CounterCurate, the data curation pipeline of both physical and semantic counterfactual image-caption pairs.
https://countercurate.github.io/
16 stars 0 forks source link

Metric mismatch with paper #4

Closed ozmig77 closed 3 months ago

ozmig77 commented 6 months ago

Hello nice work, thanks for the code and dataset release.

I have one question regard calculation of metric in position task. In paper, the score is calculated as text-to-image manner: image But in code, the score is calculated with image-to-text manner, S(C, I) > S(C', I) + S(C', I') > S(C, I') https://github.com/HanSolo9682/CounterCurate/blob/main/benchmark/CLIP_test_position.py#L30 Can you clarify this?

Thanks

HanSolo9682 commented 6 months ago

Hi, thank you for your discovery! It seems like the paper did have a typo in how we conducted our experiments. It indeed should be the “image-to-text” manner you mentioned, which corresponds to our code, instead of what the paper currently describes. We will fix this in the next revised version of the paper.