search

Cuberick-Orion / CIRR

Official repository of ICCV 2021 - Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models
https://cuberick-orion.github.io/CIRR/
MIT License
100 stars 3 forks source link

Clarification on Table 3: "Retrieval performance on CIRR" #3

Closed ABaldrati closed 3 years ago

ABaldrati commented 3 years ago

Hello

I would like to have some clarifications on Table3: “Retrieval performance on CIRR”: In particular I would like to ask whether the query image (a.k.a the reference image) is removed from the retrieved results when calculating metrics.

From the results, particularly the Random (theoretical) row, we can infer this. In fact if the query image was not removed the _Recallsubset@1 should be 1/6 (16.6%) instead of 1/5 (20%).

Before proceeding with my experiments I would like to have a confirmation directly from you.

Thanks for the amazing work

Cuberick-Orion commented 3 years ago

Hi,

Thank you for your interest in our work.

To answer your question: Yes, the query (reference) image is removed when computing metrics - this applies to both the Recall and the Recall_Subset metrics.

The evaluation procedures can be found here starting from line 274 to line 299. In particular, the removal of the query image is on line 296.

Please note that the above links point to a different repository that hosts our model (hence, also the evaluation code).

The behavior is consistent with previous work (TIRG).

ABaldrati commented 3 years ago

Great! Thanks @Cuberick-Orion for the prompt and detailed reply!