search

google-deepmind / magiclens

[ICML'24 Oral] "MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions"
Apache License 2.0
141 stars 8 forks source link

Evaluation and Reproduction #6

Closed baohl00 closed 3 months ago

baohl00 commented 3 months ago

As I've read the MagicLens paper, you use mAP@k as the evaluation metric for CIRCO dataset. I want to ask for the evaluation code because I tried to reproduce the result but it seems not exactly as the numbers in paper.

Thanks for your help!

drogozhang commented 3 months ago

Hi, Thanks for your interest in our work. The results of CIRCO will be a JSON file, which should be submitted to https://circo.micc.unifi.it/evaluation for evaluation because the test set of CIRCO is not publicly available.

For the evaluation code, please refer to their implementation: https://github.com/miccunifi/CIRCO/blob/main/src/evaluation.py

baohl00 commented 3 months ago

As I notice, CoCa cannot be called by scenic repo so I cannot reproduce this model either. Can I ask for this model archive or how I can use it for your code.

Moreover, is it possible to change the vision tower in MagicLens config into another version?

Thank you for your help again!

drogozhang commented 3 months ago

Hi CoCa and its variants can not be released due to policy. There's nothing I can do about that.