Split zero-shot and retrieval results csv files

[rwightman]

I feel the results csv has been getting a lot of eyes with it here closer to the models, esp with the new model additions.

Would it be worth adding retrieval results in a separate csv?

[gabrielilharco]

We have Flickr and COCO retrieval numbers on that table, are you thinking of some other datasets?

[gabrielilharco]

Right now we show average R@1 (image-to-text and text-to-image), but maybe we should expand this to show a breakdown (also R@5, R@10)

[rwightman]

@gabrielilharco ah, didn't even realize that was retrieval w/o the R@

[rwightman]

although, might make sense to break out into a diff table as zero-shot ranking and retrieval ranking be of separate interest to many down stream uses and those rankings are clearly different...

[rwightman]

further, if we split retrieval into its own csv, makes sense to add more columns (R@1, R@5, R@10, image-to-text, and text-to-image) also useful because comparing to papers there isn't always consistency in terms of which numbers they're using

[gabrielilharco]

Will do @rwightman. We also have WinoGAViL as part of the 38 datasets (https://arxiv.org/abs/2207.12576). Maybe we can leave those three datasets (Flickr, MSCOCO and WinoGAViL) in the main table for completion and consistency with the datacomp benchmark, and keep a separate table for the three retrieval datasets?

[rwightman]

@gabrielilharco yeah, could leave the main table as is, the full summary (also the link is out there so shouldn't change the name)... maybe even split off a zs-only variant since it's just post-filtering? I think more people would be happy to see a more detailed retrieval table as it's a big use case for these models.

[gabrielilharco]

Agreed, just did a PR, let me know how those look to you. Was surprised at how the rankings change!

[rwightman]

@gabrielilharco awesome, yeah that's pretty useful to have them organized like this!