isi-vista / unified-io-inference

Apache License 2.0
0 stars 0 forks source link

Run VizWiz captioning baseline and ALT prompts and scoring on larger (~5K images) dataset #25

Open danielnapierski opened 1 year ago

danielnapierski commented 1 year ago

I have run the baseline captioning VizWiz on the standard test 8000 image leaderboard.

danielnapierski commented 1 year ago

Baseline results on standard 8000 test:

[{"test": {"B1": 55.59, "B2": 37.76, "B3": 25.32, "B4": 16.71, "METEOR": 17.56, "ROUGE-L": 38.71, "CIDEr": 48.25, "SPICE": 13.13}}]

vizwiz_joined_captions.json.txt

I split the execution across 10 GPUs by splitting the vizwiz test.json annotation file into 10 files. I ran 10 docker processes (each using a single GPU on 1 of 3 gaia-lg machines). I then combined all the results and submitted one json file.

danielnapierski commented 1 year ago
Team B1 B2 B3 B4 ROUGE-L METEOR CIDEr SPICE
Unified-io-inference docker 55.59 37.76 25.32 16.71 38.71 17.56 48.25 13.13
danielnapierski commented 1 year ago

The baseline captioning implementation would place 9th on the VizWiz Captioning 2021 Leaderboard: https://eval.ai/web/challenges/challenge-page/739/leaderboard/2006