search

salaniz / pycocoevalcap

Python 3 support for the MS COCO caption evaluation tools
Other
293 stars 82 forks source link

Any plans for adding SPICE metric? #6

Closed v-iashin closed 4 years ago

v-iashin commented 4 years ago

Are there any plans for adding SPICE metric?

The official repo has it already.

salaniz commented 4 years ago

I think that's a good idea. I had a quick look and it seems that using SPICE requires an initial bash script to run. Since I would like this repository to be easy to install via pip, I will have a look how to best implement this. The options I see now is to use setup.py to either run bash commands from within python or rewrite the bash script to a python script. I am leaning towards the second option because it might allow for better error handling since everything is in python.

salaniz commented 4 years ago

On a different note, including SPICE will significantly increase the installed package size. The SPICE library is approx. 385 MB in size. So I think it might actually be better to only load this data when it is first used, instead of doing it in setup.py during pip installation.

salaniz commented 4 years ago

This is finally fixed in b6e7a050c3e325ac4bc18afd0163eafb4325aec4.

Note that my implementation automatically downloads the Stanford library (385 MB) required by SPICE the first time a SPICE evaluation is performed. Moreover, SPICE takes a while the first time it is called, but it is sped up subsequently due to caching. Because of these two points, however, the first run will be quite slow.