search

huggingface / evaluate

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.
https://huggingface.co/docs/evaluate
Apache License 2.0
1.95k stars 247 forks source link

Add the CIDEr metric? #114

Open zuujhyt opened 3 years ago

zuujhyt commented 3 years ago

Hi, I find the api in https://huggingface.co/metrics quite useful. I am playing around with video/image captioning task, where CIDEr is a popular metric. Do you plan to add this into the HF datasets library? Thanks.

Kamino666 commented 2 years ago

same

lvwerra commented 2 years ago

For reference: https://github.com/vrama91/cider

cc @sashavor

Guitaricet commented 2 years ago

Official CIDER implementation uses PTBTokenizer from Stanford CoreNLP. Unfortunately, it requires java and a .jar file with the library. What do you think could be an acceptable way to add CIDER to Evaluate?

dinhanhx commented 11 months ago

Official CIDER implementation uses PTBTokenizer from Stanford CoreNLP. Unfortunately, it requires java and a .jar file with the library. What do you think could be an acceptable way to add CIDER to Evaluate?

We can do like NLTK library, have the users provide path to jar file.

gabrielmotablima commented 5 months ago

Relevant topic. I'm working on vision-language models for low-resource languages, and it would be really helpful to have metrics like CIDEr-D, CIDEr-R, SPICE, and SPIDEr available in Evaluate 🐧🐧🐧🐧🐧🐧

CurryYuan commented 4 months ago

Any progress?

Kami-chanw commented 3 weeks ago

I've opened a pr.