search

Lightning-AI / torchmetrics

Torchmetrics - Machine learning metrics for distributed, scalable PyTorch applications.
https://lightning.ai/docs/torchmetrics/
Apache License 2.0
2.07k stars 395 forks source link

Add `BaryScore` #852

Open stancld opened 2 years ago

stancld commented 2 years ago

🚀 Feature

Add BaryScore

Sources:

Motivation

The recent NLG metrics are more often based on BERT (or related) embeddings. As such, I believe, we should also start adding such metrics into TorchMetrics with an extra dependency on transformers if a user wants to use any of these metrics. The BaryScore metric is from a family of untrained metrics (i.e. the model is not fine-tuned on any specific task) so it should be easier for us to begin with it.

Abstract:

A new metric BaryScore to evaluate text generation based on deep contextualized embeddings e.g., BERT, Roberta, ELMo) is introduced. This metric is motivated by a new framework relying on optimal transport tools, i.e., Wasserstein distance and barycenter. By modelling the layer output of deep contextualized embeddings as a probability distribution rather than by a vector embedding; this framework provides a natural way to aggregate the different outputs through the Wasserstein space topology. In addition, it provides theoretical grounds to our metric and offers an alternative to available solutions e.g., MoverScore and BertScore). Numerical evaluation is performed on four different tasks: machine translation, summarization, data2text generation and image captioning. Our results show that \texttt{BaryScore} outperforms other BERT based metrics and exhibits more consistent behaviour in particular for text summarization.

ashutoshml commented 2 years ago

Interested!

stancld commented 2 years ago

Great, @ashutoshml! Just let's wait on #849 as we discussed as there'll be many common dependencies :]

stancld commented 2 years ago

Hi @ashutoshml - I'd like to apologize for my delay here. I've been quite busy and unable to complete my part yet. Gonna try to proceed this weekend as much as possible.

stancld commented 2 years ago

Hello @ashutoshml, very sorry for my really huge delay, last months have been pretty turbulent for me. Let me know if you're still interested in contributing this metric to torchmetrics. If so, I believe you can checkout branch metric/InfoLM and start working on BaryScore metric implementation. One thing is that the reference metrics are still not possible to install as a package. We are thus enforced to hardcode the results in our tests. If you want to generate test results for your metric, I really recommend you to use my repo infolm-docker. Would be also happy if you send a PR there to contain all reference results on one place.

ashutoshml commented 2 years ago

Hi @stancld. I hope things are ok now. I will be able to take up this only after Aug 5.

stancld commented 2 years ago

Hi @ashutoshml, it's completely fine - the time is really flying so we're almost there. If you're then interested, just let me know. I have some WIP we can discuss and can create a MR together :]

stancld commented 2 years ago

Helo @ashutoshml, any updates here? :]

ashutoshml commented 2 years ago

Hi @stancld - Was occupied with some work. I will have a look at it this weekend.

Borda commented 1 year ago

I will have a look at it this weekend.

great, looking forward @ashutoshml :rocket: