Add `DepthScore`

[stancld]

🚀 Feature

Add DepthScore

Sources:

• Paper: A Pseudo-Metric between Probability Distributions based on Depth-Trimmed Regions
• Repo

Motivation

The recent NLG metrics are more often based on BERT (or related) embeddings. As such, I believe, we should also start adding such metrics into TorchMetrics with an extra dependency on transformers if a user wants to use any of these metrics. The DepthScore metric is from a family of untrained metrics (i.e. the model is not fine-tuned on any specific task) so it should be easier for us to begin with it.

Abstract:

The design of a metric between probability distributions is a longstanding problem motivated by numerous applications in Machine Learning. Focusing on continuous probability distributions on the Euclidean space ℝ^d, we introduce a novel pseudo-metric between probability distributions by leveraging the extension of univariate quantiles to multivariate spaces. Data depth is a nonparametric statistical tool that measures the centrality of any element x∈ℝ^d with respect to (w.r.t.) a probability distribution or a data set. It is a natural median-oriented extension of the cumulative distribution function (cdf) to the multivariate case. Thus, its upper-level sets -- the depth-trimmed regions -- give rise to a definition of multivariate quantiles. The new pseudo-metric relies on the average of the Hausdorff distance between the depth-based quantile regions w.r.t. each distribution. Its good behavior w.r.t. major transformation groups, as well as its ability to factor out translations, are depicted. Robustness, an appealing feature of this pseudo-metric, is studied through the finite sample breakdown point. Moreover, we propose an efficient approximation method with linear time complexity w.r.t. the size of the data set and its dimension. The quality of this approximation as well as the performance of the proposed approach are illustrated in numerical experiments.

[hassiahk]

Hi. Can I work on this?

[stancld]

Hi @hassiahk, yes you can take this one! But please give us some extra time to finish #849 first as there will be some common dependencies. I'll ping you once #849 is ready.

[hassiahk]

Sure!

[stancld]

Hi @hassiahk - I'd like to apologize for my delay here. I've been quite busy and unable to complete my part yet. Gonna try to proceed this weekend as much as possible.

[hassiahk]

Hi @hassiahk - I'd like to apologize for my delay here. I've been quite busy and unable to complete my part yet. Gonna try to proceed this weekend as much as possible.

That's totally fine! @stancld :]

[stancld]

Hello @hassiahk , very sorry for my really huge delay, last months have been pretty turbulent for me. Let me know if you're still interested in contributing this metric to torchmetrics. If so, I believe you can checkout branch metric/InfoLM and start working on DepthScore metric implementation. One thing is that the reference metrics are still not possible to install as a package. We are thus enforced to hardcode the results in our tests. If you want to generate test results for your metric, I really recommend you to use my repo infolm-docker. Would be also happy if you send a PR there to contain all reference results on one place.

[hassiahk]

Hello @hassiahk , very sorry for my really huge delay, last months have been pretty turbulent for me. Let me know if you're still interested in contributing this metric to torchmetrics. If so, I believe you can checkout branch metric/InfoLM and start working on DepthScore metric implementation. One thing is that the reference metrics are still not possible to install as a package. We are thus enforced to hardcode the results in our tests. If you want to generate test results for your metric, I really recommend you to use my repo infolm-docker. Would be also happy if you send a PR there to contain all reference results on one place.

@stancld will start working on this.

[stancld]

@hassiahk Great! Definitely let me know if you need any help :]

[stancld]

Hello @hassiahk, any updates here? O:]