search

KevinMusgrave / pytorch-metric-learning

The easiest way to use deep metric learning in your application. Modular, flexible, and extensible. Written in PyTorch.
https://kevinmusgrave.github.io/pytorch-metric-learning/
MIT License
5.95k stars 657 forks source link

Best way to add the NDCG to the AccuracyCalculator #339

Open elias-ramzi opened 3 years ago

elias-ramzi commented 3 years ago

Hi Kevin,

I am looking to add the Normalized Discounted Cumulative Gain (NDCG) metric to the AccuracyCalculator while also using the GlobalEmbeddingSpaceTester.

The NDCG allows the use of non binary similarity labels in contrast to other metrics already implemented in the AccuracyCalculator.

In my case I want to compute the NDCG for a dataset with labels belonging to a Hierarchical tree (eg. Stanford Online Products).

My first thought was to use multiple label per sample (one for each level of the hierarchy tree) to compute the relevance for the NDCG. But in GlobalEmbeddingSpaceTester only a single label is kept, so in the AccuracyCalculator I do not have access to multi-labels.

I also want to keep the other metrics (eg. mAP@R) so I don't think I can pass a custom label_comparison_fn ?

Do you see an easier way to compute the NDCG ?

Thanks for any tips !

KevinMusgrave commented 3 years ago

Hmm, I can't think of an easy way to implement this. Perhaps the design can be improved.

My first thought was to use multiple label per sample (one for each level of the hierarchy tree) to compute the relevance for the NDCG. But in GlobalEmbeddingSpaceTester only a single label is kept, so in the AccuracyCalculator I do not have access to multi-labels.

Given that AccuracyCalculator can operate on 2d labels, it would make sense for GlobalEmbeddingSpaceTester to allow this.

I also want to keep the other metrics (eg. mAP@R) so I don't think I can pass a custom label_comparison_fn ?

AccuracyCalculator could take some init argument that tells it which label dim each metric should use.

elias-ramzi commented 3 years ago

Thanks for the response !

I will think of a way to do that. If I find a not too hacky way of doing that I will submit a PR.

Including a change to the GlobalEmbeddingSpaceTester, allowing to pass 2d labels to the AccuracyCalculator.

Given that AccuracyCalculator can operate on 2d labels, it would make sense for GlobalEmbeddingSpaceTester to allow this.

And a more modular use of the hierarchy level in the AccuracyCalculator.

AccuracyCalculator could take some init argument that tells it which label dim each metric should use.