NDCG computation

[malonsocortes]

Hello @benfred. I think there might be an issue with the computation of NDCG in ranking_metrics_at_k. I believe NDCG should be calculated by first sorting the recommended ids by the recommendation scores, so that the logarithmic discount is applied in the correct ranking of the recommendations. Right now, Implicit's recommend methods do not return the ids sorted by their descending scores, so the way NDCG is being calculated does not take into account the order of the ranking.

I would appreciate your opinion, maybe I'm missing something and I'm wrong. Thanks in advance.

If changes turn out to be necessary, I would suggest adding some lines on this part of ranking_metrics_at_k

while start_idx < len(to_generate):
        batch = to_generate[start_idx: start_idx + batch_size]
        ids, scores = model.recommend(batch, train_user_items[batch], N=K)
        start_idx += batch_size

        # added
        sorted_scores = np.flip(np.argsort(scores, axis=1), axis=1)
        ids = ids[np.arange(ids.shape[0])[:, None], sorted_scores]

[benfred]

The recommend methods in implicit should return ids sorted by their scores - so I don't think this is a problem

Are you seeing instances where models aren't returning ids sorted by their scores?

[malonsocortes]

Thanks for your reply.

I was having unsorted ids with the CosineRecommender from the ItemItemRecommenders. Here are some examples:

You can see how 0,5 goes after 0,9:

And here, it is in between the 0,9s:

[Bernhard-Steindl]

I guess this issue can be closed, since the bug of sorting the output of KNN models was fixed in November 2022 with #629. @malonsocortes Let us know if the issue still remains unsolved for you.