Qrels and Run query ids do not match

[paguasmar]

Describe the bug I am evaluating my test set against my algorithm recommendations, but it gives the error Qrels and Run query ids do not match

To Reproduce Steps to reproduce the behavior: metrics = ["recall@10", "mrr@10","ndcg@10"] person_date_indexs = df_recs_train_top10['person_date_index'].unique() run = tranform_recs_to_ranx(df_recs_train_top10, person_date_indexs, "person_date_index", "gym_id", "rank") qrels = tranform_test_to_ranx(df_test_checkins_new_col_renamed, df_recs_train_top10) evaluate(qrels, run, metrics)

Expected behavior print the recall, mrr and ndcg

Screenshots

Desktop (please complete the following information): Ubuntu 22.04

Additional context It worked fine with another test set. The only difference is that in this test set I removed items that the user already interacted in train.

[cadurosar]

Does your run contains all the elements on the qrel (and vice-versa)? I had this problem many times working with a dataset that had many queries that did not have relevance labels. It can also happen in the opposite sense (although rarer), if you have relevance data for a query that is not evaluated by your system it will also crash (reading your additional context it could be this case if you have the same qrel file for train and test)

[AmenRa]

Hi @EmotionFever and thank @cadurosar,

As @cadurosar told you, this is not a bug, it is a sanity check to let you know your qrels and run query ids to not match, thus you have results for queries without relevance judgements or queries without results, or both.

Maybe, you should just remove the queries you removed from qrels from the run too.

Please let me know if this solve your issue and, if you like it, consider giving ranx a star.

Thank you,

Elias

[AmenRa]

Closing for inactivity. Feel free to re-open.