Open mnhqut opened 2 weeks ago
I forgot to mention the way I spitted training and testing data was:
train, test = python_stratified_split(df, ratio=0.75)
We need to review this @SimonYansenZhao @anargyri @daviddavo @loomlike and even @yueguoguo
How do you get the all_predictions
? Can you provide the full code to reproduce the issue? Or are you just using the deep dive notebook?
The problem seems to be that when you use by_threshold, the k
at some equation terms remains being top_k
For example, in precision@k:
It divides by 10 (default k
value), instead of by 50 (the specified by_threshold value).
The other metrics have similar problems.
Perhaps this is what by_threshold
is intended to do. Is it a way of changing how many items you want, even though you are calculating by k
?? I don't really understand how by_threshold should work so I don't really know if this is a bug or intended behaviour.
I can solve the bug by just using threshold
instead of k
when necessary, but then by_threshold
and top_k
would be exactly the same.
Btw, here is a notebook that replicates the issue in Google Colab
Description
Hello!
I encountered an issue while evaluating the BPR (Bayesian Personalized Ranking) model with basically the same code provided in the example on a different dataset. Specifically, when using the "by_threshold" relevancy method with ranking metrics, the computed values for precision@k, ndcg@k, and map@k exceed 1, which seems incorrect. This issue does not occur when switching the relevancy method to "top_k."
How do we replicate the issue?
I use the following parameter for BPR (all using the default seed):
Using these evaluation
Here is the dataset I test on: https://github.com/mnhqut/rec_sys-dataset/blob/main/data.csv
My result: MAP: 1.417529 NDCG: 1.359902 Precision@K: 2.256466
Willingness to contribute