Closed lukas-wegmeth closed 3 months ago
@lukas-wegmeth Hi! The longer valid/test time for sampling eval is normal. For full eval, we restore all the user and item embeddings to avoid repeat computations but not for negative sampling evaluation. You can check the source code for details. Note that predict
is used for negative sampling evaluations and full_sort_predict
is used for full eval.
@BishopLiu Thanks for replying. I have looked at the code and profiled the run time of the functions. I can see where negative sampling evaluation requires more time, but it is still unintuitive to me why it is. I believe negative sampling evaluation should be faster because fewer interactions must be predicted. Also, if restoring the embeddings is much quicker, why is it not done in negative sampling evaluation? Please let me know if I misunderstood anything about this.
@lukas-wegmeth Thank you for your attention to RecBole! The models in RecBole are implemented by different developers. Our first goal is to make sure the model is consistent with the original paper and runs correctly. And different developers have their own considerations. I'm sorry that I cannot answer why restoring embeddings is not done in negative sampling.
@BishopLiu I understand. Thanks for your response. Although my problem with high validation time persists, I can at least verify how this happens in the code now.
Describe the bug I was benchmarking many RecBole algorithms on a few data sets with "uni100" and "uni50" validation modes and noticed that validation took an unexpectedly long time. Therefore, I have tested multiple combinations of settings to figure out if this only happens rarely, but I can reproduce it consistently. I provide the results of my tests below. I wonder if this is a bug because, in my understanding, validation mode "unixxx" should be faster than "full". And I would expect "full" to take much longer than it does. Please check if the data provided in the tables looks like you would expect, and if so, help me understand why "unixxx" takes so long compared to "full".
To Reproduce Steps to reproduce the behavior:
Expected behavior Validation modes "unixxx" should be faster than "full".
Desktop (please complete the following information):