search

tensorflow / recommenders

TensorFlow Recommenders is a library for building recommender system models using TensorFlow.
Apache License 2.0
1.83k stars 274 forks source link

how to evaluate a model? #249

Open zhifeng-huang opened 3 years ago

zhifeng-huang commented 3 years ago

I have followed the tutorial to implement a model for my own dataset. However, I have no idea how good the performance it is.

I am turning hyperparameter.

What does it mean if I get the following result? Top-100 accuracy (train): 0.28. Top-100 accuracy (test): 0.09.

What does it mean if I get the following result? Top-100 accuracy (train): 0.22. Top-100 accuracy (test): 0.22.

I trained the model so the loss did not go down anymore. I still could not bring up the accuracy. Any suggestion what I can do?

ageofneil commented 3 years ago

Your first result indicate the model is overfitting. This means it wont generalize well. It's memorizing the training data. Your second model has a lower accuracy than the first, but it generalizes well to new data.

I'de recommend reading "Examples of Bias and Variance" by Andrew Ng for a basic overview of what these numbers mean.

dgoldenberg-audiomack commented 3 years ago

I think the doc would benefit from a specific how-to on how to tell if it's overfitting and what to try to combat the issue.

almirb commented 2 years ago

I think the doc would benefit from a specific how-to on how to tell if it's overfitting and what to try to combat the issue.

I think this would clarify what could be done a little: https://www.tensorflow.org/tutorials/keras/overfit_and_underfit

dgoldenberg-audiomack commented 2 years ago

@almirb Yeah, nice but won't this take months or years to get all done and right? I think there are mlops frameworks out there to automate this kind of thing, otherwise it's an endless trial-and-error loop...

almirb commented 2 years ago

@almirb Yeah, nice but won't this take months or years to get all done and right? I think there are mlops frameworks out there to automate this kind of thing, otherwise it's an endless trial-and-error loop...

Yes.. I am also getting a hard time trying to get all those things to work and create a production grade recsys. TFRS is a great framework! But I'm new to keras and tensorflow. A lot of things to learn...