search

PredictionIO / template-scala-parallel-universal-recommendation

PredictiionIO Template for Universal Recommender
112 stars 48 forks source link

no evaluation in the engine template #9

Open gearmonkey opened 9 years ago

gearmonkey commented 9 years ago

in contrast to the vanilla CF recommender engine template, this engine template lacks any evaluation. This makes it far more labor intensive to tune. Perhaps the other recommender engine's eval (https://github.com/PredictionIO/template-scala-parallel-recommendation/blob/develop/src/main/scala/Evaluation.scala) is a sensible place to start, though my scala skill is nearly non-existant, so not exactly sure what's going on in their eval code.

pferrel commented 9 years ago

The engine is pretty much self-tuning so far. Would like to add some diagnostic output that analyzes the input data to indicate things like which actions are not helping much.

pferrel commented 9 years ago

There are three tuning params and if you use a precision metric it will always favor making them larger. For instance "maxQueryEvents". Evaluation always favors more but this may loose recency of user intent, which in the real world may be more important. So for all but one issue tuning is left to A/B testing with real users.

The one remaining issue is which user data to use in training, we'll address this with a MAP eval per event. This will give an indication of ranking of event importance, and some may not be worth the performance or data gathering effort.