[Question] TFX Evaluator in a recommender pipeline

[TaylorZowtuk]

I apologize if an issue isn't the appropriate place for my question - but if what I am trying to do here is sensible/possible then maybe this could be considered a documentation improvement request.

In the recommender retrieval + TFX tutorial, we go from training the model to exporting it. In the Keras + TFX tutorial, they show how to use the Evaluator component. I am trying to add the Evaluator component to the retrieval pipeline but I am having trouble figuring out how to instantiate the EvalConfig (specifically, I am having trouble with the metrics).

I will start with my general questions, and if it would be helpful I can post what I have already tried.

1. Is using the Evaluator component in the retrieval pipeline a sensible thing to do, or is there a reason it wasn't included in the example? By sensible here, I mean is there anything preventing us from comparing two retrieval models?
2. If it is sensible to use the Evaluator component, could someone please share a code snippet (eval_config, model_resolver, evaluator) with me for what it would look like in the retrieval pipeline example for the case where we want to compare against the latest blessed model?
3. What would be an appropriate metric for comparing two retrieval models?
   • I tried the FactorizedTopK metric, but I couldn't figure out:
     • how to pass the candidates in the tfma.MetricConfig
     • how to define thresholds for a metric that returns a collection of scores (for each k)
     • whether it was fair to use a metric to compare a candidate and blessed model which used the product_model that was trained alongside the candidate user_model
   • from here, it seems like NDCG or MinLabelPosition could be options although I don't really understand either of them (a simple explanation would be greatly appreciated) and MinLabelPosition in particular seems to only be applicable for ranking?
     • for these, I again don't know how to specify thresholds for them when the metrics_specs are created using tfma.metrics.specs_from_metrics(...)

[j-wine]

Hi, I am having a similar problem. Did you manage to progress? I am sorry to say I can't help you with it.

But I'm also having trouble defining the correct metrics for a recommender, which is basically this one: https://www.tensorflow.org/tfx/tutorials/tfx/recommenders#the_dataset

Any metric added produces a runtimerror, even those added ad training time like this model.compile(optimizer=tf.keras.optimizers.Adagrad(learning_rate=0.1) , metrics=[tf.keras.metrics.AUC, tf.keras.metrics.Precision, tf.keras.metrics.Recall, tf.keras.metrics.mean_absolute_error]

[TaylorZowtuk]

Unfortunately I wasn't able to progress further on this and my questions remain.