New kpi for newsreel evaluation

[davidemalagoli]

We agreed to integrate new kpi in the newsreel evaluator, could you please verify how to implement:

• response time
• throughput tot
• coverage
• ctr

If I'm not wrong only throughput tot and coverage are missing, and the latter should have the information of all the items. How can we implement that?

[davidemalagoli]

Perhaps the simplest way to implement the coverage is to calculate the number of distinct items returned

[andreas-dai]

In the NewsREEL challenge the coverage metric does neither improve the perceived quality of the recommendations nor it correlates with business-oriented metrics. It is very easy to overfit the coverage metric, suggesting every newly created item once. In the online evaluation the coverage could encourage users reading a broader spectrum of news; but in the offline evaluation (characterized by a pre-recorded ground truth) the meaning of coverage is not clear to me.

[davidemalagoli]

@andreas-dai what about this comment? can we confirm that we don't need this kpi?

[andreas-dai]

The newsreel evaluator computes CTR, throughput, and response time. Coverage is not relevant in the NewsREEL scenario. I think, we can close the issue.