search

QingyaoAi / Unbiased-Learning-to-Rank-with-Unbiased-Propensity-Estimation

This is an implementation of the Dual Learning Algorithm with multi-layer feed-forward neural network for online unbiased learning to rank.
Apache License 2.0
89 stars 22 forks source link

How to train DLA with real click data log? #1

Open Ekkalak-T opened 5 years ago

Ekkalak-T commented 5 years ago

The provided DLA training steps are required to generate a click log based on human relevance annotations. However, In our case, we have only click data log and want to train DLA.

Can we do both training a ranker and reducing the bias at the same time?

QingyaoAi commented 5 years ago

We are currently working on a new version of the code that provides the API for training with external click data. So far, the easiest way to train DLA with external click logs is to replace the synthetic click data (line 379-385 in /Unbiased_LTR/DLA/DLA_model.py) with the true click data.

DLA naturally supports the training of a ranker and a propensity model (bias model) at the same time. However, please note that we still need unbiased relevance annotations for the final evaluation.

Ekkalak-T commented 5 years ago

@QingyaoAi Thank you so much for the answer. In fact, we have total 500 queries of unbiased relevance annotations which we use for evaluating relevance prediction from click log.

Currently, we are experimenting with the DBN model and found that there is around 50% agreement in pairwise preferences between click and editorial judements.

Could you please explain the detailed steps of cleaning click data process when you experimented with the DBN model as you wrote below? 

To insure the quality of relevance estimation with click models, we removed ranked lists with less than 10 click sessions in the cleaning process described previously

(from your paper) Thank you :)