How the loss manages labels from click models

[pommedeterresautee]

In the examples provided with the code the labels are 1 or 0. In the 3 papers (KRNN, CNN, entity duet), you are training and testing on logs where the relevance is computed by a click model (meaning label are not anymore 1 or 0 but some floats between 0 and 1). However in the 3 papers it is said that the loss is the standard pairwise learning (and the code matches that point).

I am wondering how you take into account the relevance labels (when they are floats between 0 and 1)?

In KRNN paper it is said that the relevance scores are mapped to relevance grades. Does it mean that you generate pairs like : (grade 4 as positive doc Vs grade 3 negative doc) ?

[EdwardZH]

The detail experimental setting you can find in the K-NRM paper. In our experiments, the relevance labels are inferred by user clicks. For example, for a query q, document 1 is clicked 10 times and document 2 id clicked 2 times. As CTR model (a kind of click model) you can get their relevance label. And for pairwise loss document 1 is the positive one and document 2 is the negative one.