search

mpkato / interleaving

A python library for conducting interleaving, which compares two or multiple rankers based on observed user clicks by interleaving their results.
MIT License
121 stars 13 forks source link

Is Probablistic.evaluate different from that in the original paper? #26

Closed mpkato closed 7 years ago

mpkato commented 7 years ago

Sections 4.2 and 4.3 in "A Probabilistic Method for Inferring Preferences from Clicks, CIKM2011" explain how to infer the preference based on user clicks.

They seems different from the current implementation.

tmanabe commented 7 years ago

Yes. My implementation of evaluation() just indicates which ranker won, i.e., an array of binaries. To indicate outcomes (-1, 0, or 1) between rankers and to calculate expected values of outcome, we need some other functions.

mpkato commented 7 years ago

Can I ask you to address this issue? Please use your own implementation if necessary.

tmanabe commented 7 years ago

Are you in a hurry? If not, I can do it.

mpkato commented 7 years ago

Not at all

tmanabe commented 7 years ago

OK, let me try.

mpkato commented 7 years ago

Thanks!