search

acbull / Unbiased_LambdaMart

Code for WWW'19 "Unbiased LambdaMART: An Unbiased Pairwise Learning-to-Rank Algorithm", which is based on LightGBM
MIT License
224 stars 50 forks source link

Unbiased_lambdamart ndcg is low than original lambdamart #3

Closed ubuntuone closed 5 years ago

ubuntuone commented 5 years ago

I do the follow step:

  1. download the generate_dataset
  2. python evaluation/scripts/generate_data.py evaluation/click_model/user_browsing_model_0.1_1_4_1.json to generate train and test data
  3. run ./lightgbm config="train.conf" and get test data ndcg@10=0.546817
  4. build a original lightgbm, version=2.1.1 that same to Unbiased_LambdaMart
  5. run ./lightgbm config="train.conf" and get test data ndcg@10=0.556632
  6. Why Unbiased_lambdamart ndcg is low than original lambdamart? The paper say Unbiased_LambdaMart is better than origial?
acbull commented 5 years ago

Sorry for the incomplete README and some bugs in the config file before...

(1) The original training directory is mistaken. the train_data should be "train_data = ../test_data/letor.user_browsing_model_0_train". (I've already fixed it). (2) The data in generate_data are synthetic click data, which is not that meaningful for evaluation. What we really interested in is the result for real data. Therefore, after we get the model output result, we need to:

cd ../scripts
python eval.py ../configs/LightGBM_predict_result.txt  #or any other model output.

to do evaluation using trec (I've also added in the README now).

Also by the way, the experiment of the big table in my paper is the one not using regularization (L-0). The default regularization term in this Github is 0.5 (L-0.5), which will lead to a little bit lower result, but still much better than the original lightgbm without debiasing (which can be shown in figure 3.b).

Here is my evaluation output for your reference, the NDCG@10 with L-0.5 regularizer is 0.7535, while the original lightgbm without debiasing in my previous experiment is 0.716:

image

And if you want to re-implement the result of our table, you can change the regularization term back to 0, by modifying 

double _eta = 1.0 / (1 + 0.5);

in the src/objective/rank_objective.hpp (Line 418) to 

double _eta = 1.0 / (1 + 0);

Sorry for the mistakes... And hope it solves your problems.

ubuntuone commented 5 years ago

Thanks a lot. I have use eval.py to evaluate test file and get original lightgbm is lower than Unbiased_LambdaMart on NDCG@10.

ubuntuone commented 5 years ago

Yes, I have checked trec's document and seen ndcg cal is diff. Thks!