xgboost ranking objective functions have strange behaviors

[jfrery]

In xgboost you have the possibility of chosing a ranking objective function. If you do so, as I understood, the algorithm becomes lambdaMART in a way.

You have 3 different ranking function:

"rank:pairswise"
"rank:map"
"rank:ndcg"

It seems that "rank:pairwise" does a pretty good job. However the two others have bad performances (very bad tbh). I am wondering if it comes from the implementation (I've seen that it's a beta implementation of these objective functions) or if we should build our model differently ?

Another question, those objective functions should be quadratic (we need to swap each example with every single example in the training set in order to compute the gradients), however it seems that the learning time is the same as "binary:logistic" for example which is not quadratic. What am I missing there ?

Thanks ! :)

[khotilov]

Only “rank:pairwise” is a ranking objective. "map" and "ndcg" are performance metrics for ranking.

Ranking objective performance is quadratic within each group, not the whole training set.

[jfrery]

Please correct me if I am wrong but "rank:map" and "rank:ndcg" are objective functions. map and ndcg are performance metrics, yes, but during the training process you can optimise those metrics as it is done in LambdaMART (by multiplying each pair loss by a coefficient which is the difference in map or in ndcg when you swap the examples in the rank). If I understood well, this is what they tried to do here in xgboost. Am I wrong ?

[Laurae2]

@jf10 rank:pairwise is the objective function for gradient descent optimization (so that xgboost understands what to minimize), while rank:map and rank:ncdg are the metrics you monitor for the performance (they have zero impact on the model).

[jfrery]

@Laurae2 In .../src/objective/rank_obj.cc, it computes different objective function regarding your objective parameter (rank:map, rank:pairwise, rank:ndcg). If this is only for monitoring then I don't understand why you have an evaluation metric parameter.

Plus, it does have a strong impact on the model.

There are some typos in this file too. (i.e: PairwieRankObj).

Could someone explain how those objectives function works because I am lost now ? :/

[Laurae2]

@jf10 They (map, ncdg) are currently in testing and you should not expect any good performance from them.

If you want to understand how they work, all you need is to read the code (you don't need C++ knowledge to understand how they work, most functions are obvious in their meanings, like Sigmoid, CalcDCG, etc.).

What's important is:

• The objective function is the gradient of the loss function
• The loss function may not be derivable and thus require a proxy function (ex: MAE, ranking problems)

Therefore, there's a lot more going on which is not in the code itself. You may try getting and plotting the gradients to check what is happening exactly, because they guide xgboost / gradient descent on how to reduce the error. This will also help you find out exactly why they behave poorly.

[jfrery]

@Laurae2 Thank you for you answers. I will try what you suggest.