xgboost linear model doesn't support set_weight()

[19930323]

XGBoost linear model doesn't support set_weight() function the settings of my model is as follow

        "eval_metric": ["ndcg@5", "ndcg@10"],
        "booster": "gblinear",
        "updater": "shotgun",
        "feature_selector": "cyclic"}

param_grid = ParameterGrid(
    {"learning_rate": [1, 0.3, 0.25, 0.2, 0.15, 0.1, 0.05, 0.01, 0.005, 0.001],
      "alpha": [0, 1e-4, 1e-3, 1e-2, 0.1, 1], 
     "lambda": [0, 1e-4, 1e-3, 1e-2, 0.1, 1]
    })

and I tried to set weight for each instance using dmatrix.set_weight(weights) weights is a array contains the weight for each data point since it's a listwise loss function that optimizes NDCG, I also use the function set_group()

after running the model, all the metrics are the same so we can't pick up a optimized hyper-parameter, and when we remove set_weight, the model can run as desired.

with gbtree, the function works fine, but with gblinear it's not. I also tried to set the weight to 1.0 for all instance, but it still doesn't work well.

valid_dmatrix = DMatrix(x_valid, y_valid)
test_dmatrix = DMatrix(x_test)

train_dmatrix.set_group(group_train)
valid_dmatrix.set_group(group_valid)

weight_vector_train = [4 for ii in range(len(group_train))]
weight_vector_valid = [4 for ii in range(len(group_valid))]

train_dmatrix.set_weight(weight_vector_train)
valid_dmatrix.set_weight(weight_vector_valid)

params = {"objective": "rank:ndcg",
        "booster": "gblinear",
        "eval_metric": ["ndcg@5", "ndcg@10"]}`

param_grid = ParameterGrid(
    { "alpha": [0, 1e-3, 1e-2, 0.1, 1], 
     "lambda": [0, 1e-3, 1e-2, 0.1, 1]
    }
)

for exp_index, exp_params in enumerate(param_grid):
    print('------------------------------')
    print('Running experiment #{}'.format(exp_index + 1))
    params.update(exp_params)
    print('Params are {0}'.format(params))
    evals_result = {}
    model = xgb.train(params, 
                      train_dmatrix, 
                      num_boost_round=50, 
                      evals=[(valid_dmatrix, 'validation')],
                      evals_result=evals_result,
                      early_stopping_rounds=20,
                      verbose_eval=False)

    for metric in params['eval_metric']:
        print('{} on validation data: {}'.format(metric, evals_result['validation'][metric][-1]))

Running experiment #1 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0, 'lambda': 0} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #2 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0, 'lambda': 0.001} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #3 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0, 'lambda': 0.01} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #4 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0, 'lambda': 0.1} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #5 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0, 'lambda': 1} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #6 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0.001, 'lambda': 0} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #7 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0.001, 'lambda': 0.001} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #8 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0.001, 'lambda': 0.01} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #9 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0.001, 'lambda': 0.1} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #10 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0.001, 'lambda': 1} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #11 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0.01, 'lambda': 0} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #12 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0.01, 'lambda': 0.001} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #13 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0.01, 'lambda': 0.01} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #14 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0.01, 'lambda': 0.1} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #15 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0.01, 'lambda': 1} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #16 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0.1, 'lambda': 0} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #17 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0.1, 'lambda': 0.001} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #18 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0.1, 'lambda': 0.01} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #19 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0.1, 'lambda': 0.1} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #20 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 0.1, 'lambda': 1} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #21 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 1, 'lambda': 0} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #22 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 1, 'lambda': 0.001} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #23 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 1, 'lambda': 0.01} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #24 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 1, 'lambda': 0.1} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

Running experiment #25 Params are {'objective': 'rank:ndcg', 'booster': 'gblinear', 'eval_metric': ['ndcg@5', 'ndcg@10'], 'alpha': 1, 'lambda': 1} ndcg@5 on validation data: 0.494262 ndcg@10 on validation data: 0.585927

[trivialfis]

The weight is used in the objective. It's might just be the linear model not converging well, I can't be sure at the moment.

[trivialfis]

Please try to remove the regularization term and try again. Also in general tree models might be easier to tune.