XGBoost eval_metric and prediction results are different

[aParsecFromFuture]

def acc(y_true, y_pred):
        return accuracy_score(y_true, y_pred.round())

  params = {
      'objective': 'binary:logistic',
      'enable_categorical': True,
      'tree_method': 'hist',
      'eval_metric': acc
  }

  model = xgb.XGBClassifier(**params)

  model.fit(X_train, y_train, 
            eval_set=[(X_valid, y_valid)])

  y_pred = model.predict(X_valid)
  score = accuracy_score(y_valid, y_pred)
  print(score)

Accuracy score should be same score of the last iteration but I get 73% and 69%. Validation data has 10k rows. Any idea?

[trivialfis]

I can't reproduce using the latest xgboost. Could you please share a reproducible example?

[aParsecFromFuture]

I can't reproduce using the latest xgboost. Could you please share a reproducible example?

It's from a Kaggle competition. I can't make the notebook public. Do you have any Kaggle account that I can share the notebook with?

I have used the xgboost library a lot, I only face such a problem with this dataset.

[aParsecFromFuture]

I get the same behavior with built-in error metric. (1 - error) is not equal to accuracy score.

Class distribution: (0: 32496, 1: 20523)

[trivialfis]

Let me get back to this tomorrow. I have an account with the same email address on my github profile.

[aParsecFromFuture]

Let me get back to this tomorrow. I have an account with the same email address on my github profile.

I shared the notebook with trivialfis user. Good luck.

[trivialfis]

This is a limited-participation competition. Only invited users may participate.

[aParsecFromFuture]

This is a limited-participation competition. Only invited users may participate.

I'm going to open it publicly for today. Let me know when your review is done. Link: https://www.kaggle.com/greysky/temporary-notebook-it-will-be-deleted

[trivialfis]

thank you for sharing, I can see the notebook, it's the dataset that I can't access. I need to participate in the competition before seeing the dataset, but the competition is invite only.

[aParsecFromFuture]

thank you for sharing, I can see the notebook, it's the dataset that I can't access. I need to participate in the competition before seeing the dataset, but the competition is invite only.

I have created the same notebook with copy-dataset and invited you. Could you try again?

[trivialfis]

Hi, I think it's possible that this is caused by the verbose=20 in the fit method, the latest evaluation result is not displayed as the model print only at each 20 iterations.

You can obtain the list of evaluation results by clf.evals_result().

[trivialfis]

NVM, later iteration is changed, still looking.

[aParsecFromFuture]

Hi, I think it's possible that this is caused by the verbose=20 in the fit method, the latest evaluation result is not displayed as the model print only at each 20 iterations.

You can obtain the list of evaluation results by clf.evals_result().

There is no iteration with output accuracy. I noticed early that even if we do hyper parameter optimization (like Optuna), the output accuracy remains 0.61 as eval_metric output approaches 0.70. I was thinking that this happens because the LogLoss value is constantly increasing. But "disable_default_eval_metric" option didn't prevent that strange behavior.

[trivialfis]

Thank you for raising the issue and for the helpful assistance! I have reproduced the error using the dataset from the competition.