the classProbability of XGBoost is different between python and coreml model

[YuchengChien]

❓Question

I had trained a XGBoost model 'bst' and used coremltools to get a CoreML model. And I used the same input but the output is different. How could I do to get the same results ? The code is below:

# get coreml model
import coremltools
coremlmodel = coremltools.converters.xgboost.convert(bst, force_32bit_float=False, mode='classifier', n_classes=7)
# predict
bst_result = bst.predict(xgb.DMatrix(X_test.iloc[0:1].values))
dictinput = {'f{}'.format(x): X_test.iloc[0, x] for x in range(X_test.shape[1])}
coreml_result = coremlmodel.predict(dictinput)
# print result
print('bst result: ', bst_result)
print('coreml result', coreml_result)

The result is below:

bst result:  [[0.28688842 0.00184444 0.00135211 0.70432913 0.00201192 0.00176237
  0.00181162]]

coreml result {'target': 3, 'classProbability': {0: 1.325793311052, 5: -3.7666387138099995, 1: -3.7211242350999996, 6: -3.7390789153600004, 2: -4.0316371452, 3: 2.223946214053901, 4: -3.634210373570001}}

The target of coreml result is right, but the classProbability is not in 0~1. How could I do to make the two results is the same? Thanks!

System Information

Mac OS: 10.15.3 python version: 3.7.6 XGBoost version: 0.9 coremltools version: 3.4

[anilkatti]

@YuchengChien could you provide the model so, we can take a closer look?

[YuchengChien]

model.zip

@anilkatti Thank you for your prompt reply. Attached is my XGBoost model, and the code for testing is below:

import coremltools
import xgboost as xgb

# load xgboost model
bst = xgb.Booster({'nthread': 4})  # init model
bst.load_model('model/bst.bin')  # load data
feature_names = ['f{}'.format(x) for x in range(10)]

# get coreml model
coremlmodel = coremltools.converters.xgboost.convert(
    bst,
    feature_names=feature_names,
    force_32bit_float=False, 
    mode='classifier', 
    n_classes=7)

# predict
testInput = [x for x in range(10)]
bst_result = bst.predict(xgb.DMatrix(testInput))
dictinput = {'f{}'.format(x): testInput[x] for x in range(len(testInput))}
coreml_result = coremlmodel.predict(dictinput)

# print result
print('bst result: ', bst_result)
print('coreml result', coreml_result)

The result is below:

bst result:  [[0.01892603 0.00355117 0.1308418  0.06450443 0.09977078 0.6677399
  0.01466583]]
coreml result {'target': 5, 'classProbability': {0: -2.1363359517809997, 5: 1.4270251751000003, 1: -3.8095957623199994, 6: -2.3913538194600017, 2: -0.20288498279900002, 3: -0.9101402674720002, 4: -0.47399873714899976}}

The target of coreml result is right, but the classProbability is not in range 0..1

[anilkatti]

Thanks. classProbability is definitely not correct. We will investigate this and get back to you.

[anilkatti]

@YuchengChien this is in fact a bug in the the way Core ML is computing class probabilities for GBDTs. We have a fix coming in one of the next OS releases. However, we verified that the predicted class should be accurate and match with what XGBoost provides in multiple cases.

[YuchengChien]

@anilkatti Thank you for fixing the bug. I'm looking forward to the next release.

[wlwg]

I wonder if this is related to #605