Open JustGitting opened 4 years ago
While I did not try to reproduce your bug, I have run into problems using xgboost and caret. Instead I wrote my own wrappers around XGBoost and avoid caret altogether.
Thank you @davidhaan
Do you mind sharing your wrapper or any tips?
@JustGitting I've just run into what I think is the same issue. I'm working on a credit risk model (similarly imbalanced at about 10:1) and found the cross-validated AUROC provided by caret to be substantially worse than the test set performance (on which I calculate AUROC myself and know the calculation to be reliable).
I suppose the best way for me to test if it's the same problem is to fit the XGB model again w/ the tuned parameters on the response w/ the levels reversed?
Update: I did the test as described and sure enough once reversing the levels of the response, the ROC is now in line w/ what I'm seeing in the test set. Great news!
I am working on a tricky classification problem that has unbalanced classes (10 to 1 ratio), with around 200,000 instances and approximately 10 features. I managed to get reasonable performance using XGBoost's xgb.train(), but wanted to do a systematic parameter grid search with CV. So I opted to use Caret to do the heavy lifting.
However, I was getting very poor results using the same data and even selecting the exact same parameters. I was stumped for a few days as I could not see why there was a major discrepancy.
I realised it was to do with how the original numerical target values are converted to factors for Caret.
xgb.train()
accepts a numerical y via thexgb.DMatrix()
function and can be trained for regression or classification by setting the appropriateobjective
parameter. Whilstcaret::train()
trains the method for classification by acceptingy
as a factor.The problem arises due to the ordering of the y factor levels, e.g. levels 'pos', 'neg' VS levels 'neg', 'pos'
Below is an example using the
yeast4
data set from theimbalance
package. As there needs to be sufficient number of instances and large enough class imbalance for the problem to become apparent.Refactor the Class and add a Class column that has the levels reversed.
Create the training and test sets.
After using a grid search I found the following parameters for XGBoost to produce good results. YMMV. Also specify Caret training control.
First we'll train using y with Factor w/ 2 levels "positive","negative" ordering.
Next we'll train using y with the level order reversed, i.e. Factor w/ 2 levels "negative","positive".
Below is the performance comparison between the two runs.
It is clear that the Sensitivity and Specificity values have reversed compared to the original caret trained XGboost model. As well as the SD is zero, ie no variation in performance over the 10 folds.
Next we can plot the Sensitivity vs Specificity over a range of decision thresholds to further illustrate the problem.
The plot shows the original Caret trained XGBoost model with Sens/Spec forming the usual crossover pattern, see for example. Whilst the Sens/Spec plot for the
caret_train_revLvls
model has a hard crossover, which is not expected.I guess this problem occurs because CARET is converting the y factors to numerical values assuming the first level is the positive class. However, it only becomes apparent on unbalanced data sets.
It would be great if someone could verify this behaviour, or I'm imagining it.
A simple work around is to ensure the first level of the
y
factor is the positive class. As illustrated in the above example.