search

dmlc / xgboost

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow
https://xgboost.readthedocs.io/en/stable/
Apache License 2.0
26.13k stars 8.7k forks source link

xgboost/src/tree/../common/row_set.h:56: Check failed: e.begin != nullptr access element that is not in the set #2800

Closed pseudotensor closed 6 years ago

pseudotensor commented 6 years ago

Any ideas? @khotilov @hcho3 ?

This is using current head of master.

X = [[ 1.6,  0.2 , 4.8],[ 5.1,  1.6  ,6. ], [ 5.8,  2.2,  6.5], [ 1.5,  0.1 , 5.2]]
y = [ 3.4,  2.7 , 3. ,  4.1]
import numpy as np
nan = np.nan
params = {'base_score': 0.5, 'booster': 'gbtree', 'colsample_bylevel': 1, 'colsample_bytree': 0.8, 'gamma': 0, 'learning_rate': 0.02, 'max_delta_step': 0, 'max_depth': 4, 'min_child_weight': 10, 'missing': nan, 'n_estimators': 100, 'n_jobs': 1, 'nthread': None, 'objective': 'reg:linear', 'random_state': 11675, 'reg_alpha': 0, 'reg_lambda': 1, 'scale_pos_weight': 1, 'seed': None, 'silent': 0, 'subsample': 0.7, 'eval_metric': 'rmse', 'debug_verbose': 2, 'tree_method': 'hist'}
kwargs = params
import xgboost as xgb
model = xgb.XGBRegressor(**kwargs)
model.fit(X,y)

If I change sample choices to 1.0, then no failure.

Or if I choose gpu_hist instead, then no failure.

Otherwise, this gives the failure mode of below. So, hist logic in the sampling is flawed somehow.

[13:21:18] Tree method is selected to be 'hist', which uses a single updater grow_fast_histmaker.
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:73: Generating gmat: 0.000110865 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 (23.39%)
InitNewNode:       0.0000 ( 4.68%)
BuildHist:         0.0000 (17.54%)
EvaluateSplit:     0.0000 (10.53%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 8.05%)
InitNewNode:       0.0000 ( 1.34%)
BuildHist:         0.0000 ( 7.38%)
EvaluateSplit:     0.0000 ( 4.70%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 7.50%)
InitNewNode:       0.0000 ( 1.67%)
BuildHist:         0.0000 ( 5.83%)
EvaluateSplit:     0.0000 ( 5.00%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 5.34%)
InitNewNode:       0.0000 ( 0.76%)
BuildHist:         0.0000 ( 5.34%)
EvaluateSplit:     0.0000 ( 4.58%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 4.32%)
InitNewNode:       0.0000 ( 0.72%)
BuildHist:         0.0000 ( 5.76%)
EvaluateSplit:     0.0000 ( 4.32%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 4.51%)
InitNewNode:       0.0000 ( 0.75%)
BuildHist:         0.0000 ( 5.26%)
EvaluateSplit:     0.0000 ( 4.51%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 5.22%)
InitNewNode:       0.0000 ( 1.74%)
BuildHist:         0.0000 ( 6.09%)
EvaluateSplit:     0.0000 ( 4.35%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 5.22%)
InitNewNode:       0.0000 ( 1.74%)
BuildHist:         0.0000 ( 6.09%)
EvaluateSplit:     0.0000 ( 4.35%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 5.17%)
InitNewNode:       0.0000 ( 0.86%)
BuildHist:         0.0000 ( 6.03%)
EvaluateSplit:     0.0000 ( 4.31%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 5.88%)
InitNewNode:       0.0000 ( 1.68%)
BuildHist:         0.0000 ( 5.04%)
EvaluateSplit:     0.0000 ( 5.04%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 5.38%)
InitNewNode:       0.0000 ( 0.77%)
BuildHist:         0.0000 ( 5.38%)
EvaluateSplit:     0.0000 ( 4.62%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 6.20%)
InitNewNode:       0.0000 ( 0.78%)
BuildHist:         0.0000 ( 3.88%)
EvaluateSplit:     0.0000 ( 4.65%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 6.14%)
InitNewNode:       0.0000 ( 0.88%)
BuildHist:         0.0000 ( 5.26%)
EvaluateSplit:     0.0000 ( 5.26%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 6.50%)
InitNewNode:       0.0000 ( 0.81%)
BuildHist:         0.0000 ( 4.88%)
EvaluateSplit:     0.0000 ( 4.88%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 5.38%)
InitNewNode:       0.0000 ( 0.77%)
BuildHist:         0.0000 ( 4.62%)
EvaluateSplit:     0.0000 ( 4.62%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 5.34%)
InitNewNode:       0.0000 ( 0.76%)
BuildHist:         0.0000 ( 4.58%)
EvaluateSplit:     0.0000 ( 3.82%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 4.84%)
InitNewNode:       0.0000 ( 0.81%)
BuildHist:         0.0000 ( 5.65%)
EvaluateSplit:     0.0000 ( 4.84%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 5.04%)
InitNewNode:       0.0000 ( 0.84%)
BuildHist:         0.0000 ( 5.04%)
EvaluateSplit:     0.0000 ( 4.20%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 5.43%)
InitNewNode:       0.0000 ( 1.55%)
BuildHist:         0.0000 ( 5.43%)
EvaluateSplit:     0.0000 ( 3.88%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 5.60%)
InitNewNode:       0.0000 ( 1.60%)
BuildHist:         0.0000 ( 4.80%)
EvaluateSplit:     0.0000 ( 4.80%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 6.20%)
InitNewNode:       0.0000 ( 1.55%)
BuildHist:         0.0000 ( 4.65%)
EvaluateSplit:     0.0000 ( 4.65%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 5.26%)
InitNewNode:       0.0000 ( 1.75%)
BuildHist:         0.0000 ( 5.26%)
EvaluateSplit:     0.0000 ( 4.39%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 6.09%)
InitNewNode:       0.0000 ( 1.74%)
BuildHist:         0.0000 ( 5.22%)
EvaluateSplit:     0.0000 ( 4.35%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 5.26%)
InitNewNode:       0.0000 ( 1.50%)
BuildHist:         0.0000 ( 5.26%)
EvaluateSplit:     0.0000 ( 3.76%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 4.58%)
InitNewNode:       0.0000 ( 2.29%)
BuildHist:         0.0000 ( 4.58%)
EvaluateSplit:     0.0000 ( 3.82%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 6.09%)
InitNewNode:       0.0000 ( 1.74%)
BuildHist:         0.0000 ( 6.09%)
EvaluateSplit:     0.0000 ( 4.35%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 5.69%)
InitNewNode:       0.0000 ( 1.63%)
BuildHist:         0.0000 ( 4.88%)
EvaluateSplit:     0.0000 ( 4.88%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 5.30%)
InitNewNode:       0.0000 ( 0.76%)
BuildHist:         0.0000 ( 3.79%)
EvaluateSplit:     0.0000 ( 4.55%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 4.58%)
InitNewNode:       0.0000 ( 1.53%)
BuildHist:         0.0000 ( 5.34%)
EvaluateSplit:     0.0000 ( 3.82%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 5.00%)
InitNewNode:       0.0000 ( 1.43%)
BuildHist:         0.0000 ( 5.00%)
EvaluateSplit:     0.0000 ( 3.57%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 6.36%)
InitNewNode:       0.0000 ( 0.91%)
BuildHist:         0.0000 ( 4.55%)
EvaluateSplit:     0.0000 ( 5.45%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 4.92%)
InitNewNode:       0.0000 ( 0.82%)
BuildHist:         0.0000 ( 5.74%)
EvaluateSplit:     0.0000 ( 4.92%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 0 extra nodes, 0 pruned nodes, max_depth=0
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/updater_fast_hist.cc:252: 
InitData:          0.0000 ( 6.42%)
InitNewNode:       0.0000 ( 1.83%)
BuildHist:         0.0000 ( 6.42%)
EvaluateSplit:     0.0000 ( 4.59%)
ApplySplit:        0.0000 ( 0.00%)
========================================
Total:             0.0000
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/gbm/gbtree.cc:210: CommitModel(): 0.0000 sec
[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/dmlc-core/include/dmlc/././logging.h:300: [13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/../common/row_set.h:56: Check failed: e.begin != nullptr access element that is not in the set

Stack trace returned 10 entries:
[bt] (0) /home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/libxgboost.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7ff9f27f5eac]
[bt] (1) /home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/libxgboost.so(_ZN7xgboost4tree13FastHistMakerINS0_9GradStatsENS0_12NoConstraintEE7Builder6UpdateERKNS_6common16GHistIndexMatrixERKNS6_21GHistIndexBlockMatrixERKNS6_12ColumnMatrixERKSt6vectorINS_6detail18bst_gpair_internalIfEESaISJ_EEPNS_7DMatrixEPNS_7RegTreeE+0x102a) [0x7ff9f295bb1a]
[bt] (2) /home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/libxgboost.so(_ZN7xgboost4tree13FastHistMakerINS0_9GradStatsENS0_12NoConstraintEE6UpdateERKSt6vectorINS_6detail18bst_gpair_internalIfEESaIS8_EEPNS_7DMatrixERKS5_IPNS_7RegTreeESaISG_EE+0x213) [0x7ff9f294eb43]
[bt] (3) /home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/libxgboost.so(_ZN7xgboost3gbm6GBTree13BoostNewTreesERKSt6vectorINS_6detail18bst_gpair_internalIfEESaIS5_EEPNS_7DMatrixEiPS2_ISt10unique_ptrINS_7RegTreeESt14default_deleteISD_EESaISG_EE+0x9ce) [0x7ff9f289bd8e]
[bt] (4) /home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/libxgboost.so(_ZN7xgboost3gbm6GBTree7DoBoostEPNS_7DMatrixEPSt6vectorINS_6detail18bst_gpair_internalIfEESaIS7_EEPNS_11ObjFunctionE+0xb50) [0x7ff9f289d1b0]
[bt] (5) /home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/libxgboost.so(_ZN7xgboost11LearnerImpl13UpdateOneIterEiPNS_7DMatrixE+0x22b) [0x7ff9f28aa0db]
[bt] (6) /home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/libxgboost.so(XGBoosterUpdateOneIter+0x27) [0x7ff9f27e7de7]
[bt] (7) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7ff9b74f9e40]
[bt] (8) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x2eb) [0x7ff9b74f98ab]
[bt] (9) /home/jon/.pyenv/versions/3.6.1/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2cf) [0x7ff9b773bc4f]

Traceback (most recent call last):
  File "xgb.crash.py", line 34, in <module>
    model.fit(X,y)
  File "/home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/sklearn.py", line 294, in fit
    verbose_eval=verbose, xgb_model=xgb_model)
  File "/home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/training.py", line 204, in train
    xgb_model=xgb_model, callbacks=callbacks)
  File "/home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/training.py", line 74, in _train_internal
    bst.update(dtrain, i, obj)
  File "/home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/core.py", line 1003, in update
    dtrain.handle))
  File "/home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/core.py", line 130, in _check_call
    raise XGBoostError(_LIB.XGBGetLastError())
xgboost.core.XGBoostError: b'[13:21:18] /home/jenkins/slave_dir_from_mr-0xc1/workspace/h2o4gpu_master-C5X2X3XWTDU4EUPD74QCYXN4IG4GLHK5GBESSDCFPPEID4CNXQYA/xgboost/src/tree/../common/row_set.h:56: Check failed: e.begin != nullptr access element that is not in the set\n\nStack trace returned 10 entries:\n[bt] (0) /home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/libxgboost.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7ff9f27f5eac]\n[bt] (1) /home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/libxgboost.so(_ZN7xgboost4tree13FastHistMakerINS0_9GradStatsENS0_12NoConstraintEE7Builder6UpdateERKNS_6common16GHistIndexMatrixERKNS6_21GHistIndexBlockMatrixERKNS6_12ColumnMatrixERKSt6vectorINS_6detail18bst_gpair_internalIfEESaISJ_EEPNS_7DMatrixEPNS_7RegTreeE+0x102a) [0x7ff9f295bb1a]\n[bt] (2) /home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/libxgboost.so(_ZN7xgboost4tree13FastHistMakerINS0_9GradStatsENS0_12NoConstraintEE6UpdateERKSt6vectorINS_6detail18bst_gpair_internalIfEESaIS8_EEPNS_7DMatrixERKS5_IPNS_7RegTreeESaISG_EE+0x213) [0x7ff9f294eb43]\n[bt] (3) /home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/libxgboost.so(_ZN7xgboost3gbm6GBTree13BoostNewTreesERKSt6vectorINS_6detail18bst_gpair_internalIfEESaIS5_EEPNS_7DMatrixEiPS2_ISt10unique_ptrINS_7RegTreeESt14default_deleteISD_EESaISG_EE+0x9ce) [0x7ff9f289bd8e]\n[bt] (4) /home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/libxgboost.so(_ZN7xgboost3gbm6GBTree7DoBoostEPNS_7DMatrixEPSt6vectorINS_6detail18bst_gpair_internalIfEESaIS7_EEPNS_11ObjFunctionE+0xb50) [0x7ff9f289d1b0]\n[bt] (5) /home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/libxgboost.so(_ZN7xgboost11LearnerImpl13UpdateOneIterEiPNS_7DMatrixE+0x22b) [0x7ff9f28aa0db]\n[bt] (6) /home/jon/.pyenv/versions/3.6.1/lib/python3.6/site-packages/xgboost/libxgboost.so(XGBoosterUpdateOneIter+0x27) [0x7ff9f27e7de7]\n[bt] (7) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7ff9b74f9e40]\n[bt] (8) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x2eb) [0x7ff9b74f98ab]\n[bt] (9) /home/jon/.pyenv/versions/3.6.1/lib/python3.6/lib-dynload/_ctypes.cpython-36m-x86_64-linux-gnu.so(_ctypes_callproc+0x2cf) [0x7ff9b773bc4f]\n'
khotilov commented 6 years ago

@hcho3 it looks like the hist updater cannot handle the corner case when subsampling picks zero rows. Minimal reprex:

import xgboost as xgb
X = [[ 1.6,  0.2 , 4.8],[ 5.1,  1.6  ,6. ], [ 5.8,  2.2,  6.5]]
y = [ 3.4,  2.7 , 3.]
params = {'subsample': 0.2, 'max_depth': 2, 'n_estimators': 100, 'tree_method': 'hist',
          'objective': 'reg:linear', 'random_state': 123, 'silent': 0, 'debug_verbose': 2}
model = xgb.XGBRegressor(**params)
model.fit(X,y)
hcho3 commented 6 years ago

Let me take a look at it and get back to you. Thanks!

pseudotensor commented 6 years ago

Yes, it should be an easy fix and should behave like gpu_hist.

hcho3 commented 6 years ago

@pseudotensor I've submitted a small PR to handle the edge case. See #2817.

pseudotensor commented 6 years ago

Looks good. Not how handled in the gpu algorithms, but hopefully sufficient.

hcho3 commented 6 years ago

I believe the GPU updater uses a different method to keep track of instance sets (i.e. which row belongs to which node).

RAMitchell commented 6 years ago

I just zeroed the gradients for 'not selected' rows to achieve subsampling in the current gpu algos. I will probably do it differently for the next version.