search

microsoft / LightGBM

A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.
https://lightgbm.readthedocs.io/en/latest/
MIT License
16.7k stars 3.83k forks source link

GOSS/GBDT difference using C-API #3182

Closed yalwan-iqvia closed 4 years ago

yalwan-iqvia commented 4 years ago

LightGBM version: 2.3.1 Language: C-API, via julia FFI Platform: Linux

Context: Julia wrapper was missing support for boosting parameter (and commensurate parameters supporting it). So we worked on adding support, including tests.

We wrote a test which involved essentially the following assertion:

If system receives different parameter (or dart/goss hyperparameters) then model will be measurably different.

This worked fine for DART, but when we varied GOSS parameters model was coming out as identical to GOSS model with default parameters. So we went a step further and wrote a test to check that DART-GBDT-GOSS models are different. This test showed that GBDT and GOSS models appear identical.

So far, this is at point of higher julia interface, which is not LightGBM teams concern. We accepted the possibility we might have made mistake in implementation, so we continued to test by using low level C-API constructs only. But we still continued to find GDBT and GOSS models identical.

Below is some julia code (corresponding to C-API calls only) and python code. What we are confused by is the fact that using GOSS and GBDT clearly results in different models when used via python API, so now we are simply confused as to what we are doing wrong. We thought that GOSS might require some higher level implementation support, but could not find any suggestion of this in the python code.

Any advice you can provide on what we might be doing wrong here would be greatly appreciated.

Julia C-API code:

import LightGBM
X = randn(2000, 20)
y = rand(2000)
ds = LightGBM.LGBM_DatasetCreateFromMat(X, "")
LightGBM.LGBM_DatasetSetField(ds, "label", y)
gbdt = LightGBM.LGBM_BoosterCreate(ds, "boosting=gbdt")
goss = LightGBM.LGBM_BoosterCreate(ds, "boosting=goss")
LightGBM.LGBM_BoosterUpdateOneIter(gbdt)
LightGBM.LGBM_BoosterUpdateOneIter(goss)
p_gbdt = LightGBM.LGBM_BoosterPredictForMat(gbdt, X, 0, 0)
p_goss = LightGBM.LGBM_BoosterPredictForMat(goss, X, 0, 0)
sum(abs.(p_goss - p_gbdt))

^ This printed 0.0 (it also printed log line like this: [LightGBM] [Info] Using GOSS when the GOSS booster initialise)

Python code:

import lightgbm                                                                                                                                                                                                                       
import numpy                                                                                                                                                                                                                          
X = numpy.random.randn(2000, 20)                                                                                                                                                                                                      
y = numpy.random.rand(2000)                                                                                                                                                                                                           
goss = lightgbm.LGBMRegressor(boosting_type="goss")                                                                                                                                                                                   
gbdt = lightgbm.LGBMRegressor(boosting_type="gbdt")                                                                                                                                                                                   
goss.fit(X,y)                                                                                                                                                                                                                         
gbdt.fit(X,y)                                                                                                                                                                                                                         
p_gbdt = gbdt.predict(X)                                                                                                                                                                                                             
p_goss = goss.predict(X)                                                                                                                                                                                                             
numpy.sum(numpy.abs(p_gbdt - p_goss))

^ This printed a very large number.

guolinke commented 4 years ago

@yalwan-iqvia did you check the content of the learned model?

yalwan-iqvia commented 4 years ago

I did not, however I can show here now:

GBDT

tree
version=v3
num_class=1
num_tree_per_iteration=1
label_index=0
max_feature_idx=19
objective=regression
feature_names=Column_0 Column_1 Column_2 Column_3 Column_4 Column_5 Column_6 Column_7 Column_8 Column_9 Column_10 Column_11 Column_12 Column_13 Column_14 Column_15 Column_16 Column_17 Column_18 Column_19
feature_infos=[-3.2733013881585866:3.4833147682996555] [-3.2804328814272052:3.6959268483744041] [-3.3820303808666008:3.4212808400394574] [-4.7172174803834848:3.5857266628388405] [-3.8403702450771515:3.4902505670388391] [-3.1640235925651798:3.0515853490167379] [-4.079473640247409:3.5991186914356708] [-3.4138515830336367:3.8674678067634929] [-3.3975133176179018:3.0293573610041067] [-3.3888213336762805:3.3662896576252259] [-3.4605226385548802:3.5404887273301289] [-3.4799410331095313:3.5151452058437713] [-3.2999910772996079:3.1288688234760484] [-3.5823103761399491:3.3806482030234224] [-3.199507796108723:3.0755552763755598] [-3.4898004972833396:3.7818545791069131] [-4.0358890780268215:3.443026581307882] [-3.1233375903319156:3.3650166331790512] [-3.2850581094472333:3.696365236857281] [-3.4673233391646456:3.2033143251598686]
tree_sizes=2738

Tree=0
num_leaves=31
num_cat=0
split_feature=7 16 0 18 12 7 1 18 9 4 12 8 2 11 11 2 14 1 17 4 0 6 12 5 14 15 8 17 2 8
split_gain=0.949824 1.52552 1.19322 0.95592 0.897048 0.883276 1.03114 0.782851 0.685373 0.664125 0.626097 0.757236 0.617914 0.617859 0.589616 0.752389 0.759118 0.771848 0.646372 0.590284 0.756141 0.776763 0.584936 0.576096 0.560077 0.811515 0.556985 0.520277 0.754051 0.523781
threshold=0.07518347536971616 0.79086949960697051 -0.030617704092431624 -0.24013028690466706 1.0504212804306532 0.70610643137463491 0.89474271437253095 -1.1834546494961431 -0.27517864833053901 1.6463656736395158 0.2293231109901083 0.24629583835501664 0.49830554578347119 0.67244817181616379 0.37602232551414866 0.15593206745447583 -1.3183123519403408 1.1375961853451699 -0.95121422636250486 -1.0240308720030742 0.39591965582335181 -0.051171191201899384 1.2949579800233029 -0.064240548796357155 0.13660857442584698 0.60943384344546858 0.14919370056658213 -1.1613399734975405 0.29145898895013189 -0.17900290315282594
decision_type=2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
left_child=9 2 3 12 5 10 -7 -3 -9 14 23 -12 13 -2 15 16 -1 24 -17 -20 21 -21 -6 -4 -18 -26 -10 -5 -29 -30
right_child=1 7 4 27 22 6 -8 8 26 -11 11 -13 -14 -15 -16 18 17 -19 19 20 -22 -23 -24 -25 25 -27 -28 28 29 -31
leaf_value=0.50662271383786595 0.50196627124271864 0.49926121152469349 0.50299978308178839 0.47606309857763701 0.5109922419635986 0.47657818662211648 0.49920853669833742 0.47199734318547831 0.47814921727752829 0.48227426952191876 0.47068466331189851 0.49580497402177676 0.49290797478787673 0.52015955951919512 0.49161103559936337 0.51057010112915646 0.49417975450939788 0.50519186660088755 0.48636713307946855 0.5028259499133112 0.50977763818158284 0.4878038747385216 0.48976449536979827 0.48862400702464198 0.48074108075657185 0.49953769540648019 0.49299998474082246 0.49836115932001324 0.4934943412732391 0.47709975297959972
leaf_weight=39 84 23 50 20 25 125 24 70 56 45 28 21 54 24 388 51 148 38 36 79 59 61 27 63 98 30 46 110 40 38
leaf_count=39 84 23 50 20 25 125 24 70 56 45 28 21 54 24 388 51 148 38 36 79 59 61 27 63 98 30 46 110 40 38
internal_value=0 -0.0234223 -0.00251006 0.0374531 -0.0432439 -0.063571 -0.11914 -0.102031 -0.125201 0.020276 -0.0124613 -0.106868 0.095048 0.138719 0.0254862 0.0441571 0.0132707 -0.0030724 0.082279 0.0601323 0.0814491 0.0414328 0.078328 0.0284763 -0.021469 -0.069908 -0.0729072 -0.00740447 0.00890808 -0.0663011
internal_weight=0 928 733 370 363 311 149 195 172 1072 162 49 162 108 1027 639 353 314 286 235 199 140 52 113 276 128 102 208 188 78
internal_count=2000 928 733 370 363 311 149 195 172 1072 162 49 162 108 1027 639 353 314 286 235 199 140 52 113 276 128 102 208 188 78
shrinkage=1

end of trees

feature importances:
Column_2=3
Column_8=3
Column_12=3
Column_0=2
Column_1=2
Column_4=2
Column_7=2
Column_11=2
Column_14=2
Column_17=2
Column_18=2
Column_5=1
Column_6=1
Column_9=1
Column_15=1
Column_16=1

parameters:
[boosting: gbdt]
[objective: regression]
[metric: ]
[tree_learner: serial]
[device_type: cpu]
[data: ]
[valid: ]
[num_iterations: 100]
[learning_rate: 0.1]
[num_leaves: 31]
[num_threads: 0]
[max_depth: -1]
[min_data_in_leaf: 20]
[min_sum_hessian_in_leaf: 0.001]
[bagging_fraction: 1]
[pos_bagging_fraction: 1]
[neg_bagging_fraction: 1]
[bagging_freq: 0]
[bagging_seed: 3]
[feature_fraction: 1]
[feature_fraction_bynode: 1]
[feature_fraction_seed: 2]
[early_stopping_round: 0]
[first_metric_only: 0]
[max_delta_step: 0]
[lambda_l1: 0]
[lambda_l2: 0]
[min_gain_to_split: 0]
[drop_rate: 0.1]
[max_drop: 50]
[skip_drop: 0.5]
[xgboost_dart_mode: 0]
[uniform_drop: 0]
[drop_seed: 4]
[top_rate: 0.2]
[other_rate: 0.1]
[min_data_per_group: 100]
[max_cat_threshold: 32]
[cat_l2: 10]
[cat_smooth: 10]
[max_cat_to_onehot: 4]
[top_k: 20]
[monotone_constraints: ]
[feature_contri: ]
[forcedsplits_filename: ]
[forcedbins_filename: ]
[refit_decay_rate: 0.9]
[cegb_tradeoff: 1]
[cegb_penalty_split: 0]
[cegb_penalty_feature_lazy: ]
[cegb_penalty_feature_coupled: ]
[verbosity: 1]
[max_bin: 255]
[max_bin_by_feature: ]
[min_data_in_bin: 3]
[bin_construct_sample_cnt: 200000]
[histogram_pool_size: -1]
[data_random_seed: 1]
[output_model: LightGBM_model.txt]
[snapshot_freq: -1]
[input_model: ]
[output_result: LightGBM_predict_result.txt]
[initscore_filename: ]
[valid_data_initscores: ]
[pre_partition: 0]
[enable_bundle: 1]
[max_conflict_rate: 0]
[is_enable_sparse: 1]
[sparse_threshold: 0.8]
[use_missing: 1]
[zero_as_missing: 0]
[two_round: 0]
[save_binary: 0]
[header: 0]
[label_column: ]
[weight_column: ]
[group_column: ]
[ignore_column: ]
[categorical_feature: ]
[predict_raw_score: 0]
[predict_leaf_index: 0]
[predict_contrib: 0]
[num_iteration_predict: -1]
[pred_early_stop: 0]
[pred_early_stop_freq: 10]
[pred_early_stop_margin: 10]
[convert_model_language: ]
[convert_model: gbdt_prediction.cpp]
[num_class: 1]
[is_unbalance: 0]
[scale_pos_weight: 1]
[sigmoid: 1]
[boost_from_average: 1]
[reg_sqrt: 0]
[alpha: 0.9]
[fair_c: 1]
[poisson_max_delta_step: 0.7]
[tweedie_variance_power: 1.5]
[max_position: 20]
[lambdamart_norm: 1]
[label_gain: ]
[metric_freq: 1]
[is_provide_training_metric: 0]
[eval_at: ]
[multi_error_top_k: 1]
[num_machines: 1]
[local_listen_port: 12400]
[time_out: 120]
[machine_list_filename: ]
[machines: ]
[gpu_platform_id: -1]
[gpu_device_id: -1]
[gpu_use_dp: 0]

end of parameters

GOSS

tree
version=v3
num_class=1
num_tree_per_iteration=1
label_index=0
max_feature_idx=19
objective=regression
feature_names=Column_0 Column_1 Column_2 Column_3 Column_4 Column_5 Column_6 Column_7 Column_8 Column_9 Column_10 Column_11 Column_12 Column_13 Column_14 Column_15 Column_16 Column_17 Column_18 Column_19
feature_infos=[-3.2733013881585866:3.4833147682996555] [-3.2804328814272052:3.6959268483744041] [-3.3820303808666008:3.4212808400394574] [-4.7172174803834848:3.5857266628388405] [-3.8403702450771515:3.4902505670388391] [-3.1640235925651798:3.0515853490167379] [-4.079473640247409:3.5991186914356708] [-3.4138515830336367:3.8674678067634929] [-3.3975133176179018:3.0293573610041067] [-3.3888213336762805:3.3662896576252259] [-3.4605226385548802:3.5404887273301289] [-3.4799410331095313:3.5151452058437713] [-3.2999910772996079:3.1288688234760484] [-3.5823103761399491:3.3806482030234224] [-3.199507796108723:3.0755552763755598] [-3.4898004972833396:3.7818545791069131] [-4.0358890780268215:3.443026581307882] [-3.1233375903319156:3.3650166331790512] [-3.2850581094472333:3.696365236857281] [-3.4673233391646456:3.2033143251598686]
tree_sizes=2738

Tree=0
num_leaves=31
num_cat=0
split_feature=7 16 0 18 12 7 1 18 9 4 12 8 2 11 11 2 14 1 17 4 0 6 12 5 14 15 8 17 2 8
split_gain=0.949824 1.52552 1.19322 0.95592 0.897048 0.883276 1.03114 0.782851 0.685373 0.664125 0.626097 0.757236 0.617914 0.617859 0.589616 0.752389 0.759118 0.771848 0.646372 0.590284 0.756141 0.776763 0.584936 0.576096 0.560077 0.811515 0.556985 0.520277 0.754051 0.523781
threshold=0.07518347536971616 0.79086949960697051 -0.030617704092431624 -0.24013028690466706 1.0504212804306532 0.70610643137463491 0.89474271437253095 -1.1834546494961431 -0.27517864833053901 1.6463656736395158 0.2293231109901083 0.24629583835501664 0.49830554578347119 0.67244817181616379 0.37602232551414866 0.15593206745447583 -1.3183123519403408 1.1375961853451699 -0.95121422636250486 -1.0240308720030742 0.39591965582335181 -0.051171191201899384 1.2949579800233029 -0.064240548796357155 0.13660857442584698 0.60943384344546858 0.14919370056658213 -1.1613399734975405 0.29145898895013189 -0.17900290315282594
decision_type=2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
left_child=9 2 3 12 5 10 -7 -3 -9 14 23 -12 13 -2 15 16 -1 24 -17 -20 21 -21 -6 -4 -18 -26 -10 -5 -29 -30
right_child=1 7 4 27 22 6 -8 8 26 -11 11 -13 -14 -15 -16 18 17 -19 19 20 -22 -23 -24 -25 25 -27 -28 28 29 -31
leaf_value=0.50662271383786595 0.50196627124271864 0.49926121152469349 0.50299978308178839 0.47606309857763701 0.5109922419635986 0.47657818662211648 0.49920853669833742 0.47199734318547831 0.47814921727752829 0.48227426952191876 0.47068466331189851 0.49580497402177676 0.49290797478787673 0.52015955951919512 0.49161103559936337 0.51057010112915646 0.49417975450939788 0.50519186660088755 0.48636713307946855 0.5028259499133112 0.50977763818158284 0.4878038747385216 0.48976449536979827 0.48862400702464198 0.48074108075657185 0.49953769540648019 0.49299998474082246 0.49836115932001324 0.4934943412732391 0.47709975297959972
leaf_weight=39 84 23 50 20 25 125 24 70 56 45 28 21 54 24 388 51 148 38 36 79 59 61 27 63 98 30 46 110 40 38
leaf_count=39 84 23 50 20 25 125 24 70 56 45 28 21 54 24 388 51 148 38 36 79 59 61 27 63 98 30 46 110 40 38
internal_value=0 -0.0234223 -0.00251006 0.0374531 -0.0432439 -0.063571 -0.11914 -0.102031 -0.125201 0.020276 -0.0124613 -0.106868 0.095048 0.138719 0.0254862 0.0441571 0.0132707 -0.0030724 0.082279 0.0601323 0.0814491 0.0414328 0.078328 0.0284763 -0.021469 -0.069908 -0.0729072 -0.00740447 0.00890808 -0.0663011
internal_weight=0 928 733 370 363 311 149 195 172 1072 162 49 162 108 1027 639 353 314 286 235 199 140 52 113 276 128 102 208 188 78
internal_count=2000 928 733 370 363 311 149 195 172 1072 162 49 162 108 1027 639 353 314 286 235 199 140 52 113 276 128 102 208 188 78
shrinkage=1

end of trees

feature importances:
Column_2=3
Column_8=3
Column_12=3
Column_0=2
Column_1=2
Column_4=2
Column_7=2
Column_11=2
Column_14=2
Column_17=2
Column_18=2
Column_5=1
Column_6=1
Column_9=1
Column_15=1
Column_16=1

parameters:
[boosting: goss]
[objective: regression]
[metric: ]
[tree_learner: serial]
[device_type: cpu]
[data: ]
[valid: ]
[num_iterations: 100]
[learning_rate: 0.1]
[num_leaves: 31]
[num_threads: 0]
[max_depth: -1]
[min_data_in_leaf: 20]
[min_sum_hessian_in_leaf: 0.001]
[bagging_fraction: 1]
[pos_bagging_fraction: 1]
[neg_bagging_fraction: 1]
[bagging_freq: 0]
[bagging_seed: 3]
[feature_fraction: 1]
[feature_fraction_bynode: 1]
[feature_fraction_seed: 2]
[early_stopping_round: 0]
[first_metric_only: 0]
[max_delta_step: 0]
[lambda_l1: 0]
[lambda_l2: 0]
[min_gain_to_split: 0]
[drop_rate: 0.1]
[max_drop: 50]
[skip_drop: 0.5]
[xgboost_dart_mode: 0]
[uniform_drop: 0]
[drop_seed: 4]
[top_rate: 0.2]
[other_rate: 0.1]
[min_data_per_group: 100]
[max_cat_threshold: 32]
[cat_l2: 10]
[cat_smooth: 10]
[max_cat_to_onehot: 4]
[top_k: 20]
[monotone_constraints: ]
[feature_contri: ]
[forcedsplits_filename: ]
[forcedbins_filename: ]
[refit_decay_rate: 0.9]
[cegb_tradeoff: 1]
[cegb_penalty_split: 0]
[cegb_penalty_feature_lazy: ]
[cegb_penalty_feature_coupled: ]
[verbosity: 1]
[max_bin: 255]
[max_bin_by_feature: ]
[min_data_in_bin: 3]
[bin_construct_sample_cnt: 200000]
[histogram_pool_size: -1]
[data_random_seed: 1]
[output_model: LightGBM_model.txt]
[snapshot_freq: -1]
[input_model: ]
[output_result: LightGBM_predict_result.txt]
[initscore_filename: ]
[valid_data_initscores: ]
[pre_partition: 0]
[enable_bundle: 1]
[max_conflict_rate: 0]
[is_enable_sparse: 1]
[sparse_threshold: 0.8]
[use_missing: 1]
[zero_as_missing: 0]
[two_round: 0]
[save_binary: 0]
[header: 0]
[label_column: ]
[weight_column: ]
[group_column: ]
[ignore_column: ]
[categorical_feature: ]
[predict_raw_score: 0]
[predict_leaf_index: 0]
[predict_contrib: 0]
[num_iteration_predict: -1]
[pred_early_stop: 0]
[pred_early_stop_freq: 10]
[pred_early_stop_margin: 10]
[convert_model_language: ]
[convert_model: gbdt_prediction.cpp]
[num_class: 1]
[is_unbalance: 0]
[scale_pos_weight: 1]
[sigmoid: 1]
[boost_from_average: 1]
[reg_sqrt: 0]
[alpha: 0.9]
[fair_c: 1]
[poisson_max_delta_step: 0.7]
[tweedie_variance_power: 1.5]
[max_position: 20]
[lambdamart_norm: 1]
[label_gain: ]
[metric_freq: 1]
[is_provide_training_metric: 0]
[eval_at: ]
[multi_error_top_k: 1]
[num_machines: 1]
[local_listen_port: 12400]
[time_out: 120]
[machine_list_filename: ]
[machines: ]
[gpu_platform_id: -1]
[gpu_device_id: -1]
[gpu_use_dp: 0]

end of parameters

I put these through a difftool and only the boosting parameter field differs, everything else is identical.

yalwan-iqvia commented 4 years ago

Having showed this I realised that it is stupid to expect models to vary on basis of gradients if the model is never given chance to update from gradients, so I call UpdateOneIter twice for each model, so we have at least one gradient update step, model result is similar:

GBDT

tree
version=v3
num_class=1
num_tree_per_iteration=1
label_index=0
max_feature_idx=19
objective=regression
feature_names=Column_0 Column_1 Column_2 Column_3 Column_4 Column_5 Column_6 Column_7 Column_8 Column_9 Column_10 Column_11 Column_12 Column_13 Column_14 Column_15 Column_16 Column_17 Column_18 Column_19
feature_infos=[-3.2733013881585866:3.4833147682996555] [-3.2804328814272052:3.6959268483744041] [-3.3820303808666008:3.4212808400394574] [-4.7172174803834848:3.5857266628388405] [-3.8403702450771515:3.4902505670388391] [-3.1640235925651798:3.0515853490167379] [-4.079473640247409:3.5991186914356708] [-3.4138515830336367:3.8674678067634929] [-3.3975133176179018:3.0293573610041067] [-3.3888213336762805:3.3662896576252259] [-3.4605226385548802:3.5404887273301289] [-3.4799410331095313:3.5151452058437713] [-3.2999910772996079:3.1288688234760484] [-3.5823103761399491:3.3806482030234224] [-3.199507796108723:3.0755552763755598] [-3.4898004972833396:3.7818545791069131] [-4.0358890780268215:3.443026581307882] [-3.1233375903319156:3.3650166331790512] [-3.2850581094472333:3.696365236857281] [-3.4673233391646456:3.2033143251598686]
tree_sizes=2738 2829

Tree=0
num_leaves=31
num_cat=0
split_feature=7 16 0 18 12 7 1 18 9 4 12 8 2 11 11 2 14 1 17 4 0 6 12 5 14 15 8 17 2 8
split_gain=0.949824 1.52552 1.19322 0.95592 0.897048 0.883276 1.03114 0.782851 0.685373 0.664125 0.626097 0.757236 0.617914 0.617859 0.589616 0.752389 0.759118 0.771848 0.646372 0.590284 0.756141 0.776763 0.584936 0.576096 0.560077 0.811515 0.556985 0.520277 0.754051 0.523781
threshold=0.07518347536971616 0.79086949960697051 -0.030617704092431624 -0.24013028690466706 1.0504212804306532 0.70610643137463491 0.89474271437253095 -1.1834546494961431 -0.27517864833053901 1.6463656736395158 0.2293231109901083 0.24629583835501664 0.49830554578347119 0.67244817181616379 0.37602232551414866 0.15593206745447583 -1.3183123519403408 1.1375961853451699 -0.95121422636250486 -1.0240308720030742 0.39591965582335181 -0.051171191201899384 1.2949579800233029 -0.064240548796357155 0.13660857442584698 0.60943384344546858 0.14919370056658213 -1.1613399734975405 0.29145898895013189 -0.17900290315282594
decision_type=2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
left_child=9 2 3 12 5 10 -7 -3 -9 14 23 -12 13 -2 15 16 -1 24 -17 -20 21 -21 -6 -4 -18 -26 -10 -5 -29 -30
right_child=1 7 4 27 22 6 -8 8 26 -11 11 -13 -14 -15 -16 18 17 -19 19 20 -22 -23 -24 -25 25 -27 -28 28 29 -31
leaf_value=0.50662271383786595 0.50196627124271864 0.49926121152469349 0.50299978308178839 0.47606309857763701 0.5109922419635986 0.47657818662211648 0.49920853669833742 0.47199734318547831 0.47814921727752829 0.48227426952191876 0.47068466331189851 0.49580497402177676 0.49290797478787673 0.52015955951919512 0.49161103559936337 0.51057010112915646 0.49417975450939788 0.50519186660088755 0.48636713307946855 0.5028259499133112 0.50977763818158284 0.4878038747385216 0.48976449536979827 0.48862400702464198 0.48074108075657185 0.49953769540648019 0.49299998474082246 0.49836115932001324 0.4934943412732391 0.47709975297959972
leaf_weight=39 84 23 50 20 25 125 24 70 56 45 28 21 54 24 388 51 148 38 36 79 59 61 27 63 98 30 46 110 40 38
leaf_count=39 84 23 50 20 25 125 24 70 56 45 28 21 54 24 388 51 148 38 36 79 59 61 27 63 98 30 46 110 40 38
internal_value=0 -0.0234223 -0.00251006 0.0374531 -0.0432439 -0.063571 -0.11914 -0.102031 -0.125201 0.020276 -0.0124613 -0.106868 0.095048 0.138719 0.0254862 0.0441571 0.0132707 -0.0030724 0.082279 0.0601323 0.0814491 0.0414328 0.078328 0.0284763 -0.021469 -0.069908 -0.0729072 -0.00740447 0.00890808 -0.0663011
internal_weight=0 928 733 370 363 311 149 195 172 1072 162 49 162 108 1027 639 353 314 286 235 199 140 52 113 276 128 102 208 188 78
internal_count=2000 928 733 370 363 311 149 195 172 1072 162 49 162 108 1027 639 353 314 286 235 199 140 52 113 276 128 102 208 188 78
shrinkage=1

Tree=1
num_leaves=31
num_cat=0
split_feature=7 16 0 18 12 7 1 18 3 4 2 14 13 5 6 11 2 16 11 8 5 12 17 5 4 11 7 7 13 19
split_gain=0.769357 1.23568 0.966504 0.774295 0.726609 0.719941 0.849724 0.641725 0.590124 0.537941 0.50051 0.586315 0.637403 0.480706 0.477893 0.477589 0.675233 0.689575 0.664009 0.618048 0.903145 0.729807 0.610353 0.5935 0.533542 0.498309 0.888207 0.827166 0.580535 0.551922
threshold=0.07518347536971616 0.79086949960697051 -0.030617704092431624 -0.24013028690466706 1.0504212804306532 0.71960727763707266 0.89474271437253095 -0.46604958044660177 -0.89550849332134053 1.6463656736395158 0.49830554578347119 0.84191925784387556 0.3698616903086655 -0.31199572877269699 0.94218293560020827 0.37602232551414866 1.846403153203857 -0.071131098990188338 -0.77211816234398567 -0.2112936732694671 -0.3745391013059432 0.89617181103099364 -0.52623686325725294 -0.25624475018049847 0.50441875751133014 0.84012371516118478 -0.03809903015425755 -0.17791698943648537 -0.44302272677630805 0.48133230342310057
decision_type=2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
left_child=9 2 3 10 5 14 -7 13 -9 15 11 12 -2 -3 -4 16 17 19 22 -1 -21 24 -19 -24 -22 27 28 -17 -27 -30
right_child=1 7 4 -5 -6 6 -8 8 -10 -11 -12 -13 -14 -15 -16 25 -18 18 -20 20 21 -23 23 -25 -26 26 -28 -29 29 -31
leaf_value=0.0050868381175953221 0.022674663873728029 -0.011765928234419097 0.0010786925068324699 -0.00066640245500960971 0.0070495228059231675 -0.014219922644131987 0.0063640768639743328 0.00091889705415815117 -0.016087395659394075 -0.0088767636194825167 0.00069357149164985736 -0.0020830210277133368 0.0047440991301329867 0.0067417007094870019 -0.013931055910885335 -0.001614064513705671 0.019758240838956016 0.012934986352255302 0.0098350704549323015 -0.014957566158670299 -0.0072330795995838625 0.017674482028399194 0.0058721120578839499 -0.01147249958458099 0.010291803288071054 0.008475354412803426 0.022858781050890686 -0.021242742866277697 -0.0075121297051062748 0.0068842645832582526
leaf_weight=127 55 23 140 208 52 122 24 25 111 45 54 22 31 36 25 152 26 28 209 53 71 21 34 47 23 64 20 25 89 38
leaf_count=127 55 23 140 208 52 122 24 25 111 45 54 22 31 36 25 152 26 28 209 53 71 21 34 47 23 64 20 25 89 38
internal_value=0 -0.0210801 -0.00225905 0.0337078 -0.0389195 -0.0572139 -0.108363 -0.0918279 -0.129612 0.0182484 0.0855432 0.124847 0.162113 -0.00473138 -0.0119551 0.0229376 0.0397414 0.0330466 0.0653507 -0.00177612 -0.0415729 0.00820234 0.00207559 -0.0419205 -0.0294508 -0.00473674 0.0280863 -0.0438648 0.00709137 -0.0320455
internal_weight=0 928 733 370 363 311 146 195 136 1072 162 108 86 59 165 1027 639 613 318 295 168 115 109 81 94 388 211 177 191 127
internal_count=2000 928 733 370 363 311 146 195 136 1072 162 108 86 59 165 1027 639 613 318 295 168 115 109 81 94 388 211 177 191 127
shrinkage=0.1

end of trees

feature importances:
Column_7=6
Column_2=5
Column_11=5
Column_12=5
Column_4=4
Column_5=4
Column_8=4
Column_18=4
Column_0=3
Column_1=3
Column_14=3
Column_16=3
Column_17=3
Column_6=2
Column_13=2
Column_3=1
Column_9=1
Column_15=1
Column_19=1

parameters:
[boosting: gbdt]
[objective: regression]
[metric: ]
[tree_learner: serial]
[device_type: cpu]
[data: ]
[valid: ]
[num_iterations: 100]
[learning_rate: 0.1]
[num_leaves: 31]
[num_threads: 0]
[max_depth: -1]
[min_data_in_leaf: 20]
[min_sum_hessian_in_leaf: 0.001]
[bagging_fraction: 1]
[pos_bagging_fraction: 1]
[neg_bagging_fraction: 1]
[bagging_freq: 0]
[bagging_seed: 3]
[feature_fraction: 1]
[feature_fraction_bynode: 1]
[feature_fraction_seed: 2]
[early_stopping_round: 0]
[first_metric_only: 0]
[max_delta_step: 0]
[lambda_l1: 0]
[lambda_l2: 0]
[min_gain_to_split: 0]
[drop_rate: 0.1]
[max_drop: 50]
[skip_drop: 0.5]
[xgboost_dart_mode: 0]
[uniform_drop: 0]
[drop_seed: 4]
[top_rate: 0.2]
[other_rate: 0.1]
[min_data_per_group: 100]
[max_cat_threshold: 32]
[cat_l2: 10]
[cat_smooth: 10]
[max_cat_to_onehot: 4]
[top_k: 20]
[monotone_constraints: ]
[feature_contri: ]
[forcedsplits_filename: ]
[forcedbins_filename: ]
[refit_decay_rate: 0.9]
[cegb_tradeoff: 1]
[cegb_penalty_split: 0]
[cegb_penalty_feature_lazy: ]
[cegb_penalty_feature_coupled: ]
[verbosity: 1]
[max_bin: 255]
[max_bin_by_feature: ]
[min_data_in_bin: 3]
[bin_construct_sample_cnt: 200000]
[histogram_pool_size: -1]
[data_random_seed: 1]
[output_model: LightGBM_model.txt]
[snapshot_freq: -1]
[input_model: ]
[output_result: LightGBM_predict_result.txt]
[initscore_filename: ]
[valid_data_initscores: ]
[pre_partition: 0]
[enable_bundle: 1]
[max_conflict_rate: 0]
[is_enable_sparse: 1]
[sparse_threshold: 0.8]
[use_missing: 1]
[zero_as_missing: 0]
[two_round: 0]
[save_binary: 0]
[header: 0]
[label_column: ]
[weight_column: ]
[group_column: ]
[ignore_column: ]
[categorical_feature: ]
[predict_raw_score: 0]
[predict_leaf_index: 0]
[predict_contrib: 0]
[num_iteration_predict: -1]
[pred_early_stop: 0]
[pred_early_stop_freq: 10]
[pred_early_stop_margin: 10]
[convert_model_language: ]
[convert_model: gbdt_prediction.cpp]
[num_class: 1]
[is_unbalance: 0]
[scale_pos_weight: 1]
[sigmoid: 1]
[boost_from_average: 1]
[reg_sqrt: 0]
[alpha: 0.9]
[fair_c: 1]
[poisson_max_delta_step: 0.7]
[tweedie_variance_power: 1.5]
[max_position: 20]
[lambdamart_norm: 1]
[label_gain: ]
[metric_freq: 1]
[is_provide_training_metric: 0]
[eval_at: ]
[multi_error_top_k: 1]
[num_machines: 1]
[local_listen_port: 12400]
[time_out: 120]
[machine_list_filename: ]
[machines: ]
[gpu_platform_id: -1]
[gpu_device_id: -1]
[gpu_use_dp: 0]

end of parameters

GOSS

tree
version=v3
num_class=1
num_tree_per_iteration=1
label_index=0
max_feature_idx=19
objective=regression
feature_names=Column_0 Column_1 Column_2 Column_3 Column_4 Column_5 Column_6 Column_7 Column_8 Column_9 Column_10 Column_11 Column_12 Column_13 Column_14 Column_15 Column_16 Column_17 Column_18 Column_19
feature_infos=[-3.2733013881585866:3.4833147682996555] [-3.2804328814272052:3.6959268483744041] [-3.3820303808666008:3.4212808400394574] [-4.7172174803834848:3.5857266628388405] [-3.8403702450771515:3.4902505670388391] [-3.1640235925651798:3.0515853490167379] [-4.079473640247409:3.5991186914356708] [-3.4138515830336367:3.8674678067634929] [-3.3975133176179018:3.0293573610041067] [-3.3888213336762805:3.3662896576252259] [-3.4605226385548802:3.5404887273301289] [-3.4799410331095313:3.5151452058437713] [-3.2999910772996079:3.1288688234760484] [-3.5823103761399491:3.3806482030234224] [-3.199507796108723:3.0755552763755598] [-3.4898004972833396:3.7818545791069131] [-4.0358890780268215:3.443026581307882] [-3.1233375903319156:3.3650166331790512] [-3.2850581094472333:3.696365236857281] [-3.4673233391646456:3.2033143251598686]
tree_sizes=2738 2829

Tree=0
num_leaves=31
num_cat=0
split_feature=7 16 0 18 12 7 1 18 9 4 12 8 2 11 11 2 14 1 17 4 0 6 12 5 14 15 8 17 2 8
split_gain=0.949824 1.52552 1.19322 0.95592 0.897048 0.883276 1.03114 0.782851 0.685373 0.664125 0.626097 0.757236 0.617914 0.617859 0.589616 0.752389 0.759118 0.771848 0.646372 0.590284 0.756141 0.776763 0.584936 0.576096 0.560077 0.811515 0.556985 0.520277 0.754051 0.523781
threshold=0.07518347536971616 0.79086949960697051 -0.030617704092431624 -0.24013028690466706 1.0504212804306532 0.70610643137463491 0.89474271437253095 -1.1834546494961431 -0.27517864833053901 1.6463656736395158 0.2293231109901083 0.24629583835501664 0.49830554578347119 0.67244817181616379 0.37602232551414866 0.15593206745447583 -1.3183123519403408 1.1375961853451699 -0.95121422636250486 -1.0240308720030742 0.39591965582335181 -0.051171191201899384 1.2949579800233029 -0.064240548796357155 0.13660857442584698 0.60943384344546858 0.14919370056658213 -1.1613399734975405 0.29145898895013189 -0.17900290315282594
decision_type=2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
left_child=9 2 3 12 5 10 -7 -3 -9 14 23 -12 13 -2 15 16 -1 24 -17 -20 21 -21 -6 -4 -18 -26 -10 -5 -29 -30
right_child=1 7 4 27 22 6 -8 8 26 -11 11 -13 -14 -15 -16 18 17 -19 19 20 -22 -23 -24 -25 25 -27 -28 28 29 -31
leaf_value=0.50662271383786595 0.50196627124271864 0.49926121152469349 0.50299978308178839 0.47606309857763701 0.5109922419635986 0.47657818662211648 0.49920853669833742 0.47199734318547831 0.47814921727752829 0.48227426952191876 0.47068466331189851 0.49580497402177676 0.49290797478787673 0.52015955951919512 0.49161103559936337 0.51057010112915646 0.49417975450939788 0.50519186660088755 0.48636713307946855 0.5028259499133112 0.50977763818158284 0.4878038747385216 0.48976449536979827 0.48862400702464198 0.48074108075657185 0.49953769540648019 0.49299998474082246 0.49836115932001324 0.4934943412732391 0.47709975297959972
leaf_weight=39 84 23 50 20 25 125 24 70 56 45 28 21 54 24 388 51 148 38 36 79 59 61 27 63 98 30 46 110 40 38
leaf_count=39 84 23 50 20 25 125 24 70 56 45 28 21 54 24 388 51 148 38 36 79 59 61 27 63 98 30 46 110 40 38
internal_value=0 -0.0234223 -0.00251006 0.0374531 -0.0432439 -0.063571 -0.11914 -0.102031 -0.125201 0.020276 -0.0124613 -0.106868 0.095048 0.138719 0.0254862 0.0441571 0.0132707 -0.0030724 0.082279 0.0601323 0.0814491 0.0414328 0.078328 0.0284763 -0.021469 -0.069908 -0.0729072 -0.00740447 0.00890808 -0.0663011
internal_weight=0 928 733 370 363 311 149 195 172 1072 162 49 162 108 1027 639 353 314 286 235 199 140 52 113 276 128 102 208 188 78
internal_count=2000 928 733 370 363 311 149 195 172 1072 162 49 162 108 1027 639 353 314 286 235 199 140 52 113 276 128 102 208 188 78
shrinkage=1

Tree=1
num_leaves=31
num_cat=0
split_feature=7 16 0 18 12 7 1 18 3 4 2 14 13 5 6 11 2 16 11 8 5 12 17 5 4 11 7 7 13 19
split_gain=0.769357 1.23568 0.966504 0.774295 0.726609 0.719941 0.849724 0.641725 0.590124 0.537941 0.50051 0.586315 0.637403 0.480706 0.477893 0.477589 0.675233 0.689575 0.664009 0.618048 0.903145 0.729807 0.610353 0.5935 0.533542 0.498309 0.888207 0.827166 0.580535 0.551922
threshold=0.07518347536971616 0.79086949960697051 -0.030617704092431624 -0.24013028690466706 1.0504212804306532 0.71960727763707266 0.89474271437253095 -0.46604958044660177 -0.89550849332134053 1.6463656736395158 0.49830554578347119 0.84191925784387556 0.3698616903086655 -0.31199572877269699 0.94218293560020827 0.37602232551414866 1.846403153203857 -0.071131098990188338 -0.77211816234398567 -0.2112936732694671 -0.3745391013059432 0.89617181103099364 -0.52623686325725294 -0.25624475018049847 0.50441875751133014 0.84012371516118478 -0.03809903015425755 -0.17791698943648537 -0.44302272677630805 0.48133230342310057
decision_type=2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
left_child=9 2 3 10 5 14 -7 13 -9 15 11 12 -2 -3 -4 16 17 19 22 -1 -21 24 -19 -24 -22 27 28 -17 -27 -30
right_child=1 7 4 -5 -6 6 -8 8 -10 -11 -12 -13 -14 -15 -16 25 -18 18 -20 20 21 -23 23 -25 -26 26 -28 -29 29 -31
leaf_value=0.0050868381175953221 0.022674663873728029 -0.011765928234419097 0.0010786925068324699 -0.00066640245500960971 0.0070495228059231675 -0.014219922644131987 0.0063640768639743328 0.00091889705415815117 -0.016087395659394075 -0.0088767636194825167 0.00069357149164985736 -0.0020830210277133368 0.0047440991301329867 0.0067417007094870019 -0.013931055910885335 -0.001614064513705671 0.019758240838956016 0.012934986352255302 0.0098350704549323015 -0.014957566158670299 -0.0072330795995838625 0.017674482028399194 0.0058721120578839499 -0.01147249958458099 0.010291803288071054 0.008475354412803426 0.022858781050890686 -0.021242742866277697 -0.0075121297051062748 0.0068842645832582526
leaf_weight=127 55 23 140 208 52 122 24 25 111 45 54 22 31 36 25 152 26 28 209 53 71 21 34 47 23 64 20 25 89 38
leaf_count=127 55 23 140 208 52 122 24 25 111 45 54 22 31 36 25 152 26 28 209 53 71 21 34 47 23 64 20 25 89 38
internal_value=0 -0.0210801 -0.00225905 0.0337078 -0.0389195 -0.0572139 -0.108363 -0.0918279 -0.129612 0.0182484 0.0855432 0.124847 0.162113 -0.00473138 -0.0119551 0.0229376 0.0397414 0.0330466 0.0653507 -0.00177612 -0.0415729 0.00820234 0.00207559 -0.0419205 -0.0294508 -0.00473674 0.0280863 -0.0438648 0.00709137 -0.0320455
internal_weight=0 928 733 370 363 311 146 195 136 1072 162 108 86 59 165 1027 639 613 318 295 168 115 109 81 94 388 211 177 191 127
internal_count=2000 928 733 370 363 311 146 195 136 1072 162 108 86 59 165 1027 639 613 318 295 168 115 109 81 94 388 211 177 191 127
shrinkage=0.1

end of trees

feature importances:
Column_7=6
Column_2=5
Column_11=5
Column_12=5
Column_4=4
Column_5=4
Column_8=4
Column_18=4
Column_0=3
Column_1=3
Column_14=3
Column_16=3
Column_17=3
Column_6=2
Column_13=2
Column_3=1
Column_9=1
Column_15=1
Column_19=1

parameters:
[boosting: goss]
[objective: regression]
[metric: ]
[tree_learner: serial]
[device_type: cpu]
[data: ]
[valid: ]
[num_iterations: 100]
[learning_rate: 0.1]
[num_leaves: 31]
[num_threads: 0]
[max_depth: -1]
[min_data_in_leaf: 20]
[min_sum_hessian_in_leaf: 0.001]
[bagging_fraction: 1]
[pos_bagging_fraction: 1]
[neg_bagging_fraction: 1]
[bagging_freq: 0]
[bagging_seed: 3]
[feature_fraction: 1]
[feature_fraction_bynode: 1]
[feature_fraction_seed: 2]
[early_stopping_round: 0]
[first_metric_only: 0]
[max_delta_step: 0]
[lambda_l1: 0]
[lambda_l2: 0]
[min_gain_to_split: 0]
[drop_rate: 0.1]
[max_drop: 50]
[skip_drop: 0.5]
[xgboost_dart_mode: 0]
[uniform_drop: 0]
[drop_seed: 4]
[top_rate: 0.2]
[other_rate: 0.1]
[min_data_per_group: 100]
[max_cat_threshold: 32]
[cat_l2: 10]
[cat_smooth: 10]
[max_cat_to_onehot: 4]
[top_k: 20]
[monotone_constraints: ]
[feature_contri: ]
[forcedsplits_filename: ]
[forcedbins_filename: ]
[refit_decay_rate: 0.9]
[cegb_tradeoff: 1]
[cegb_penalty_split: 0]
[cegb_penalty_feature_lazy: ]
[cegb_penalty_feature_coupled: ]
[verbosity: 1]
[max_bin: 255]
[max_bin_by_feature: ]
[min_data_in_bin: 3]
[bin_construct_sample_cnt: 200000]
[histogram_pool_size: -1]
[data_random_seed: 1]
[output_model: LightGBM_model.txt]
[snapshot_freq: -1]
[input_model: ]
[output_result: LightGBM_predict_result.txt]
[initscore_filename: ]
[valid_data_initscores: ]
[pre_partition: 0]
[enable_bundle: 1]
[max_conflict_rate: 0]
[is_enable_sparse: 1]
[sparse_threshold: 0.8]
[use_missing: 1]
[zero_as_missing: 0]
[two_round: 0]
[save_binary: 0]
[header: 0]
[label_column: ]
[weight_column: ]
[group_column: ]
[ignore_column: ]
[categorical_feature: ]
[predict_raw_score: 0]
[predict_leaf_index: 0]
[predict_contrib: 0]
[num_iteration_predict: -1]
[pred_early_stop: 0]
[pred_early_stop_freq: 10]
[pred_early_stop_margin: 10]
[convert_model_language: ]
[convert_model: gbdt_prediction.cpp]
[num_class: 1]
[is_unbalance: 0]
[scale_pos_weight: 1]
[sigmoid: 1]
[boost_from_average: 1]
[reg_sqrt: 0]
[alpha: 0.9]
[fair_c: 1]
[poisson_max_delta_step: 0.7]
[tweedie_variance_power: 1.5]
[max_position: 20]
[lambdamart_norm: 1]
[label_gain: ]
[metric_freq: 1]
[is_provide_training_metric: 0]
[eval_at: ]
[multi_error_top_k: 1]
[num_machines: 1]
[local_listen_port: 12400]
[time_out: 120]
[machine_list_filename: ]
[machines: ]
[gpu_platform_id: -1]
[gpu_device_id: -1]
[gpu_use_dp: 0]

end of parameters

Again if you put them through difftool, they differ only by boosting parameter

guolinke commented 4 years ago

@yalwan-iqvia I just remember the goss will use GBDT mode in the first k (k=1/learning_rate) iterations. So you can try to run more iterations.

yalwan-iqvia commented 4 years ago

@guolinke

This was the missing understanding! Thank you 😄

Is this documented somewhere that I missed, or could documentation do with a note regarding this?

guolinke commented 4 years ago

@yalwan-iqvia It seems it is not documented. Maybe we should enhance it.

github-actions[bot] commented 1 year ago

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.