jameslamb
commented
2 months ago Excellent report, thanks very much!
Could you try installing pandas and inspecting the results of gbm.trees_to_dataframe(), to see if maybe the later trees are very shallow?
That could be one reason for lower GPU utilization... the split-finding part of training can benefit from parallelization, but there's a sync-up after each search where the model has to be updated. I wonder if maybe in the later iterations, LightGBM is training much shallower trees (and therefore spending proportionally more time in those non-parallelized code paths).
num_leaves=255 does not guarantee that every tree with have 255 leaves.
LightGBM will stop growing a particular tree under a few conditions:
- no remaining splits which provide gain
>= min_gain_to_split - no remaining splits which satisfy
min_data_in_leaformin_sum_hessian_in_leaf - no remaining splits which also satisfy
interaction_constraintsormonotone_constraints - any of the above, limited by
max_depth
Unrelated, some notes on those parameters:
# this is the default, you can omit this
"tree_learner": "serial"
# these are only relevant for the CLI, omit them when using the Python package
"task": "train"
"is_training_metric": "false"
HonLZL
I wanted test speed in python. I tried to replicate gpu(L40) and cpu(28cores) experiment with higgs. The following are the experimental results.
num_iterations(500): cuda(28s) version was slower than cpu(71s). num_iterations(5000): cuda(570s) version was slower than cpu(403s).
Within ten minutes, Volatile GPU-Util gradually decreased from 80% to within 10%
dataset and parameter settings from: https://github.com/microsoft/LightGBM/blob/master/docs/GPU-Tutorial.rst Dataset Preparation from https://github.com/guolinke/boosting_tree_benchmarks/blob/master/data/higgs2libsvm.py
Code:
cpu test
gpu test