search

microsoft / LightGBM

A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.
https://lightgbm.readthedocs.io/en/latest/
MIT License
16.53k stars 3.82k forks source link

Why does changing top_rate and other_rate not affect training accuracy at all? #2998

Closed chenzhiliang94 closed 4 years ago

chenzhiliang94 commented 4 years ago

This is not a bug, but rather a really weird observation with respect to the parameters top_rate and other_rate. AFAIK, they are the key ideas behind GOSS of lightgbm.

Error message

There is no difference in training accuracy when I change the top_rate and other_rate for the model. Even for really varied changes (0.05 -> 0.5). When I say "no difference", I mean exactly the same accuracy (I checked, there is no error in my code)

Reproducible examples

I change the parameters like this: param["top_rate"] = 0.5

model = lg.train(param, lg_data)

param["top_rate"] = 0.05

model = lg.train(param, lg_data) image

Steps to reproduce

  1. Just change top_rate in params; observe no change in training/validation accuracy (numerically identical)
guolinke commented 4 years ago

What is your learning rate and num_iteration? return to https://github.com/microsoft/LightGBM/blob/9843506e63c546b989df16e6235261fd38ce247d/src/boosting/goss.hpp#L129, the goss is disabled at the beginning.

chenzhiliang94 commented 4 years ago

Thank you! This is a direct answer to my question. When I increased number of iteration higher, the accuracy changes.