jameslamb
commented
1 month ago Thanks for using LightGBM and taking the time to open this.
I'm -1 on adding this to LightGBM. I understand why this might be useful, but I don't think LightGBM is the right place for this logic. This would introduce some non-trivial maintenance burden and complexity.
This would be better handled outside of LightGBM, in code that you use to invoke it.
Since you mentioned .fit(), I assume you're specifically talking about using lightgbm (the Python package for LightGBM). You could, for example, use asyncio's builtin support for timing out Python function calls: https://docs.python.org/3/library/asyncio-task.html#timeouts.
Alternatively, you could use a lightgbm callback for this purpose. Something like the following:
import lightgbm as lgb
from datetime import datetime
from sklearn.datasets import make_regression
X, y = make_regression(n_samples=10_000, n_features=20)
dtrain = lgb.Dataset(X, label=y)
class TimeoutCallback:
def __init__(self, timeout_seconds: int):
self.before_iteration = False
self.timeout_seconds = timeout_seconds
self._start = datetime.utcnow()
def __call__(self, *args, **kwargs) -> None:
if (datetime.utcnow() - self._start).total_seconds() > self.timeout_seconds:
raise RuntimeError(f"timing out: elapsed time has exceeded {self.timeout_seconds} seconds")
bst = lgb.train(
params={
"objective": "regression",
"num_leaves": 100
},
train_set=dtrain,
num_boost_round=1000,
callbacks=[TimeoutCallback(2)]
)I just tested that with LightGBM 4.5.0 and saw the following:
[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.001736 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 5100
[LightGBM] [Info] Number of data points in the train set: 10000, number of used features: 20
[LightGBM] [Info] Start training from score 0.256686
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/jlamb/miniforge3/envs/lgb-dev/lib/python3.11/site-packages/lightgbm/engine.py", line 317, in train
cb(
File "<stdin>", line 8, in __call__
RuntimeError: timing out: elapsed time has exceeded 2 secondsThat's not perfect, as it only runs after each iteration and individual iterations could run for much longer on a realistic dataset. But hopefully that imperfection also shows one example of how complex this would be to implement in LightGBM.
I'm only one vote here though, maybe other maintainers will have a different perspective.
fingoldo
Adding the timeout parameter to the .fit() method, that should force the library to return best known solution found so far as soon as provided number of seconds since the start of training are passed, will allow to satisfy training SLAs, when a user has only a limited time budget to finish certain model training. Also, this will make possible fair comparison of different hyperparameters.
Reaching the timeout should have the same effect as reaching max iterations, maybe with additional warning and/or attribute set so that the training job's finishing reason is clear to the end user.