search

mljar / mljar-supervised

Python package for AutoML on Tabular Data with Feature Engineering, Hyper-Parameters Tuning, Explanations and Automatic Documentation
https://mljar.com
MIT License
3.06k stars 410 forks source link

LightGBM causes Optuna to go out of memory and crash #687

Open AkshayNovacene opened 11 months ago

AkshayNovacene commented 11 months ago

I was wondering if there was a way to batch the inputs when using Optuna or reduce the load on it.

Optuna seems to go out of memory and crash on google colab when I try to run it on my big dataset. This is observed only when using LightGBM, but other models seem to work fine.

pplonski commented 11 months ago

Hi @AkshayNovacene,

LightGBM has some issues with memory management https://github.com/microsoft/LightGBM/issues/4239

The solution would be to train each model in separate process, to always release full memory after training. I'm thinking about using Ray framework https://github.com/ray-project/ray for this (you can train each model in separate process and you can distribute training on multiple machines).

AkshayNovacene commented 11 months ago

Thanks for the prompt reply. I am still trying to dig into the issue because lightGBM works fine with Optuna when I use Optuna out of the box. I was using the lgb.LGBMClassifier component for training manually. It goes out of memory only when I use the optuna mode through the automl package and try to train Lightgbm.