search

mthorrell / gboost_module

Gradient Boosting Modules for pytorch
MIT License
8 stars 0 forks source link

Can catboost be added? #11

Closed mat-ej closed 2 days ago

mat-ej commented 6 days ago

Hi thanks for an amazing repo.

Can catboost be added in similar manner to lgbm and xgb ?

mthorrell commented 5 days ago

I'm sure it can be, though I'm less familiar with Catboost. The tricky parts are (a) tricking it into producing multiple outputs (b) caching predictions for speed (c) feeding gradients/hessians in the right way.

Maybe helpful maybe not -- I asked chatGPT to make a catboost equivalent version of xgbmodule. It gave me the code below. I'm not able to test this properly for a little bit, but do you want to give it a shot?

-- EDIT -- I deleted the chatgpt code because it didn't work.

mthorrell commented 3 days ago

@mat-ej , I gave it a shot, and it turns out it is more difficult than I thought.

There is a branch that has this enabled and works on some basic problems I considered here: https://github.com/mthorrell/gboost_module/pull/16 (EDIT: note that this branch was merged. More info below).

BUT you will find this branch to be much slower than CatBoost in general because the python interface for catboost doesn't seem to support points (b) and (c) I mentioned in my previous comment. I opened an issue in that project: https://github.com/catboost/catboost/issues/2704

mthorrell commented 3 days ago

@mat-ej , I added catboost support here gboost_module/experimental//catmodule.py in #16

it's in the experimental folder for the speed reasons I mentioned above. You can see a basic usage of CatModule in its test file: https://github.com/mthorrell/gboost_module/blob/main/gboost_module/tests/experimental/test_catmodule.py

Give it a shot and let me know what you think. Once I get a bit more info on caching for catboost, I'll likely close this issue and open a separate issue that focuses on speed.

mthorrell commented 2 days ago

I'm closing this ticket in favor of #19 . If you give it a whirl, let me know how it goes!

mat-ej commented 23 hours ago

Wow, that was so fast I could not even react, amazing work. Will try this week.

edit: I quickly checked the experimental code and yeah, I also could not get around their calc_multi_ders seemingly outputing a signle target for the multi output regression, I discussed this in catb. repo some time ago.

https://github.com/catboost/catboost/issues/2615#issuecomment-2018320036