Open 111qqz opened 1 year ago
Interesting, not a problem that I have encountered before. Your settings look fine. Some ideas / questions:
finline=False
, that shouldn't really change anything but compilation will finish much faster.@siboehm 1. Yes. OOM during compilation. I have tried setting finline=False
, and the compilation can end normally without OOM.
I'm still thinking about this but here are my current conclusions:
finline=True
, lleaves compiles the whole forest into a single function. This would be roughly like writing a single C function that is 30MB in filesize, which is obviously enormous. The optimization passes in the compiler don't expect the functions to be this large, and some passes may have mem requirements that are non-linear in function size, which would explain your OOM problems. Disabling inlining via fininline=False
solves this problem as lleaves now generates 1000 functions (one for each tree in the forest). Disabling inlining does slow down prediction quite a bit if the trees are small (yours aren't) due to function call overhead and smaller optimization scope, but you should do your own benchmarking as this may not be a big problem in your case.So my advice would be:
If it is true that the OOM error occurs on the line that I mentioned in Conclusion (3), then I'd consider this an edgecase of lleaves usage (your model.txt is enormous) and not something I'll mitigate in the near future.
@siboehm Thank you very much for your detailed suggestions, I will try what you said later and update the progress here
Just a note for myself: The way to fix this problem would be to split the model internally into functions (blocks of trees iteration over data), to ensure that every call to lleaves contains exactly N function calls. Then I can disable inlining for these N functions, meaning they're compiled independently (=much less memory demand than one large function), and the cost is still reasonable (a few function calls per predict
invocation is super cheap).
I have similar memory issues. I'm compiling forests with 500 trees and 250 leaves per tree on a 32GB machine. The compilation time is way too long and I run out of memory.
1) Setting finline=False worked for me. But it made things so slow that it was actually slower than the LGBM out of the box inference. 2) Your idea to batch is a great idea. I was about to open an issue suggesting this and then I read this thread :). 3) Is there a way to have n_jobs for compiling? If you don't inline or if you batch the inlining then you're describing different independent compilations. I have machines with 100-200 cores, but lleaves can only use one of them at a time.
Thanks for your great work. I want to try it out, but the memory consumption is staggering. I can't use it on a machine with 32G memory, out of memory will occur Is there something I haven't set right?