search

openml-labs / gama

An automated machine learning tool aimed to facilitate AutoML research.
https://openml-labs.github.io/gama/master/
Apache License 2.0
92 stars 29 forks source link

Evaluating individuals after deadline #190

Open leightonvg opened 1 year ago

leightonvg commented 1 year ago

I was running GAMA multiple times and sometimes it happens that the search phase does not end within the max_total_time. I have to interrupt GAMA to stop the search. The logs show the following behavior: image image

I notice that the errors are raised in gama.genetic_programming.operator_set. I suppose that within Gama._search_phase() it happens that gama.genetic_programming.compilers.scikitlearn.evaluate_individual is called beyond the deadline. image Causing timeout issues in gama.genetic_programming.compilers.scikitlearn.evaluate_individual because the timeout is negative. image

I am not sure whether this is the cause of the errors and the non-stopping search space execution, and why it happens exactly. But do note that in the logs the timeout values are negative, and seem to correspond to when I would have expected the deadline based on a 120 second run. Therefore I believe sometimes individuals are evaluated after the deadline, causing GAMA to go on and eventually crash with memory issues.

PGijsbers commented 1 year ago

Thanks for the report! I was under the impression that this issue had been resolved. The error is originally raised here.My memory is a bit fuzzy on this, but I think the deadline was a workaround because sometimes the stopit module would not stop runtime evaluation in time, so I added a thin layer of manual checks (the deadline stuff). Ideally I'd like to fix the core issue by either using a different module to accomplish the same thing as stopit in a more reliable manner, or figure out why stopit wasn't reliable in the first place. But a quick fix to the deadline logic to patch things over would also be OK for now. I am a bit busy right now, but if that sounds like something you'd like to work on I am open to PRs :)

simonprovost commented 7 months ago

@leightonvg Any advancements in this direction?