search

aibasel / machetli

GNU General Public License v3.0
6 stars 3 forks source link

Issue72 #89

Closed ClemensBuechner closed 1 year ago

FlorianPommerening commented 1 year ago

I ran the code on out two test cases (toy-problem and error-test) both locally and on the grid. I also ran it on one of the real use cases (issue247_pddl_sas). Everything looked fine but the test cases do not completely cover all the corner cases we looked at. Maybe we should extend error-test a bit to specifically cause some of the errors we now catch.

I also noticed that we currently do not report the evaluator running out of resources correctly. There is an exit code for it, but it is not used. In such cases, I think the error would be treated as an unexplained error and the search would stop (treating it like any other evaluator crash). This was also the previous behavior (our stance was that it is the job of the evaluator script to enforce resource limits if it calls external processes). To correctly catch cases where the evaluator runs out of resources, we have to call it as an external process rather than importing a module and executing it inside the current Python context. This is a larger change that I suggest we leave for a separate issue.

FlorianPommerening commented 1 year ago

We decided to leave the handling of out-of-resources errors, moving the evaluator into an executable script, and restarting the search as follow-up issues.