aibasel / machetli

GNU General Public License v3.0
6 stars 3 forks source link

Create eval directory and batch directories with executable evaluator script in all Environments #90

Open FlorianPommerening opened 1 year ago

FlorianPommerening commented 1 year ago

Currently, only the SlurmEnvironment writes states to disk. We want to unify this and use this setup for all cases (#53). As part of this change, we also want to turn the evaluator script into an executable script (#82, this also fixes #79). The evaluation can then use resource limits and react correctly to the evaluator running out of resources (left over from #72). With one run_dir for each run, it is also easier to keep track and report errors that should be investigated but do not terminate the search (e.g., a critical error in an evaluator). The plan is to leave all run directories on the disk after the search and logging behavior becomes easier (related to #57, #77).