I think this would make sense. We could even make this interactive (in addition, not instead), so users don't have to remember complicated command lines:
$ machetli pddl
Please enter the filepath to a PDDL domain file
> /benchmarks/bugged_domain.pddl
Please enter the filepath to a PDDL problem file
> /benchmarks/bugged_problem.pddl
How can I recognize the behavior you are interested in?
(1) string in the output (exception, error message, etc.)
(2) exit code
(3) different value than reference planner
(4) something else
> 1
Where is the string printed?
(1) stdout
(2) stderr
(3) file created by the planner
> 2
What string should I look for?
> Chocolate generation out of memory
OK, starting search
...
I found a smaller task and wrote it to output_domain.pddl/output_problem.pddl
I think this would make sense. We could even make this interactive (in addition, not instead), so users don't have to remember complicated command lines: