Open athewsey opened 5 months ago
Thanks for bringing this up @athewsey! What you said seems to make absolute sense. We will take another look at this decision, why it was done this way, and if there isn't one, we will update it soon.
Any update on this? Would be useful to be able to actually save evaluation results where we want
This is in our backlog. We are currently evaluating different proposals. We will update this soon.
Today the
save
parameter ofEvalAlgorithmInterface.evaluate()
is just a boolean.From hunting around I found that the output path is taken either from an environment variable or else a default under /tmp... And that it should also be overrideable by setting the obviously-supposed-to-be-private property
eval_algo._eval_results_path
.IMO it's harder and less obvious than it should be for a developer using the library to save results in a folder they want. It'd be much easier if we could support
eval_algo.evaluate(save="my/cool/folder")
and ideally automatically create the provided folder if it doesn't already exist?