Let us assume I run a tuning A and the results are written to './results'. When I change my mind and want to run a different tuning B and I do not specify a different output folder the new outputs of B are also written to './results'. This can easily happen when I use a script for A, adapt it for B, and forget to change the output directory (or when I use the default of DeepOBS). The problem this implies is:
The outputs of A and B are both in the './results' path and further analyses (e.g. getting the best hyperparameter setting with the analyzer) are performed on all the runs. If I am only interested in B, but results of A are included I may end up with the best setting of A.
Proposed solution:
I know that we could simply expect the user to be smart enough to delete the results of A first (or to change the output directory of B), but if the users forgets to do so it can be really a mess. Therefore, I suggest to implement a sanity check in the tuner that prompts or warns the user when tuning is run on an already existing output directory. As far as I know the rerunning of the best setting is based on the runner class and not the tuner class. Therefore, rerunning the best setting would not prompt the user (which is fine).
However, just see this as an idea for improvement. The exact design may differ from my suggestion.
Let us assume I run a tuning A and the results are written to './results'. When I change my mind and want to run a different tuning B and I do not specify a different output folder the new outputs of B are also written to './results'. This can easily happen when I use a script for A, adapt it for B, and forget to change the output directory (or when I use the default of DeepOBS). The problem this implies is:
The outputs of A and B are both in the './results' path and further analyses (e.g. getting the best hyperparameter setting with the analyzer) are performed on all the runs. If I am only interested in B, but results of A are included I may end up with the best setting of A.
Proposed solution: I know that we could simply expect the user to be smart enough to delete the results of A first (or to change the output directory of B), but if the users forgets to do so it can be really a mess. Therefore, I suggest to implement a sanity check in the tuner that prompts or warns the user when tuning is run on an already existing output directory. As far as I know the rerunning of the best setting is based on the runner class and not the tuner class. Therefore, rerunning the best setting would not prompt the user (which is fine).
However, just see this as an idea for improvement. The exact design may differ from my suggestion.