Given how long and costly benchmarking runs are, and how potentially costly a mistake is to start the benchmark again withouth having edited the robot script to skip test cases already run and/or addding --skip-all in the right place in the shebang while debugging report generation, skipping the execution of a test case should be the default policy whenever output data from a previous run already exists.
Only the user should decide when to go nuclear on previous data.
Implementation considerations
If the output folder for a given test case already exist, log a warning, skip, and move on to the next.
Feature description
Given how long and costly benchmarking runs are, and how potentially costly a mistake is to start the benchmark again withouth having edited the robot script to skip test cases already run and/or addding --skip-all in the right place in the shebang while debugging report generation, skipping the execution of a test case should be the default policy whenever output data from a previous run already exists.
Only the user should decide when to go nuclear on previous data.
Implementation considerations
If the output folder for a given test case already exist, log a warning, skip, and move on to the next.