Default behaviour when running `benchcab` consecutive times

[SeanBryan51]

After running benchcab, the source code, executables and inputs are generated in the working directory. When benchcab is run again in the working directory, what should the default behaviour be? Approaches could include:

• cleaning the existing files and then run benchcab
• do nothing or throw an error message
• conditionally clean existing files: e.g if revision in source code is outdated. Then run benchcab
• ...

[ccarouge]

I guess there are several use cases to consider. I think we shouldn't error. During code development, I think it is a valid workflow to run benchcab over and over again.

Running benchcab during development In this case, the user wants quick feedback. So we would want the re-run to re-use whatever we can. That probably means:

• keep trunk checkout and executable if branch up-to-date with remote
• dev. branch checkout but re-run the build (maybe?)
• clean the outputs and re-run the tasks

Keeping an executable is risky if the user wants to re-run with a different compiler for example. We don't have a way to check for that at the moment. Ideally, we need the config to be committed to git when running benchcab then we have a way to check if the config has changed for the re-run.

Running benchcab for submission Once someone is ready to get their code in the main CABLE version, we want them to run a clean benchcab. We can obviously just update the checkout branches and not checkout from scratch. The rest needs to be cleaned up.

[SeanBryan51]

Ideally, we need the config to be committed to git when running benchcab then we have a way to check if the config has changed for the re-run.

So if the config file has changed, should we run a clean benchcab? We could also cache the config file, and compare what changes were made and clean the working directory accordingly.

Also, committing the config file every time we want to make a change and run benchcab seems like it would pollute the commit history. What about doing something similar to what Make does to track when files have been modified?

Edit: we want to commit to git so that tests can be reproduced

[ccarouge]

Ideally, we want to clean the run/ directory and be smart to decide what needs updating under src/. But we don't want to introduce smart logic with svn when CABLE will be moving to git soon. For the moment, we'll do a clean of src/ as well and revise once everything is using git.

[bschroeter]

Hey team! Please add your planning poker estimate with Zenhub @ccarouge @SeanBryan51