Configurable Post-Processing

[TimothyWillard]

Could be useful to have configurable post-processing. A potential way this could work is given below, it allows for both arbitrary command execution and "special" commands for common tasks, but open to suggestions. Basically would create a bash script that would then be executed at the end of a flepiMoP run. name would be a human readable name for logging purposes and cmd is the command. If cmd is a list construct the arbitrary command and execute that, but if cmd is a string lookup whatever the "special" command is.

postprocessing:
- name: Foobar
  cmd:
  - "/usr/bin/python"
  - "$FLEPI_PATH/postprocessing/reformat_foobar.py"
  - "--arg1"
  - "'abc'"
  - "--arg2"
  - 12345
- name: Plot Predictions
  cmd: plot_predictions
- name: AWS S3 Upload
  cmd: aws_upload

Current post-processing scripts:

1. https://github.com/HopkinsIDD/flepiMoP/blob/main/batch/postprocessing-scripts.sh
2. https://github.com/HopkinsIDD/flepiMoP/blob/main/batch/SLURM_postprocess_runner.run

Related to GH-79?

[saraloo]

Thanks Tim, I like this idea. We've talked about it before but as always have had no time to think about implementing it.

I think the scripts themselves would have to be rethought a bit to write them in a more optimised way. Currently the main thing I've found is that jobs will time out or run out of memory for a lot of automatic runs if we try to do anything too complicated.