[Algoa Bay Forecast] Improve and harden model job pipeline

[zachsa]

Allow configurable start date Triggering the Algoa Bay Forecast model run manually, you can specify a run date. this run date is captured by the Actions runner and made available to steps via ${{ needs.envs.outputs.MODEL_RUN_DATE }}. But this value needs to be passed to all scripts that use a run date, which may require updating MatLab/Python code

• [x] When the model is compiled, is the current date used at all?
• [x] Downloading boundary data (forcing inputs). The run date should be passed via a flag to the python script --rundate 20220701 for example
• [x] The motuclient should retry on error (mercator download)
• [x] The make-forcings (MatLab) step
• [x] The run_croco.bash script probably needs to be updated too
• [x] Post processing (the output files are named for the run time date i think)

Retry download boundary data step on curl and motuclient errors The job is configured to retry on failure, however errors in the download don't result in failing the job since these errors are swallowed by the Python script where curl and motuclient are called.

• [x] Python script for downloading data needs to throw errors correctly

[zachsa]

Here is the error that Python is swallowing

[zachsa]

Here is the error that Python is swallowing

These warnings are okay:

[zachsa]

The motuclient fails without restarting still