Batch timeseries analysis in slurm

[ledm]

Closes #118

It's looking like this script is working now.

This PR adds a slurm queue based batch parallel processing of single job timeseries tool.

It's got the following features:

• Loads from existing input_yaml files.

• Run with a single command

• Fully parallelises the slowest part of the analysis_comparison tool: the single job analysis.

• Won't submit the same jobID if it's already existing*

• However, if two analyses call the same jobID with different suites (ie one has bgc and one has physics), it will only run the first one.

Need to do:

• Documentation in README.md
• Is it @DrYool proof?

[valeriupredoi]

oh and also - this things really do deserve a test, not thru and thru with SLURM submission, but everything up to that. I can write the test when it's about ready :+1:

[ledm]

let's make sure the piping is done correctly; also, have you tried this in practice? We don't need any special env to pass to sbatch do we? Like any special environment variables

I've been using this for a few days and it works on jasmin. Your amendment to the subprocess also works too.

If batch_timeseries fails you get normal python errors. If it fails inside inside the sbatch script, then you get error messages in the places that we tell it to fail.

[valeriupredoi]

yeh that's how we want it to behave, so stdout can be piped to eg a file. Looks good, bud! Let me write a test for it!

[ledm]

I'm not ready to merge. Still need to add documentation & maybe get @DrYool to try it.

[ledm]

The next question I have:

• Do we want to make this the default behaviour? This would mean that we run this script from inside analysis_compare when the -s flag is absent instead of calling the analysis_timeseries command?

[ledm]

Basically, the process for adding a new job, input.yml:

1. analysis_compare -s -y input.yml: This generates the job download commands, which will run overnight. It also creates an html report, but it breaks if there's no data downloaded yet.
2. Wait overnight for data to download on mass.
3. batch_timeseries -y input.yml: This submits the job timeseries onto the processing nodes.
4. analysis_compare -s -y input.yml: This job creates the html report.
5. ./rsync_to_esmeval.sh: this copies to html to the web visible location on disk.

I suspect that these can be merged into fewer commands!

[valeriupredoi]

@ledm I added some test gubbins, make sure to pull or merge so there are no conflictseses

[ledm]

The logo in the README points towards a file on the main branch, but of course it's not available yet until this PR is merged.

[ledm]

Okay @valeriupredoi, I'm happy with this now.

[valeriupredoi]

all good by me too, bud! Go ahead and merge when you good 🍺