Investigate slurm `--ntasks` and ESMValTool's `MAX_PARALLEL_TASKS` for RTW

[ehogan]

During the overview of the RTW at the workshop, @bouweandela suggested that the following line in, e.g. esmvaltool/utils/recipe_test_workflow/recipe_test_workflow/site/metoffice.cylc was incorrect:

            --ntasks = {{ MAX_PARALLEL_TASKS }}

It's true that I am perhaps confused about this (see #3569), so it would be great to understand what the slurm --ntasks means and what ESMValTool's MAX_PARALLEL_TASKS means.

[bouweandela]

While these two both have the word 'tasks' in them, they are unrelated.

• For the slurm command, you could e.g. have a look at the slurm docs this stack overflow post.
• max_parallel_tasks in ESMValCore means that this number of tasks (i.e. variables or diagnostic scripts in a recipe) will be processed in parallel using the Python multiprocessing library. More detailed explanation here: https://docs.esmvaltool.org/projects/ESMValCore/en/latest/recipe/overview.html#the-diagnostics-section-defines-tasks