ENH: Scheduler fixes and enhancements with new scheduler integration tests for all supported schedulers

[cpelley]

• Added integration testing for the schedulers.
  • Testing 'single-threaded' (dask).
  • Testing 'processes' (dask).
  • Testing 'distributed' (dask).
  • Testing 'multiprocessing' (in-house).
  • Ray not yet tested as we don't have an environment with it in (TODO - not high priority).
• Fixed 'distributed' dask scheduler usage.
  • Switching from 'Node' as dask graph keys to strings (unique string names denoting node identities using dask tokenise). Dask when utilising the distributed scheduler, accepts only tuples, bytes, int, float and str (see dask.core.validate_key).
• Enabled passing data in memory between nodes for the in-house multiprocessing scheduler.
  • Handles data lifetime (automatically throwing it away when it is not needed anymore).

Issues

• Closes https://github.com/MetOffice/dagrunner/issues/10
• https://github.com/MetOffice/improver_suite/issues/2079
• https://github.com/MetOffice/improver_suite/issues/2253

Note

Just to raise awareness as the first dagrunner PR under review - since this branch makes changes that impact the reference documentation, an auto reference documentation build commit will be pushed accordingly thanks to the dagrunner action workflow.