Add CRUTS4.05 data to flamingo

[agstephens]

NOTES about gzipped CRU TS data:

1. xarray can read .nc.gz - so we don't need to gunzip or change anything!

Tasks:

• [x] AS: Extract WPS input info from Flamingo process module/utils:
  • [x] Move to ceda-wps-assets
  • [x] Make WPS dependent on ceda-wps-assets
• [x] Update the mini-ceda-archive to add: /badc/cru/data/cru_ts/cru_ts_4.05/data/wet/cru_ts4.05.1901.2020.wet.dat.nc.gz
• [ ] AS/FE to discuss procedure and document the process:
  • Doc URL: https://secure.helpscout.net/docs/59b25bca042863033a1caf91/article/5d9c7a1b04286364bc901df0/
  • Process is:
  • Check data can be read by Xarray
  • Check data can be read by clisops package
  • Create test file(s) in the mini-ceda-archive
  • Update datasets.yml file with information about new data
  • Write additional tests in flamingo/tests
  • Run on test server with test UI
  • Connect test server to production UI (for quick look) - then disconnect
  • Deploy on production WPS cluster
  • Connect to production UI
  • Test, test, test
• [x] AS: Put config content for each dataset in to ceda-wps-assets
• [x] AS/FE: Add tests for each new dataset.
• [x] FE/AS: Deploy on to test server:
  • [x] Can we run WPS on test server and connect to production UI for testing (temporarily) - only admin access?
• [ ] FE/AS: Deploy into production (using updated Ansible playbook - pulling info from ceda-wps-assets)
• [ ] Update MOLES records with a script to add process info: https://catalogue.ceda.ac.uk/uuid/c26a65020a5e4b80b20018f148556681

[agstephens]

NOTES ABOUT PROCESS:

1. Gunzipping uses lots of memory - we only have 4GB RAM on ceda-wps-staging - so testing fails with real data. Solution could be:
   • [x] Set up to use test data only (rather than /badc/). The playbook can do this by cloning to /tmp/mini-ceda-archive/... and updating the /etc/roocs.ini file from the template. Just need to change the base path in it.

[agstephens]

Related to: #13

[agstephens]

To check:

• [ ] /etc/roocs.ini should not use ~/.mini-ceda-archive/ on ceda-wps-staging.ceda.ac.uk
  • [ ] Also, cannot use /root/.mini-ceda-archive/ because the cwps user CANNOT read the root directory!
  • [ ] So, you need to either use the real archive (see memory error below), or, make sure the test data is visible at /tmp/mini-ceda-archive/.... so that it can be read. - only test with variable wet.
• [ ] ceda-wps-staging is hitting memory allocation errors (in Flamingo jobs) - so we are asking to double to RAM from 4GB to 8GB.
• [x] Edit requirements.txt to pywps==4.4
• [ ] Update the production service so that it allows: SubsetHadUKGrid to be run.
• [ ] This requires the roocs-utils module of fixed-path-modifiers.
  • [ ] Check that there are no breaking changes in fixed-path-modifiers branch and then merge it to Master, and add it to history file. Then change requirements to use latest from Github in recipe.

[agstephens]

Try to re-deploy, with HadUK-Grid switched off, on ceda-wps-staging.

NOTE: must use pywps==4.4 at the moment; 4.5 is broken on github.

[agstephens]

Combine roll-out with: https://github.com/cedadev/flamingo/issues/13 They both appear to work.