Minutes - C3S_34e Meeting with Ouranos and CRIM: 20th Jan 2021

New actions

[ ] AS: set up a meeting with CRIM & Ouranos - EO and ML. Agenda: Letter of intent; AI collaborations/opportunities; Other info. @cehbrecht can involve people from DKRZ and GERICS to the AI meeting.
[ ] AS: push the idea to have an international 3-day sprint (running through the night across continents) for working on the Earth System Model STAC profile (https://github.com/NCAR/esm-collection-spec/issues/21). @cehbrecht can involve people from DKRZ working on intake and STAC.

1. Minutes and actions from previous meeting

Previous actions:

[X] AS: discuss with 34e team how we add support for static typing: roocs/clisops#67 - done
[X] AS: respond to Tom's message about a formal collaboration - done

2. Project update: c3s_34e [Ag]

close to production service
use of inventory files within our C3S service, used for:
- limiting access to QA'd data only
- returning original files if they are aligned with time subset
Changes to clisops/roocs_utils:
- Removing submodule; problems with per-file (get_file()) call; so using a single pytest.fixture to clone the test repo
Travis-CI replacement:
- some concern expressed about which service to use:
- GitHub actions: drop-in alternative - but is not necessarily (or always) free
- Internal GitLab (per institute), maybe with mirroring from GitHub - CEDA has some experience of this and can provide info if needed by Ouranos
Brexit impacts: changes required for some of our non-ESGF data services that serve C3S
IS-ENES3 interest: some interest in re-using our C3S WPS.

3. Project updates: relevant Ouranos/CRIM projects [David/Tom]

Ouranos:

Moving into catalog work:
- setting up an intake catalog for THREDDS content
  - would like to use STAC later
Pascal worked on a new clisops function to average over a polygon shape
- planning to integrate over next month
Travis has been working on PAVICS landing page and operational system.
Put in a PR for improved subsetting feature:
- using masking, and finding valid grid cells (e.g. on coastlines)
- will look into issue that Elle raised

CRIM:

keen to use and advance STAC usage
- primarily focussing on STAC extension for machine-learning
  - for training datasets, adaptation datasets
We will set up "Letter of Intent" with CEDA (see below)

4. CRIM/CEDA MOU - or letter of intent:

the way forward?
possible AI work?
[ ] AS: set up meeting with CRIM & Ouranos - EO and ML. Agenda: Letter of intent; AI collaborations/opportunities; Other info

5. Discussion:

Averaging process:
- all agreed that CEDA will implement a simple average_over_dims function
- we will also refactor clisops.core so that it has single functions that do one thing each - and then clisops.ops will join/chain them as required.
Regridding work:
- Martin demonstrated his latest investigations of adaptive masking with different regridding methods.
managing test datasets:
- we all agree that using the get_file() function was leading to HTTP GET failures due to github API limiting requests - we will simply use a single clone of the test data repo in testing
other activities:
- the rioxarray library is developing and is definitely worth tracking:
- https://corteva.github.io/rioxarray/stable/rioxarray.html

6. Agree next steps and next meeting

Set up a meeting about collaboration and AI
Next monthly meeting: Thursday 18th February, https://ukri.zoom.us/j/98246249580

roocs / 34e-mngmt