C3S_34e Meeting with Ouranos and CRIM: May 2021

[agstephens]

Proposed agenda - C3S_34e Meeting with Ouranos and CRIM: 27th May 2021

1. Minutes and actions from previous meeting

Previous minutes: https://github.com/roocs/34e-mngmt/issues/94

Previous actions

2. Project update: c3s_34e [Ag]

• recent developments:
  • C3S are pleased with our service - they say it is working!
  • almost at the end of first contract; expecting a renewal
  • more work required on CMIP5, CORDEX; average and regrid.
  • moved to intake catalogs as our file inventory
  • working on dynamic fixes in daops to modify Decadal CMIP6 runs
  • new usage process captures usage statistics (Demo)

3. Project updates: relevant Ouranos/CRIM projects [David/Tom]

Ouranos:

CRIM:

4. Collaborations and news

5. Discussion:

• ESGF Future Architecture progress
• STAC discussions/work
• Regridding:
• Future functionality/plans?

6. Agree next steps and next meeting

• Ag to move meetings back to 4-weekly from today - and edit meeting settings in Zoom
• Next monthly meeting: Thursday 24th June

[agstephens]

Minutes - C3S_34e Meeting with Ouranos and CRIM: 27th May 2021

1. Minutes and actions from previous meeting

Previous minutes: https://github.com/roocs/34e-mngmt/issues/94

New actions

• [ ] AS: Put regridding higher up the agenda next month
• [x] AS: move meetings back to 4-weekly from today - and edit meeting settings in Zoom
• [ ] ES: Create an issue on the xarray repository to explain the _FillValue in 1D coordinate variables, and that we believe these contravene the CF-Conventions (See "coordinate variable" section under: https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#terminology).
• [ ] MS: Raise an xESMF issue to discuss haloes

Previous actions

• [x] MS: Talk to MPI ocean modellers about grid issues.

2. Project update: c3s_34e [Ag]

• recent developments:

  • C3S are pleased with our service - they say it is working!
  • almost at the end of first contract; expecting a renewal
  • more work required on CMIP5, CORDEX; average and regrid.
  • moved to intake catalogs as our file inventory
  • working on dynamic fixes in daops to modify Decadal CMIP6 runs
  • found two "features" in xarray:
  • variables with "days" as units being decoded as numpy.timedelta64 array
    • fixed by: xr.open_dataset(..., decode_timedelta=False)
  • xarray writing _FillValue to 1D coordinate variables to NetCDF

• [ ] ES: Create an issue on the xarray repository to explain the _FillValue in 1D coord variables, and that we believe these contravene the CF-Conventions (See "coordinate variable" section under: https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#terminology).

  • new usage process captures usage statistics
  • CE demonstrated a Notebook that shows usage of the rook service
  • showed how we can capture statistics from logs for both services
  • data sources:
    • current: pywps database and nginx log files
    • future: provenance files

3. Project updates: relevant Ouranos/CRIM projects [David/Tom]

Ouranos

DH:

• Looking at getting cf_xarray to include provenance
  • Attended xarray dev meeting
  • Invited by CSIRO chap - setting up climate services in Australia
  • xarray would provide hooks for provenance tracking
  • e.g. you would have a callback function for gathering metadata about provenance

TL:

• Met with CERFACS - looking at rewriting ICCLIM (which was C++/python) to python/xarray
  • should hopefully converge with xclim
  • interested in integrating true metadata into outputs
• Demonstrated NcML THREDDS aggregations being used through a portal (DACCS):
  • adding a "realisation" aggregation
  • 430,000 files aggregated to a single dataset (1.6TB)
  • faster than doing a call to: xr.open_mfdataset(...lots of files...)
  • interesting solution for huge datasets like this
  • python scans the file system and then generates the NcML record
  • once loaded into THREDDS - it is quite fast when loading into xarray using dask chunks that are appropriate
  • dask will make one request to THREDDS per chunk

CRIM:

TL:

• Proposal for the OGC Disaster Pilot project:
  • shared the technical overview with the group
  • including CEDA as source for Sentinel data
  • ClimateEngine and Google keen to contribute:
  • ClimateEngine is working with Google to build climate indicators into Google EarthEngine
  • thanks to CE and AS for supporting this

4. Collaborations and news

• OGC climate meeting is coming up next month.

5. Discussion:

• ESGF Future Architecture progress

  • Development of Kubernetes solution

• STAC discussions/work

  • TL: we will have a mtg with the Canada Govt regarding using STAC for Machine Learning

• Regridding

  • MS: we are working on basic clisops regridding
  • MS: will open an issue on xESMF github regarding haloes

• [ ] MS: Raise an xESMF issue to discuss haloes

• [ ] AS: Put regridding higher up the agenda next month

6. Agree next steps and next meeting

• [x] AS: move meetings back to 4-weekly from today - and edit meeting settings in Zoom

• Next monthly meeting: Wed 30th June

[agstephens]

@sol1105 cannot make 24th June, so moved the next meeting back to Thurs 1st July.

[Zeitsperre]

I Just realized, but I don't think July 1st will work for Ouranos people (It's :canada: day!)

[agstephens]

@Zeitsperre @huard @tomLandry I'm moving the July 1st meeting to the day before (June 30th).