C3S_34e Meeting with Ouranos and CRIM: Apr 2021

[agstephens]

Proposed agenda - C3S_34e Meeting with Ouranos and CRIM: 14th April 2021

1. Minutes and actions from previous meeting

Previous minutes: https://github.com/roocs/34e-mngmt/issues/86

Previous actions

• [x] AS: set up a meeting with CRIM & Ouranos - EO and ML. Agenda: Letter of intent; AI collaborations/opportunities; Other info. @cehbrecht can involve people from DKRZ and GERICS to the AI meeting. Meeting was hosted by CRIM on these themes.

• [x] AS: push the idea to have an international 3-day sprint (running through the night across continents) for working on the Earth System Model STAC profile (https://github.com/NCAR/esm-collection-spec/issues/21). @cehbrecht can involve people from DKRZ working on intake and STAC. - Meeting was had with PANGEO people and STAC team (radiant-earth).

2. Project update: c3s_34e [Ag]

• current production service has been released (on 23rd March):
  • serving subset of CMIP6 to C3S service
  • including PROV records as JSON/PNG
  • visible at: https://cds.climate.copernicus.eu/cdsapp#!/dataset/projections-cmip6?tab=form
• Changes to our stack:
  • incorporated intake catalogs to act as our data inventory (used as a gatekeeper and to avoid re-reading files to extract metadata when selecting data)
  • various bug fixes and structural changes
  • inclusion of average_over_dims function/process
• C3S keen for us to use the apply_fixes approach for CMIP6 decadal runs (to fix the representation of multiple time axes in the datasets)
• some C3S projects are being extended for 3 months, no clear future project for WPS work

3. Project updates: relevant Ouranos/CRIM projects [David/Tom]

Ouranos:

CRIM:

4. Collaborations and news

• PCMDI has stopped its role with ESGF Compute Working Team (CWT)

• CEDA/DKRZ have agreed to act as co-chairs to the CWT. We have suggested that a suitable approach for the CWT would be:

  • Hold the meeting monthly
  • The standard meeting is focussed on immediate collaborations and a roadmap (in the short-term we would focus on the integration of the roocs stack with C4I within IS-ENES3, then we might pick up on C3S plans, and hopefully other projects in future).
  • Every third meeting (3-monthly), we propose a more general share-and-discuss meeting in which all groups are invited to present on their recent work in the general area of compute services/tools. We would look to combine this meeting with some existing meetings such as the quarterly Birdhouse meetings.
  • This approach would allow us to meet both needs of:
    1. prioritising a common goal/collaboration
    2. allowing wider information-sharing and discussion
  • We hope that these meetings can be combined with the quarterly birdhouse meetings.

• Still pending: Letter of intent; AI collaborations/opportunities; Others

5. Discussion:

• Regridding:
• Future functionality/plans?

6. Agree next steps and next meeting

• Next monthly meeting: Thursday 13th May, https://ukri.zoom.us/j/98246249580

[huard]

Ouranos:

• Released xESMF 0.5.3
• Released Finch 0.7.3
• Investigating performance related to xarray chunking when streaming from DAP (for averaging processes)

[agstephens]

Minutes - C3S_34e Meeting with Ouranos and CRIM: 14th April 2021

1. Minutes and actions from previous meeting

Previous minutes: https://github.com/roocs/34e-mngmt/issues/86

New actions

• [x] MS: Talk to MPI ocean modellers about grid issues.
• [ ] MS: Raise an xESMF issue to discuss haloes

Previous actions

2. Project update: c3s_34e [Ag]

AS:

• current production service has been released (on 23rd March):
  • serving subset of CMIP6 to C3S service
  • including PROV records as JSON/PNG
  • visible at: https://cds.climate.copernicus.eu/cdsapp#!/dataset/projections-cmip6?tab=form
• Changes to our stack:
  • incorporated intake catalogs to act as our data inventory (used as a gatekeeper and to avoid re-reading files to extract metadata when selecting data)
  • various bug fixes and structural changes
  • inclusion of average_over_dims function/process
• C3S keen for us to use the apply_fixes approach for CMIP6 decadal runs (to fix the representation of multiple time axes in the datasets)
• some C3S projects are being extended for 3 months, no clear future project for WPS work

TL: We should look to register the ESGF Compute API as a standard.

3. Project updates: relevant Ouranos/CRIM projects [David/Tom]

Ouranos:

• Released xESMF 0.5.3
• Released Finch 0.7.3
• Investigating performance related to xarray chunking when streaming from DAP (for averaging processes)
• Waiting on THREDDS for the ability to use Zarr
• Recently re-chunked ERA5 holdings.

CRIM:

• For the Climate Data Portal Project: formed a CMIP6 Task Force. Will require some changes to the software stack.
  • need to be able to explain to users which data/models to use; which pathways are appropriate? The scale of data is daunting.

4. Collaborations and news

• PCMDI has stopped its role with ESGF Compute Working Team (CWT)
• CEDA/DKRZ have agreed to act as co-chairs to the CWT. We have suggested that a suitable approach for the CWT would be:
  • Hold the meeting monthly
  • The standard meeting is focused on immediate collaborations and a roadmap (in the short term we would focus on the integration of the roocs stack with C4I within IS-ENES3, then we might pick up on C3S plans, and hopefully other projects in future).
  • Every third meeting (3-monthly), we propose a more general share-and-discuss meeting in which all groups are invited to present on their recent work in the general area of compute services/tools. We would look to combine this meeting with some existing meetings such as the quarterly Birdhouse meetings.
  • This approach would allow us to meet both needs of:
    1. prioritising a common goal/collaboration
    2. allowing wider information-sharing and discussion
  • We hope that these meetings can be combined with the quarterly birdhouse meetings.

5. Discussion:

• STAC discussions:

  • AS/DH: both noted that STAC might be too immature at the moment to meet our needs. Tracking it is important but neither of us has much effort available to contribute.
  • TL: Propose a strategy for STAC work:
  • Create a machine learning extension.
  • Need to make sure that STAC works with ESM data.
  • We can lobby the OGC to make ESM happen in STAC.

• Regridding:

  • MS: Regridding prototype in clisops is being built in a branch.
  • Building on xESMF. It should be able to handle grids that xESMF can handle.
  • MS has given Stefan feedback on adaptive masking.
  • Haloes: looking at pre-processing the data to find duplicate cells/rows/columns in the grid (known as haloes). Some datasets have partially duplicated columns/rows. Hard to decide how to deal with this. Planning to talk to modellers.
  • DH: ESMF might be "too smart" - i.e. it might convert everything to real-world coordinates so this might not be an issue.

• Future functionality/plans?

• [x] MS: Talk to MPI ocean modellers about grid issues.

• [ ] MS: Raise an xESMF issue to discuss haloes

6. Agree next steps and next meeting

• Next monthly meeting: Thursday 13th May, https://ukri.zoom.us/j/98246249580