Sprint 3: 17-21 Jan 2022

[backeb]

Sprint 3: 17-21 Jan 2022

Sprint objectives and tasks

Automate the running of WFLOW and the downloading of data

Workflow for WFLOW + pre/post-processing scripts

• [x] @JoostBuitink @sperna @avgils add error handling to download scripts

WFLOW Singularity container

@avgils developed a Singularity container WFLOW - a test run with ERA5 input data was succesful

• [ ] @avgils one improvement is needed because now WFLOW still needs to be installed in the container before every WFLOW run.
• [x] @JoostBuitink @sperna perform test if container works for seasonal forecast data

Protocols or clients used by ECMWF API

Refer to https://github.com/c-scale-community/use-case-high-res-land-surface-drought-analysis/issues/15#issuecomment-980083634

• [x] @oonkjbr check what protocols or clients are being used

Soil moisture data downloads from Copernicus Global Land Service

• [ ] @sperna Modify and test download notebooks available form VITO at https://notebooks.terrascope.be/hub/login
• [ ] @sperna verify which of the variable matches best with wflow variables (this is NB for assimilation)

Workflow orchestration

• [ ] @avgils investigate https://snakemake.readthedocs.io/ as a workflow orchestration option. Supports many scheduling systems https://snakemake.readthedocs.io/en/stable/tutorial/additional_features.html#cluster-execution

Objectives for next sprint

• [ ] Check OpenDA Docker container and deploy

[oonkjbr]

Apologies for not being able to attend on 17/1 at 9.30. Looking at my task above. In case the API being referred is this one: https://www.ecmwf.int/en/forecasts/access-forecasts/ecmwf-web-api then the protocol being used is WebDAV (HTTP(S)). It appears to rely on basic GET and POST, see https://github.com/ecmwf/ecmwf-api-client/blob/master/ecmwfapi/api.py

Not sure if this is of interest, depends on the amount of data to be downloaded, but there are some interesting comments regarding group (bulk) downloads and efficiency here: https://confluence.ecmwf.int/display/WEBAPI/Retrieval+efficiency

[oonkjbr]

@avgils it may be interesting to know whether container security (e.g., store creds in HashiCorp Vault) is still something you would be interested in. @avgils in case it turns out that snakemake is something you may want to pursue and need help with then let me know. I do not know it well myself, but i have some colleagues that may be able to answer questions.

[oonkjbr]

@backeb at the end of the sprint week or during post-sprint recap it may be nice if the LSDA could provide a brief overview of solution implemented to date. Knowing a bit more about the technical setup and usage would allow me to comment in a bit more detail.

[backeb]

Retrospective

Tops

• Thanks @Jaapel for contributions
• Good experience with technologies available on SURF

Tips

• More close contact with WFLOW development team at Deltares needed (official image for WFLOW needed before start of sprint)

Brief overview of solution implemented to date

• TUWien and EODC have updated SWI product and could make it available
• SEAS5 is an ensemble forecast with 50 members
• parts in blue completed, parts in grey under development
• ❓ what is the objective for the interactive figures?
• This workflow could in principle be easily redeployed on another SLURM cluster.
• [ ] @avgils @Jaapel fix Julia container next week 😺
• Re crontab
  • You need to be on the UI node where you installed your crontab in order for the environmental variables to be defined.
  • If we need to migrate UI nodes / VMs we will loose the crontab stuff
  • Workaround could be a dedicated UI node
  • scrontab being investigated by @oonkjbr

Progress updates

Workflow for WFLOW + pre/post-processing scripts

• [ ] @JoostBuitink @sperna @avgils improve error handling to download scripts

WFLOW Singularity container

• [ ] @avgils one improvement is needed because now WFLOW still needs to be installed in the container before every WFLOW run.
  • seems to work for @avgils but not for @Jaapel - if can't figure it out let @oonkjbr know

Protocols or clients used by ECMWF API

• No further action required

Soil moisture data downloads from Copernicus Global Land Service

• made progress, but not entirely there yet.
• Prep work to be done in prep for next sprint
  • [ ] Investigate which workflow (OpenEO or R) - @JoostBuitink @sperna (@Jaapel for OpenEO)
  • [ ] get stand alone downloads to work - @JoostBuitink @sperna

Workflow orchestration

• crontab/SLURM combo used instead - works fine for simple workflows.

Next sprint

Objectives

• improve error handling to download scripts - @JoostBuitink
• post processing scripts - @JoostBuitink @sperna
• fix wflow container and engage with wflow dev team to get more stable images (preference is to do this next week) - @avgils @Jaapel

Dates

31 Jan - 4 Feb

• [ ] fix wflow container and engage with wflow dev team to get more stable images on 27 Jan (@Jaapel @avgils with @sperna @JoostBuitink as tentatives)