backeb
commented
2 years ago Retrospective
Tops
- Lets have 🍻
- Lots of contact between @Jaapel and @JoostBuitink, good division of labour
- @Jaapel knowledge transfer
Tips
Progress on objectives
fix WFLOW container and engage with WFLOW dev team to get more stable images (@JoostBuitink @sperna to provide input if needed)
- got it working with WFLOW-CLI
Workflow for WFLOW and pre/post-processing scripts
-
improve error handling to download scripts
- in progress, but not a real problem
- API we're using doesn't throw a useful error
- [ ] Think about building a check inbetween the steps
-
work on post processing scripts = work in progress
- simple post-processing script to produce a plot
- good test for public folder connected to Apache Webserver (http-based) - all works well.
- started working on statistics for visualisations
Soil moisture data downloads from Copernicus Global Land Service
- Investigate which workflow (OpenEO or R)
- OpenEO API doesn't support the data we want to use -- this might be something to report on for WP4
- Managed to get it done via direct python download
objectives for next sprint
- [ ] Think about building a check inbetween the steps to improve error handling to download scripts
- [ ] continue on post processing scripts
- [ ] check on the status OpenDA and if it can run on Spider at SURF (share info with @oonkjbr)
- [ ] add performance metrics
- [ ] can the current workflow run automatically via crontab? - currently set up a test to run automatically on the 15th
progress meeting
18 Feb
Options for extending use case
- Planned to use EO soil moisture data for assimilation
- Extend use case to include data assimilation
- Offer as a service: redeploy workflow for a different region
- [ ] @backeb schedule a brainstorming session to figure out what is needed (recon budget for use case)
- could redeploy on TUWien HPC (Vienna Scientific Cluster; VSC) in a couple of months when they are ready
thinking about analytics services
- [ ] think about ideas
some options
- isolate generic parts of workflows and offer those
- offer the entire workflow to do drought forecasting for a different area - not sure about model calval step
- eScienceCentre working on STAC and dCache integration (https://github.com/NLeSC-GO-common-infrastructure/stac2dcache; https://github.com/NLeSC-GO-common-infrastructure/dcachefs)
other feedback
- some times there are issues related to access control lists in spider, when one person sets something up, another cannot always change it
- e.g. one crontab per user
- can submit helpdesk ticket to get support
- can consider tools for user based access - require a lot of work to set up
- public cloud has role based access which is what we're used to, but can find workarounds
other actions
- [ ] @all contribute to https://confluence.egi.eu/display/CSCALE/C-SCALE+Data+Logistics
- don't let yourself be influenced by the providers
- think beyond data - what do you expect from the whole ecosystem? what's less or more important for me as a user
oonkjbr
Sprint 3 1/2: 27 Jan
Objective
Sprint 4: 31 Jan - 4 Feb
Notes and communications from previous sprint can be found here: https://github.com/c-scale-community/use-case-high-res-land-surface-drought-analysis/issues/17
Objectives
Workflow for WFLOW and pre/post-processing scripts
Soil moisture data downloads from Copernicus Global Land Service