search

c-scale-community / workflow-coastal-hydrowaq

Porting and deploying the HiSea use case on C-SCALE
Apache License 2.0
3 stars 1 forks source link

readme

C-SCALE Workflow Solution: Coastal hydrodynamic and water quality modelling using Delft3D FM

C-SCALE workflow solutions are intended to give users a template and reusable components to start building monitoring, modelling and forecasting applications

With the coastal hydrodynamic and water quality modelling workflow solution a user can easily produce hydrodynamic and water quality hindcasts or forecasts for the coastal ocean for a Delft3D FM model schematisation.

The workflow solution has the following functionality

  1. Download the necessary input data for the user's Delft3D Flexible Mesh model setup. Input data include Copernicus' Global Ocean Physics Reanalysis and Global ocean biogeochemistry hindcast, ERA5 and FES2012.
  2. Prepare the data for ingestion into the user's Delft 3D Flexible Mesh hydrodynamic and water quality model. This entails the preparation of forcings, initial conditions, and boundary condiditons.
  3. Produce hydrodynamic and water quality hindcasts or forcasts based on the user's Delft3D Flexible Mesh hydrodynamic and water quality model setups.
  4. Analyse the simulation outputs in an interactive Jupyter Notebook.

The workflow is executed using Snakemake, a workflow management system for creating and executing reproducible and scalable data analyses. The workflow solution can be deployed on any provider part of the C-SCALE data and compute federation offering cloud container compute.

The workflow components and flow is summarised in the below schematic: workflow

Prerequisites

  1. Github account. Sign up here.
  2. DockerHub account and access to https://hub.docker.com/repository/docker/deltares/delft3dfm (contact software.support@deltares.nl to arrange access)
  3. CMEMS account, which can be obtained by registering at https://resources.marine.copernicus.eu/registration-form.
  4. CDS API key. Follow instructions at https://cds.climate.copernicus.eu/api-how-to#install-the-cds-api-key to generate the key.
  5. Docker needs to be installed on your computing environment.
  6. Snakemake needs to be installed on your computing environment.

Instructions

1. Build the Docker containers

Instructions on how to build and run the Docker conterainers for each workflow component can be found in the README.md files of corresponding folders of this repo:

  1. download
  2. preprocessing
  3. fm_model
  4. notebooks

2. Install Snakemake

To install Snakemake do:

  1. conda install -n base -c conda-forge mamba
  2. mamba install -c conda-forge -c bioconda snakemake click

3. Run the workflow for the example fm_model included in this repo

  1. In your CLI, cd to use-case-hisea/.
  2. Open use-case-hisea/workflow/config.yml in your preferred text editor and specify \ a. dependencies \ b. data folder locations \ c. delft3dfm folders and files \ d. area of interest (this should cover the area of your delft3dfm model schematisation) \ e. run mode
  3. Execute the workflow from the use-case-hisea/ directory by doing: snakemake --cores 4 all
  4. To run each rule individually, from the use-case-hisea/ directory do: snakemake -R <insert rule name> --cores 1

Additional instructions