This is the code repository that accompanies the paper "Community Workflows to Advance Reproducibility in Hydrologic Modeling: Separating model-agnostic and model-specific configuration steps in applications of large-domain hydrologic models" (Knoben et al., 2022).
Despite the proliferation of computer-based research on hydrology and water resources, such research is typically poorly reproducible. Published studies have low reproducibility because of both incomplete availability of the digital artifacts of research and a lack of documentation on workflow processes. This leads to a lack of transparency and efficiency because existing code can neither be checked nor re-used. Given the high-level commonalities between existing process-based hydrological models in terms of their input data and required pre-processing steps, more open sharing of code can lead to large efficiency gains for the modeling community.
Here we present a model configuration workflow that provides full reproducibility of the resulting model instantiation in a way that separates the model-agnostic preprocessing of specific datasets from the model-specific requirements that specific models impose on their input files. This workflow is applied to the Structure for Unifying Multiple Modeling Alternatives (SUMMA, Clark et al., 2015a,b) and mizuRoute (Mizukami et al., 2016), to create a model configuration that provides process-based hydrologic simulations and vector-based streamflow routing capabilities. The workflow uses open-source data with global coverage to determine model parameters and forcing, thus enabling transparent and efficient hydrologic science.
The code in this repository is the outcome of stepwise modification of an existing SUMMA instantiation developed by Andy Wood and colleagues at NCAR. This existing setup served as a testbed for consecutive changes to model input data, domain discretization and domain size, resulting in globally applicable model setup code that separates model-agnostic and model-specific configuration tasks.
All separate README files in this repository have been collated here: https://cwarhm.readthedocs.io/en/latest/.
This readme covers the following:
Use of this workflow requires accounts with various data providers. Login details with these providers are stored as plain text in the user's home directory. It is therefore strongly recommended that you choose unique, new passwords for these accounts. Using the same passwords you use elsewhere poses a security risk.
This repository and the manuscript it accompanies (Knoben et al., 2022) contain and describe a workflow used to configure a single set of hydrologic and routing models for multiple spatial domains. This is seen as the first step on a possible route to a community modeling culture. Later steps envision modification of this initial workflow to configure different experiments or models. This workflow is intentionally generalized and modular, so that such modifications of its code are possible.
The workflow uses the Structure for Unifying Multiple Modeling Alternatives (SUMMA, Clark et al., 2015a,b) hydrologic model and mizuRoute (Mizukami et al., 2016) routing model.
The workflow uses the following data sources:
The workflow can thus generate model setups with global coverage and for the past half century.
Currently, the workflow scope deliberately excludes spatial discretization and parameter estimation.
Four things are needed to execute the workflow code in this repository: the correct computational environment, an ESRI shapefile containing the modeling domain, a so-called workflow control file, and sufficient disk space.
The model configuration code is written in a combination of Bash and Python and requires various libraries and Python packages. Here we specify exact versions of each, as they were used to run the three modeling test cases presented in Knoben et al. (2022).
Libraries needed for required Python packages:
proj 7.0.1 https://proj.org/
geos 3.8.1 https://libgeos.org/
gdal 3.0.4 https://gdal.org/ Note: also used as a command line utility by certain scripts
libspatialindex 1.8.5 https://libspatialindex.org/
qgis 3.16.01 https://www.qgis.org/ Note: see Getting Started section for installation detailsPython packages are listed in the environment.yml and requirements.txt files that can be found in the root folder of this repository. See the Getting started section for instructions on how to use these with conda and pip, respectively.
Fortran libraries, needed to compile SUMMA and mizuRoute:
gcc 7.3.0 https://gcc.gnu.org/
openblas 0.3.4 https://www.openblas.net/
netcdf-fortran 4.4.4 https://www.unidata.ucar.edu/software/netcdf/fortran/docs/Optional utilities:
GNU Parallel 20180122 https://www.gnu.org/software/parallel/
netCDF Operators 4.9.5 http://nco.sourceforge.net/
Climate Data Operators (cdo) 1.9.8 https://code.mpimet.mpg.de/projects/cdoInstallation of Python packages is typically done through so-called package managers. Common ones are pip and conda. This repository contains configuration files usable by either package manager, called requirements.txt and environment.yml respectively.
All test cases described in Knoben et al. (2022) were generated using the library versions specified above, and the Python package versions specified in requirements.txt. Library and package versions in environment.yml differ slightly and were used during code development but not to generate any final results.
The scope of our workflow implementation assumes that the user has access to a basin discretization stored as an ESRI shapefile that defines the area of interest as discrete modeling elements (e.g., grid cells, sub-basins). Specifically, the workflow expects:
These shapefiles should include certain mandatory elements. The folder 0_example contains a detailed description of shapefile requirements. It also contains example shapefiles that can be used to create a model setup for the Bow River at Banff, Canada. See the Test cases section for instructions on how to replicate the Bow at Banff model configuration.
Users interact with the workflow through so-called control files that contain certain high-level decisions about the model configuration the workflow will generate. The repository contains examples of these in the folder 0_control_files. Instructions can be found in the Getting started section.
Disk space requirements are largely dependent on the size of the modeling domain (in time and space) and the number of output variables saved by SUMMA. Minimum requirements for the Bow at Banff example are as follows:
Note that data generated in intermediate steps in the workflow is saved in corresponding directories. Users may wish to manually delete these intermediate results if disk space is an issue.
This section provides brief details about the model configuration test cases shown in Knoben et al. (2022).
All control files can be found in the folder 0_control_files.
This first test case simulates hydrologic processes across planet Earth to illustrate the large-domain applicability of our approach. The global domain (excluding Greenland and Antarctica) is divided into 2,939,385 sub-basins or Grouped Response Units (GRUs; median GRU size is 36 km2; mean size is 45 km2) derived from the global MERIT basins data set (Lin et al., 2019). Simulations are run for a single month (1979-01-01 to 1979-01-31) at a 15-minute temporal resolution.
control_Africa.txt, control_Europe.txt, control_NorthAmerica.txt, control_NorthAsia.txt, control_Oceania.txt, control_SouthAmerica.txt, control_SouthAsia.txtThis second test case uses 40 years of hourly forcing data to simulate hydrologic processes over the North American continent and illustrates the combined large-domain and multi-decadal applicability of our approach. The continental domain is divided into 517,315 sub-basins (median GRU size is 33 km2; mean size is 40 km2) derived from the global MERIT basins data set (Lin et al., 2019). Simulations are run from 1979-01-01 to 2019-12-31 at a 15-minute temporal resolution.
control_NorthAmericaThis third test case uses 5 years of hourly forcing data to simulate hydrologic 436 processes over the Bow at Banff catchment located in Alberta, Canada and illustrates the ability of our workflow to deal with complex spatial discretization shapefiles. The local domain is divided into 51 sub-basins (median GRU size is 32 km2; mean size is 43 km2) derived from the global MERIT basins data set (Lin et al., 2019). These sub-basins are further discretized into Hydrological Response Units (HRUs), based on 500 meter elevation increments, resulting in 118 HRUs (median HRU size is 11 km2; mean size is 19 km2). Simulations are run from 2008-01-01 to 2013-12-31 at a 15-minute temporal resolution.
control_Bow_at_Banff.txt0_example/shapefilesThe workflow is organized around the idea that the code that generates data (i.e. the scripts that form this repo) is kept in a separate directory from the data that is downloaded and created. The connection between repository scripts and data directory is given in the control_file as control setting root_path. We strongly recommend to not put the data directory specified in root_path inside any of the repository folders, but to use a dedicated and separate location for the data instead. Note that the size requirement of the data directory depends on the size of the domain and the length and number of simulations.
A typical application would look as follows:
summaWorkflow_public/0_control_files. Copy and rename control_BowAtBanff.txt to something more descriptive of your modeling domain.root_path;summaWorkflow_public/1_folderPrep and run the notebook or Python code there to create the basic layout of your data directory..shp) into the newly created your/data/path/domain_[yourDomain]/shapefiles folder, placing the shapefiles in the catchment and river_network folders respectively.Doing so will generate a basic SUMMA and mizuRoute setup that contains the following:
This workflow requires the user to provide the catchment and river network shapefiles with certain required contents (see the relevant readme's for details). The scripts in the repository provide all the necessary code to download and pre-process forcing and parameter data, create SUMMA's and mizuRoute's required input files, and run hydrologic and routing simulations. This generates a basic SUMMA + mizuRoute setup upon which the user can improve by, for example, swapping global datasets for higher quality local ones or connecting the model setup to a calibration algorithm.
To assist in understanding the process described above, example shapefiles and a control file for the Bow river at Banff, AB, Canada, are provided as part of this repository. Shapefiles can be found in the folder 0_example. The control file can be found in 0_control_files. We strongly recommend to first use the provided shapefiles and control file to create your own setup for the Bow river at Banff. This domain is relatively small and the control file only specifies 1 year of data, which limits the download requirements. Instructions:
root_path in the file control_BowAtBanff.txt to point to your desired data directory location;./1_folder_prep. This creates a basic folder structure in your specified data directory../0_examples/shapefiles folder in this repo into the newly generated basic folder structure in your data directory. The remaining scripts in the workflow will look for the shapefiles there.The workflow uses a combination of Python and Bash. This section lists how to setup your system to use this workflow. We recommend you contact your system administrator if none of this makes sense. Note that this section is a work in progress.
The Bash code requires various libraries and command line utilities. See the Requirements section above.
The Python code requires various packages, which may be installed through either pip or conda. Please note that while conda automatically installs the necessary underlying libraries for a given package, pip does not. The user must take care to have local installs of the required libraries if using pip. See the Requirements section above. It is typically good practice to create a clean (virtual) environment and install the required packages through a package manager. The workflow was developed on Python 3.7.7. and successfully tested on Python 3.8.8.
Pip:
Package requirements specified in requirements.txt. Assumes a local install of various libraries is available. Basic instructions to create a new virtual environment:
cd /path/to/CWARHM
virtualenv cwarhm-env
source cwarhm-env/bin/activate
pip install -r requirements.txtConda:
Package requirements specified in environment.yml. Basic instructions to create a new virtual environment:
cd /path/to/CWARHM
conda env create -f environment.yml
conda activate cwarhm-envIf cwarhm-env is not automatically added as a Jupyter notebook kernel, close the notebook, run the following from a terminal and restart the notebook:
python -m ipykernel install --name cwarhm-envThe code used for geospatial data remapping use several functions from QGIS. This code can be found in folder /CWARHM/4b_remapping/1_topo/. Depending on your system, you may be able to get QGIS as a Conda package (https://anaconda.org/conda-forge/qgis) or require a stand-alone install of QGIS (https://qgis.org/en/site/). The provided notebooks (.ipynb) are designed to use QGIS as a Conda package; the Python scripts (.py) in that folder show how to use a standalone install. The folder also contains a more detailed description of QGIS setup.
This workflow (“the program”) is licensed under the GNU GPL v3.0 license. You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/. Please take note of the following: This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. In practical terms, this means that:
Our thanks to those who have contributed to improving this repository (in order of first reports):
Benham, E., Ahrens, R. J., & Nettleton, W. D. (2009). Clarification of Soil Texture Class Boundaries. United States Department of Agriculture. https://www.nrcs.usda.gov/wps/portal/nrcs/detail/ks/soils/?cid=nrcs142p2_033171
Clark, M. P., B. Nijssen, J. D. Lundquist, D. Kavetski, D. E. Rupp, R. A. Woods, J. E. Freer, E. D. Gutmann, A. W. Wood, L. D. Brekke, J. R. Arnold, D. J. Gochis, R. M. Rasmussen, 2015a: A unified approach for process-based hydrologic modeling: Part 1. Modeling concept. Water Resources Research, doi:10.1002/2015WR017198
Clark, M. P., B. Nijssen, J. D. Lundquist, D. Kavetski, D. E. Rupp, R. A. Woods, J. E. Freer, E. D. Gutmann, A. W. Wood, D. J. Gochis, R. M. Rasmussen, D. G. Tarboton, V. Mahat, G. N. Flerchinger, D. G. Marks, 2015b: A unified approach for process-based hydrologic modeling: Part 2. Model implementation and case studies. Water Resources Research, doi:10.1002/2015WR017200
Clark, M. P., B. Nijssen, J. D. Lundquist, D. Kavetski, D. E. Rupp, R. A. Woods, J. E. Freer, E. D. Gutmann, A. W. Wood, L. D. Brekke, J. R. Arnold, D. J. Gochis, R. M. Rasmussen, D. G. Tarboton, V. Mahat, G. N. Flerchinger, D. G. Marks, 2015c: The structure for unifying multiple modeling alternatives (SUMMA), Version 1.0: Technical Description. NCAR Technical Note NCAR/TN-514+STR, 50 pp., doi:10.5065/D6WQ01TD
Copernicus Climate Change Service (C3S) (2017): ERA5: Fifth generation of ECMWF atmospheric reanalyses of the global climate. Copernicus Climate Change Service Climate Data Store (CDS), 2020-03-26. https://cds.climate.copernicus.eu/cdsapp#!/home
Friedl, M., Sulla-Menashe, D. (2019). MCD12Q1 MODIS/Terra+Aqua Land Cover Type Yearly L3 Global 500m SIN Grid V006 [Data set]. NASA EOSDIS Land Processes DAAC. Accessed 2020-05-20 from https://doi.org/10.5067/MODIS/MCD12Q1.006
Hengl T, Mendes de Jesus J, Heuvelink GBM, Ruiperez Gonzalez M, Kilibarda M, Blagotić A, et al. (2017) SoilGrids250m: Global gridded soil information based on machine learning. PLoS ONE 12(2): e0169748. https://doi.org/10.1371/journal.pone.0169748
Knoben, W. J. M. (2021). Global USDA-NRCS soil texture class map, HydroShare, https://doi.org/10.4211/hs.1361509511e44adfba814f6950c6e742
Knoben, W. J. M., Clark, M. P., Bales, J., Bennett, A., Gharari, S., Marsh, C. B., Nijssen, B., Pietroniro, A., Spiteri, R. J., Tarboton, D. G. & Wood, A. W. (2022). Community Workflows to Advance Reproducibility in Hydrologic Modeling: Separating model-agnostic and model-specific configuration steps in applications of large-domain hydrologic models. Water Resources Research, e2021WR031753. https://doi.org/10.1029/2021WR031753
Knoben, W. J. M., M. P. Clark, S. Gharari, G. Tang (2022a). Community Workflows to Advance Reproducibility in Hydrologic Modeling: Separating model-agnostic and model-specific configuration steps in applications of large-domain hydrologic models - Continental basin discretizations and river networks, HydroShare, https://doi.org/10.4211/hs.46d980a71d2c4365aa290dc1bfdac823
Lin, P., M. Pan, H. E., Beck, Y. Yang, D. Yamazaki, R. Frasson, C. H. David, M. Durand, T. M. Pavelsky, G. H. Allen, C. J. Gleason, and E. F. Wood, 2019: Global reconstruction of naturalized river flows at 2.94 million reaches. Water Resources Research, doi:10.1029/2019WR025287.
Mizukami, N., Clark, M. P., Sampson, K., Nijssen, B., Mao, Y., McMillan, H., Viger, R. J., Markstrom, S. L., Hay, L. E., Woods, R., Arnold, J. R., and Brekke, L. D., 2016: mizuRoute version 1: a river network routing tool for a continental domain water resources applications, Geosci. Model Dev., 9, 2223–2238, https://doi.org/10.5194/gmd-9-2223-2016
Yamazaki, D., Ikeshima, D., Sosa, J., Bates, P.D., Allen, G.H., Pavelsky, T.M., 2019. MERIT Hydro: A High‐Resolution Global Hydrography Map Based on Latest Topography Dataset. Water Resour. Res. 55, 5053–5073. https://doi.org/10.1029/2019WR024873