ChristinaB commented 5 years ago

DATA on Hydroshare shared with paper collection to get individual DOI's and Paper collection DOI Code in this Github repo to get Zenodo DOI DOC is appendix to the paper.

[ ] HS2 DATA: Depth to Groundwater ASCII files based on date of peak saturation - Christina HS1
[ ] HS2 DATA: Snow April 1 annual ASCII files - Christina
[ ] HS2 DATA: Ice Oct 1 annual ASCII files - Christina
[ ] CODE: saturation extent processing - Nicoleta
[ ] CODE: pyDHSVM or Landlab utitlies for processing data files TBD????
[ ] HS1 DOC: Table of future climate climate change time periods and models -- Christina
[ ] HS1 DATA: GIS maps of landslide inputs- extend this list for each dataset - Ronda
[ ] HS1 CODE: Landslide component.py file - Ronda and Christina
[ ] HS1 CODE: Landslide component tutorial Jupyter Notebook runs on hydroshare - Ronda and Christina
[ ] HS3 DATA: DHSVM Climate forcings Hydrology Inputs
[ ] HS3 DATA: DHSVM Model Instance
[ ] HS2 DATA: DHSVM Model Outputs
[ ] HS4 DATA: Landlab model outputs used to make figures. Results of HS1 Code.

ChristinaB commented 5 years ago

[ ] Add Data Sharing Agreement to the Digital Appendix https://www.hydroshare.org/django_irods/download/a3d213ce180a4fbeb7c354565c35fb87/data/contents/MOA_DHSVM%20Data%20Sharing%20Agreement_2018.pdf
[ ] Update Collaboration Culture https://github.com/Freshwater-Initiative/SkagitLandslideHazards/wiki/Collaboration-Culture

ChristinaB commented 4 years ago

Note - all DHSVM data is subject to this Data Sharing Agreement between UW, SCL, and Upper Skagit Indian Tribe- most of these files are currently public, and Open for educational purposes and public use, but still in production for publication. Please contact Christina to confirm appropriate application and use as work evolves. Data Sharing Agreement for Use of the Skagit River Basin DHSVM Glacio-Hydrology Model Please let me know if there is any discussion or agreements (formal/informal) needed on Open Source Software and the Collaborative Culture

ChristinaB commented 4 years ago

Commercial Break for SCL products related to answering these questions: Blogpost Visualizing Streamflow as Climate Changes, links to This interactive tool is now available online at skagitclimatescience.org/projected-changes-in-streamflow/. The training materials for the tool are on HydroShare Skagit Future Streamflow Visualization and Learning Resources This resource also contains an easy model overview on the data - we should build on this as needed to conceptually describe DHSVM in presentations and figures.

SCL Commercial Break - to answer the remaining questions, there are three locations for SCL Skagit DHSVM modeling that may be useful (in progress, but close to complete, publications pending). Contract deliverable: Landslide Hazard Modeling in the Skagit Basin, Publication files: Slippery Future Code: Predicting future regional landslide probability using soil saturation and Slippery Future Data: Predicting future regional landslide probability using soil saturation

ChristinaB commented 4 years ago

Project Delivery Structured for CIG-WSU-HDR collaborators

1. Where is the Skagit DHSVM model?

Geographically speaking, A Google Map of links in the DHSVM digital network selected for streamflow output are available at: https://www.google.com/maps/d/viewer?mid=13-UUJ47RPVMrBjPlFvDSALjM9qS_5p8l&ll=48.61346235731046%2C-121.49554813369826&z=9 this shows the 72 links extracted from the digital network as a quick view to where there is 3 hourly, daily and other statistics processed for streamflow.

Cyberinfrastructure-wise, Where is the Skagit DHSVM model instance?

Bandaragoda, C. (2020). Skagit River Basin DHSVM model instance 2020, HydroShare, http://www.hydroshare.org/resource/aca13d845c784c7486bccd9c297a0760 is the latest model connected to the ongoing work for the Slippery Future project (UW-SCL).

[ ] To Do: Update and make public for 2018 files with all raw streamflow model outputs in tar files and all climate forcings. 2020 has only 3 GCMs.

2. Precipitation Data:

Skagit Hybrid Bias Correction Results using Livneh and WRF: Code & Data This is in tar files. Optional To Do: 1) Train/document/Onboard process to access and redesign outputs to netcdf/ArcGIS interoperable products designed especially for maps and regression inputs and groundwater. 2) Explore code used in Phuong et al. 2019 which is updated here with a Interactive notebook https://github.com/ChristinaB/Observatory a) click on a .ipynb from Github to see the wonky formatted code. OR b) Look for:

INTERACTIVE! You won't regret clicking on this badge. and launch binder to interact with the methods. There are multiple steps and processes summarized in the Skagit hyperlink above and the Notebook in that resource is specific to the Skagit climate product. These Notebooks are generic to the methods, paper, but do include the Sauk.

[ ] To Do: update environment file for Seaborn

3. Soils dataset used.

SSURGO [ascii file] and DHSVM config files soil thickness is terrain based DHSVM preprocessing, soil class SSURGO, soil thickness NOT from SSURGO for Landlab also available, soil evolution model also available DHSVM 2015 (thunder), 2016 (skagit), 2018 (soil thickness update and sauk improvement), 2020 (update parameter documentation and packaged more versions of soil and geology parameters (Transmissivity, Ksat, cohesion, friction angle). To access and download Seattle City Light Contract Deliverables, visit the Landslide Hazard Modeling in the Skagit Basin HydroShare resource which contains the working folder of data and code.

The simplest download is to 'Download all files as a zipped Bagit File'

Or you can use this link to access the folder of GIS files.

This is one click for a direct link to the soils folder which has files related to the Soil Thickness Ascii File This can be renamed to .asc for import to ArcGIS. The header is in place at the top of the file.

[ ] To Do: get soil classes shapefile from UW computer. The ascii file is used in DHSVM, see the arc/ folder and vegetation parameter of the configuration file.

4. Surficial geology dataset

This is a link to the WA_geology data folder which is inside the GIS files.

This is a link to SSURGO SQL databased on notes on soils methods in the soils folder

5. Land use/cover dataset

This is the folder for LULC data in GIS files

This is a binary DHSVM input file for vegetation classes

[ ] To do: get ascii and shapefile file formats into a GIS folder in the 2018 model instance.

Note: We have a Slippery Future notebooks to access 'interesting data'] (new 2018 NLCD) [ascii file] and DHSVM config files (older NLCD).

6. Groundwater forcings

Question: —deep percolation (water lost to deep drainage by individual grid cell)—discussion on this topic would be great, how it is calculated, is it all routed somewhere else and tracked by model, or assumed lost from the model? We want to have model output of these losses spatially and through time.

We have depth to water table:

Skagit Hybrid Bias Correction Results using Livneh and WRF: Code & Data

Skagit River Basin DHSVM model instance 2020

Slippery Future notebooks to access depth to water table outputs from DHSVM

7. Streamflow Observations

Streamflow code and dataset for instream flow is in development at Skagit Observatory download observed daily data this reads in model data and scrapes the USGS website for the latest datasets for a given USGS location.

ChristinaB commented 4 years ago

Project Delivery Structured for Seattle City Light - UW Contract

1. DHSVM model Inputs to Landslide Model

Geographically speaking, here is an easy access to see the map. A Google Map of links in the DHSVM digital network selected for streamflow output are available. The link shows the 72 links extracted from the digital network as a quick view to where there is 3 hourly, daily and other statistics processed for streamflow.

Cyberinfrastructure-wise, Where is the Skagit DHSVM model instance?
Bandaragoda, C. (2018). Skagit River Basin DHSVM model instance 2018, HydroShare, http://www.hydroshare.org/resource/xxxxxx
is the latest model connected to the Sauk report work.

[ ] To Do: Update and make public for 2018 files with all raw streamflow model outputs in tar files and all climate forcings. 2020 has only 3 GCMs.

Completed Tasks to generate Landslide Forcing:

Determine dates of storm events: Run 2018 model for project dates in Wiki for Table A1. Output: fraction of saturated area for each 3 hourly timestep (saturated_extent.txt). Unanticipated steps: final project run from 2018 only had spatial extent of Sauk. The version of the Skagit from 2018 did not have saturated extent saved/output. Project design was limited to these dates. Next step required rerunning all models for Skagit extent. Collaboration expenses include new workflow and design work for interoperable research coupling three physical process models: DHSVM with Landlab using groundwater process equations.
- Generate spatial map of depth to water as annual peak time series. Step 1: Run 2020 model version of configuration files on Hyak with Sauk dates. Tasks include workflow and design code to generate DHSVM configuration file dates. The initial project design stage using Sauk dates was delivered by Nicoleta Cristea and set to a discoverable Hydroshare resource with Edit Access permissions given to Ronda Strauch Depth to Water Table Spatial Distributions for Skagit Landslide Hazards. The public can discover that the work exists but authors would need to be contacted for access, since these are not the version of the work used in the next steps. UW and SCL staff both have edit access with storage on HydroShare under the UW PI quota.
- Step 2: Run 2020 model version of configuration files on Hyak with Skagit dates. Output is reformatted into compressed ascii files per model run, in folders per future and historic model run (3 GCMs and 2 scenarios). Unanticipated work: The model output was extended to include every date in the 1960 - 2099 time series rather than the limited Table A1 dates. The size of the data has constrained the workflow with existing open source software (xarray, Landlab, jupyter notebooks).

DHSVM Output formatted in typical DHSVM output folders with all data available, with no postprocessing. Bandaragoda, C. (2020). Skagit River Basin DHSVM model instance 2020, HydroShare, http://www.hydroshare.org/resource/aca13d845c784c7486bccd9c297a0760 is the latest model connected to the ongoing work for the Slippery Future project (UW-SCL).

Step 3: DHSVM Output formatted for Landlab forcing workflow: Tasks: design interoperable workflow and code to format DHSVM outputs and structure dataset to be interoperable with storage on HydroShare data repository and transfer to JupyterHub server using XSEDE and CYBERGIS software environments and cloud workspace.

DHSVM output folders for Landlab landslide research paper archive: Bandaragoda, C. (2020). Slippery Future Data: Predicting future regional landslide probability using soil saturation, HydroShare, https://www.hydroshare.org/resource/01b486f301864828ba2cd9ab7ac77c4e/ is the latest model connected to the ongoing work for the Slippery Future project (UW-SCL). The outputs is stored in the DHSVM folder for Slippery Future Data - click here for direct download of the folder.

Step 4: DHSVM Output formatted for Landlab forcing workflow:

2. Precipitation Data:

Skagit Hybrid Bias Correction Results using Livneh and WRF: Code & Data This is in tar files. Optional To Do: 1) Train/document/Onboard process to access and redesign outputs to netcdf/ArcGIS interoperable products designed especially for maps and regression inputs and groundwater. 2) Explore code used in Phuong et al. 2019 which is updated here with a Interactive notebook https://github.com/ChristinaB/Observatory a) click on a .ipynb from Github to see the wonky formatted code. OR b) Look for:

INTERACTIVE! You won't regret clicking on this badge. and launch binder to interact with the methods. There are multiple steps and processes summarized in the Skagit hyperlink above and the Notebook in that resource is specific to the Skagit climate product. These Notebooks are generic to the methods, paper, but do include the Sauk.

[ ] To Do: update environment file for Seaborn

3. Soils dataset used.

SSURGO [ascii file] and DHSVM config files soil thickness is terrain based DHSVM preprocessing, soil class SSURGO, soil thickness NOT from SSURGO for Landlab also available, soil evolution model also available DHSVM 2015 (thunder), 2016 (skagit), 2018 (soil thickness update and sauk improvement), 2020 (update parameter documentation and packaged more versions of soil and geology parameters (Transmissivity, Ksat, cohesion, friction angle). To access and download Seattle City Light Contract Deliverables, visit the Landslide Hazard Modeling in the Skagit Basin HydroShare resource which contains the working folder of data and code.

The simplest download is to 'Download all files as a zipped Bagit File'

Or you can use this link to access the folder of GIS files.

This is one click for a direct link to the soils folder which has files related to the Soil Thickness Ascii File This can be renamed to .asc for import to ArcGIS. The header is in place at the top of the file.

[ ] To Do: get soil classes shapefile from UW computer. The ascii file is used in DHSVM, see the arc/ folder and vegetation parameter of the configuration file.

4. Surficial geology dataset

This is a link to the WA_geology data folder which is inside the GIS files.

This is a link to SSURGO SQL databased on notes on soils methods in the soils folder

5. Land use/cover dataset

This is the folder for LULC data in GIS files

Note: We have a Slippery Future notebooks to access 'interesting data'] (new 2018 NLCD) [ascii file] and DHSVM config files (older NLCD).

6. Groundwater forcings

Question: —deep percolation (water lost to deep drainage by individual grid cell)—discussion on this topic would be great, how it is calculated, is it all routed somewhere else and tracked by model, or assumed lost from the model? We want to have model output of these losses spatially and through time.

We have depth to water table:

Skagit Hybrid Bias Correction Results using Livneh and WRF: Code & Data

Skagit River Basin DHSVM model instance 2020

Slippery Future notebooks to access depth to water table outputs from DHSVM

7. Streamflow Observations

Streamflow code and dataset for instream flow is in development at Skagit Observatory download observed daily data this reads in model data and scrapes the USGS website for the latest datasets for a given USGS location.

ChristinaB commented 4 years ago

Related resources on HydroShare:

Bandaragoda, C. (2018). Data Sharing Agreement for Use of the Skagit River Basin DHSVM Glacio-Hydrology Model, HydroShare, http://www.hydroshare.org/resource/a3d213ce180a4fbeb7c354565c35fb87

[ ] Abstract
[ ] DOI process. DOI clicker.

Bandaragoda, C., S. LEE, E. Istanbulluoglu, A. Hamlet (2020). Hydrology, Stream Temperature, and Sediment Impacts of Climate Change in the Sauk River Basin, HydroShare, http://www.hydroshare.org/resource/e5ad2935979647d6af5f1a9f6bdecdea

Bandaragoda, C. (2018). Skagit Observatory download observed daily and instantaneous (15min) streamflow data., HydroShare, http://www.hydroshare.org/resource/c92e54c72eec4867b422f81c595f3e7b

Bandaragoda, C. (2020). Skagit Hybrid Bias Correction Results using Livneh and WRF: Code & Data, HydroShare, http://www.hydroshare.org/resource/1fee322ad8234efbb4f0f0436b2d1985

Bandaragoda, C. (2020). Skagit River Basin DHSVM model instance 2020, HydroShare, http://www.hydroshare.org/resource/aca13d845c784c7486bccd9c297a0760

Bandaragoda, C., R. Strauch, E. Istanbulluoglu (2020). Slippery Future Code: Predicting future regional landslide probability using soil saturation, https://www.hydroshare.org/resource/a5b52c0e1493401a815f4e77b09d352b/, accessed 8/7/2020, replicated in HydroShare at: http://www.hydroshare.org/resource/4cac25933f6448409cab97b293129b4f

Strauch, R., E. Istanbulluoglu, S. S. Nudurupati, C. Bandaragoda (2018). Regional landslide hazard using Landlab - NOCA Observatory, HydroShare, http://www.hydroshare.org/resource/3a925bd4a5784a38944b1e8b51224de1

Strauch, R., E. Istanbulluoglu, S. S. Nudurupati, C. Bandaragoda (2017). Regional landslide hazard using Landlab - NOCA Data, HydroShare, https://doi.org/10.4211/hs.a5b52c0e1493401a815f4e77b09d352b

Strauch, R., C. L. Raymond, C. Bandaragoda (2020). Skagit Future Streamflow Visualization and Learning Resources, HydroShare, http://www.hydroshare.org/resource/02967ac1edba45058fb12b865b5cb127

Strauch, R., C. Bandaragoda, C. Raymond, N. Cristea (2020). Landslide Hazard Modeling in the Skagit Basin, HydroShare, http://www.hydroshare.org/resource/70d746c7da584ae6bd2f88deb5a4c188

Freshwater-Initiative / SkagitLandslideHazards

Deliverable & Our Slippery Future Digital Appendix (TOC) #16

Project Delivery Structured for CIG-WSU-HDR collaborators

1. Where is the Skagit DHSVM model?

2. Precipitation Data:

3. Soils dataset used.

4. Surficial geology dataset

5. Land use/cover dataset

6. Groundwater forcings

7. Streamflow Observations

Project Delivery Structured for Seattle City Light - UW Contract

1. DHSVM model Inputs to Landslide Model

2. Precipitation Data:

3. Soils dataset used.

4. Surficial geology dataset

5. Land use/cover dataset

6. Groundwater forcings

7. Streamflow Observations