search

Freshwater-Initiative / SkagitLandslideHazards

Seattle City Light is interested in improving understanding of landslide hazard and sediment transport to ensure reliable and cost-effective hydropower generation.
4 stars 4 forks source link

Table for Sanity: Design for reproducible and Notebook based figure publication #8

Open ChristinaB opened 6 years ago

ChristinaB commented 6 years ago
ChristinaB commented 5 years ago

@RondaStrauch can you please turn User Documentation into its own issue and close this issue?

ChristinaB commented 4 years ago
Table: Landlab Landslide Notebooks Purpose Name [hyperlinked] Version Status Launch Me Info
1 Notebooks for Landlab Component Welcome Tutorials 1. Learn about Landslide Component defaults using a synthetic grid 2. Explore Landslide Component Options for Hydrologic Forcing and Statistical Distributions 3. Investigate a Landlab Unit Grid with the Unit Test In Development: Pull request on branch ChristinaB-landslide waiting for merge to HEAD Landlab run on: Click this hyperlink to Launch Binder - Same as badge on readme - Same as badge on read me
2 Update Landlab 1.0 Recharge forcing & Validate Component Update - Notebooks to Replicate Landlab 1.0 Recharge forcing Notebooks. Earlier notebooks are changed as little as possible, since they were part of earlier publications, but the software env. and Landslide component have evolved. Original Pub Github Repo: Notebooks: 1. NOCA_runPaper_LandlabLandslide.ipynb 2. NOCA_runPaper_LandlabLandslide_alreadyRun.ipynb 3. Synthetic_recharge_LandlabLandslide.ipynb Updated Publication: Notebooks: 1. NOCA_runPaper_LandlabLandslide.ipynb 2. NOCA_runPaper_LandlabLandslide_alreadyRun.ipynb 3. Synthetic_recharge_LandlabLandslide.ipynb Ready to Release as an update to earlier publications. Python and Landlab updated OR environment file or kernel created to host earlier version. Instructions for Landlab v2 Notebooks are in Christina's Fork ReadMe - change from Master to Landlab-v2 Branch to see Notebooks updated April 2020/
3 KI paper updates
4 Publish Notebooks specific to Slippery Future Figures in Landlab V2 Python 3 Environment Notebooks: 1. dhsvm_landslide_model_for_fire.ipynb 2. netcdf_landslide.ipynb 3. replicate_landslide_model_for_fire_DHSVM.ipynb 4. Unit_Test_Landslides.ipynb Developer and Ongoing Development with continuously updated software environment. Open with CyberGIS-Jupyter for Water. Use the Landlab kernel
ChristinaB commented 4 years ago

Purpose: Notebooks for Landlab Component Welcome Tutorials Version Status: In Development. Pull request on branch ChristinaB-landslide waiting for merge to HEAD Landlab Launch Me Info: run on Click this hyperlink to Launch Binder - Same as badge on readme image

Same as badge on readme of the Welcome Notebook. Notebooks:

  1. Learn about Landslide Component defaults using a synthetic grid
  2. Explore Landslide Component Options for Hydrologic Forcing and Statistical Distributions
  3. Investigate a Landlab Unit Grid with the Unit Test
ChristinaB commented 4 years ago

Purpose: Update Landlab 1.0 Recharge forcing; Validate Component Update. Notebooks to Replicate Landlab 1.0 Recharge forcing Notebooks. Earlier notebooks are changed as little as possible, since they were part of earlier publications, but the software env. and Landslide component have evolved.
Version Status: Ready to Release as an update to earlier publications. Python and Landlab updated OR environment file or kernel created to host earlier version.

Publication: Strauch RL, Istanbulluoglu E, Nudurupati SS, Bandaragoda C, Gasparina N, and Tucker G. (2018). Hydro-climatological approach to predicting regional landslide probability. Earth Surface Dynamics 6:1-26. Original Pub Github Repo: Notebooks:

  1. NOCA_runPaper_LandlabLandslide.ipynb
  2. NOCA_runPaper_LandlabLandslide_alreadyRun.ipynb
  3. Synthetic_recharge_LandlabLandslide.ipynb

Updated Publication: TBD DOI ? Slippery Future Teaching Demo? Intelligent Earth HydroLearn Course Section. [Instructions for Landlab v2 Notebooks are in Christina's Fork ReadMe - change from Master to Landlab-v2 Branch to see Notebooks updated April 2020/ Notebooks:

  1. NOCA_runPaper_LandlabLandslide.ipynb
  2. NOCA_runPaper_LandlabLandslide_alreadyRun.ipynb
  3. Synthetic_recharge_LandlabLandslide.ipynb
ChristinaB commented 4 years ago

Purpose: Update Landlab 1.0 Recharge forcing used in KI paper with thunder fire example; Validate Component and python3 Update. Notebooks to Replicate Landlab 1.0 Recharge forcing Notebooks. Earlier notebooks are changed as little as possible, since they were part of earlier publications, but the software env. and Landslide component have evolved.

Version Status for 3 HydroShare Resources: Ready to Release as an update to earlier publications.

  1. Formally published DOI with journal publication; interactive code not maintained: Enabling Collaborative Numerical Modeling in Earth Sciences using Knowledge Infrastructure: Landlab Notebooks Notebooks: explore_routing_tutorial.ipynb replicate_landslide_model_for_fire.ipynb reuse_ecohydrology_gridhydromet.ipynb

  2. Maintained original software environment for original DOI. New HS version has two options for running the original code for all three notebooks. Option 1: Run with MyBinder, see Binder folder for software environment files. Comment out the conda update (unless you need it).
    Option 2: To run from HSJP07 - CyberGIS Jupyter for Water developer at illinois - change to Landlab kernel . This kernel has Landlab version 1 and python 2 setup for this publication. If you don't see (dev at hs07) as an option, go to this web app for dev CyberGIS-Jupyter for Water and add this to your Web App's by clicking on the waffle (red array of nine squares should turn green).
    See also What's New in CyberGIS-Jupyter for Water (CJW) 2020 Q2 Release When you open the Notebooks, Comment out the conda update (unless you need it).
    Direct link is this location: https://hsjp07.cigi.illinois.edu Notebooks: explore_routing_tutorial.ipynb replicate_landslide_model_for_fire.ipynb reuse_ecohydrology_gridhydromet.ipynb

  3. Developer and Ongoing Development with continuously updated software environment. [CJW] Enabling Collaborative Numerical Modeling in Earth Sciences using Knowledge Infrastructure: Landlab Notebooks !!! This is a fork from https://www.hydroshare.org/resource/5b964154ebf945848087bdc772cc921e/ with some minor modifications for CyberGIS-Jupyer for Water (CJW) platform !!! Landlab Version 2 !!!! Python 3 !!!! Option 1: use the Binder and HS MyBinder Option 2: Join the CyberGIS-Jupyter for Water Group.
    Right click on the Notebooks. Open with CyberGIS-Jupyter for Water. Use the Landlab kernel.
    Notebooks: explore_routing_tutorial.ipynb replicate_landslide_model_for_fire.ipynb

--

Python and Landlab updated OR environment file or kernel created to host earlier version.

Publications: Original Pub Github Repo: Updated Publication: Bandaragoda, C. J., A. Castronova, E. Istanbulluoglu, R. Strauch, S. S. Nudurupati, J. Phuong, J. M. Adams, et al. "Enabling Collaborative Numerical Modeling in Earth Sciences Using Knowledge Infrastructure." Environmental Modelling & Software, April 24, 2019. https://doi.org/10.1016/j.envsoft.2019.03.020.

ChristinaB commented 4 years ago

Bonus! Skagit Observatory download observed daily and instantaneous (15min) streamflow data

ChristinaB commented 4 years ago

Slippery Future

Slippery Future Data: Predicting future regional landslide probability using soil saturation

[Slippery Future Code: Predicting future regional landslide probability using soil saturation] https://www.hydroshare.org/resource/4cac25933f6448409cab97b293129b4f/

Purpose: Publish Notebooks specific to Slippery Future Figures in Landlab V2 Python 3 Environment.
Slippery Future: Multi-hazard impacts of climate change and wildfire on landslide probability: Landlab shallow landslide model for Goodell Creek, Skagit Watershed

Compute: Developer and Ongoing Development with continuously updated software environment.
Option 1: Right click on the Notebooks. Open with CyberGIS-Jupyter for Water. Use the Landlab kernel. (assumes you have joined the CyberGIS-Jupyter for Water Group. ) Option X - not pursuing for big models: use the Binder and HS MyBinder; save env. for final short versions Option 2: Desktop version

Notebooks:

  1. dhsvm_landslide_model_for_fire.ipynb

    • Load a geographic subset (Goodell Creek watershed) within the study region.
    • Use a high resolution distributed hydrologic model (DHSVM depth to water) for annual statistical distribution of maximum saturation events in a data driven approach using depth to water table (compared to lower resolution gridded hydrology model with recharge).
    • Explore landslide probability sensitivity to climate change using two scenarios, three models (CMIP GCMs), and two future time periods compared to historic and landslide observations.
    • Explore landslide probability sensitivity to fire by adjusting the cohesion parameter.
    • Save results to a new HydroShare resource.

  2. netcdf_landslide.ipynb

    • Load data from a regional Landlab landslide model (Strauch et al., 2018) developed for the North Cascades National Park, WA USA, published on HydroShare.
    • Load DHSVM data and resample to SCL domain (aligned with Landlab nodes)
    • Create a geographic subset (Goodell Creek watershed) within the study region.
    • Save a netcdf files at SCL domain and subset domain with compliant standard attributes for Landlab and HydroShare.

  3. replicate_landslide_model_for_fire_DHSVM.ipynb

    • Replicate earlier notebooks with DSHVM forcing (compared to recharge)
    • Replicate earlier notebooks with SCL forcing instead of Skagit forcing (2018 paper)
    • Compare Thunder, Goodell, and SCL average estimates (compare subsets)
    • Compare statistical distributions to data driven inputs (uniform, lognormal, lognormal spatial, data driven)

  4. Unit_Test_Landslides.ipynb Explore and learn how to use the Landlab unit test based on default SCL unit landslides.

  5. Slippery Future Citations & References.ipynb

Zotero References Updated:

ChristinaB commented 4 years ago

Reorganizational Redesign AhHah Moment - Scaling Approach to Managing Data/Model Size

Story Notebook Epic Notebook Deep Notebook

  1. Deep "Reproducible" Large - Seattle City Light Domain - Landslide Model inputs 30 part 0 - read in DEM (and late read in DHSVM forcings)

part 1: read in Observations and Location information

part 2: read in files and make netcdf

  1. EPIC "Reproducible" Large - Seattle City Light Domain - Landslide Model inputs 30 -

    • Import Landslide forcing for SCL (netcdf)
    • Import DHSVM hydrology forcing for SCL (netcdf) to compare with Depth and Recharge default range
    • Plot comparison of means and range to QC the input files to a synthetic grid
    • Run Landslide model with Lognormal-Spatial to compare historic v 4.5 v 8.5 (historic)
    • Run Landslide model with Lognormal-Spatial to compare to Landcover and observations and Goodell Creek fire (orientation)

  2. EPIC "Reproducible" Large - Seattle City Light Domain - DHSVM 150 - 1020 x 916 - Lognormal Spatial

    • Read in hydrology data file and convert to xarray to calculate mean and standard deviation on 1020 x 916 grid (934,320 grid cells)
    • HydroForcing Processing Function proof of concept for historic.
    • Function for each model: Resample to 30 m grid cell, Export Hydrology Reformatted Files: 1) annual time series as netcdf4 150 m; 2) mean and std dev as netcdf4 30 m 3) mean and std dev as pickle 30 m 4) export Subset to Goodell Creek using Mask at 30 m 5) export pickle of annual values for subset (distribution of DTW at each node; only for subset)
    • Plot comparison of means and seasonal dates to assess scale differences
    • Import Landlab landslide forcing for SCL (netcdf)
    • Run Landslide model with Lognormal-Spatial to compare historic v 4.5 v 8.5 (historic)

  3. STORY "Landlab" Landslide for SCL - compare future hydrology compared to historic

    • Import Landlab landslide forcing for SCL (netcdf)
    • Import DSHVM hydrology forcing for SCL (pickle) for historic plus six models
    • Run Landslide model with Lognormal-Spatial

  4. STORY "Landlab" Landslide for Goodell Creek - compare hydrology distributions forcing landslides with historic data

    • Import Landlab landslide forcing for Goodell (netcdf)
    • Import DSHVM hydrology forcing for Goodell (pickle) for historic
    • Run Landslide model with Lognormal-Spatial
    • Run Landslide model with Data-Driven Spatial
    • To compare landslide parameterization and sensitivity range

  5. STORY "Landlab" Landslide with Fire for Goodell Creek - - compare future hydrology compared to historic

    • Import Landlab landslide forcing for Goodell (netcdf)
    • Import DSHVM hydrology forcing for Goodell (pickle) for historic plus six models
    • Run Landslide model with Data-Driven Spatial to compare historic v 4.5 v 8.5 (historic v one future)
    • Run FIRE+ Landslide model with Data-Driven Spatial to compare historic v 4.5 v 8.5 (historic v one future)

  6. DEEP "computer scientist" Notebook

    • Part 1: Visualization of Storms seasonal dates and mean static map for each 150 model; and animations of the subsets at Goodell Creek 30 m
    • Run Landslide Model
    • Part 2: Visualization of Landslide seasonal dates and means for lognormal spatial and animations for each 150 model and subset at Goodell Creek 30 m
ChristinaB commented 4 years ago

@RondaStrauch here are the three notebooks we updated today

Go to this HydroShare Resource

Slippery Future Code: Predicting future regional landslide probability using soil saturation

Launch these from the CyberGIS for Water Landlab kernel

Slippery Future Citations & References.ipynb observedlandslides_data+landlab+netcdf.ipynb replicate_landslide_depthmodel_for_fire.ipynb

Update To Do list

RondaStrauch commented 4 years ago

I think observedlandslides_datalandlabnecdf.ipynb is the latest...not 20200806

2 notebooks are listed in jupyter when open up with CyberGIS, but aren't in the hydroshare resource...this is even during an overwrite. The _DHSVM and _20200806 are not in resource

Line 4 - still says /notebooks/ doesn't exist, but it does so thinking we don't need this line, comment out?

The final upload of the notebook to github doesn't show an updated notebook??

@ChristinaB I cleaned up the observed data notebook and named it _clean. There are some things for you to check and these are called out by either DELETE, CHECK, or CHRISTINA. The os.chdr() still throws an error if the folder is already present. Perhaps we can have a code line that creates this only if it is not present.