search

rogerlew / wepppy

Other
12 stars 7 forks source link

Calculate Stats #37

Closed marianadobre closed 6 years ago

marianadobre commented 6 years ago

Here's the stats file. I combined the two files into one.

Also, I am saving a Blackwood_forStats folder with the Blackwood run in weppcloud_runs.

Stats.docx

rogerlew commented 6 years ago

Thanks @marianadobre ,

can you upload the blackwood_observed.csv here? I guess it wasn't added to the commit. What do the column labels stand for?

marianadobre commented 6 years ago

Here it is as excel. It would not let me upload csv's.

Columns are:

WaterYear | Date | Streamflow (mm) | Suspended Sediment Concentration (kg/day) | Total Phosphorus (kg/day) | Soluble Reactive Phosphorus (kg/day) | Particulate Phosphorus (kg/day)

blackwood_observed.xlsx

rogerlew commented 6 years ago

Thanks, how about these from the soil.dat files:

Poros (%) ??? Keff (mm/hr) time-invariant effective conductivity Suct (mm) ??? FC (mm/mm) field capacity WP (mm/mm) Wilt point Rough (mm) Manning roughness coefficient Ki Adjusment baseline interrill erodibility parameter (kg*s/m^4) Kr Adjusment baseline rill erodibility parameter (s/m) Tauc Adjusment ???

marianadobre commented 6 years ago

We've played a bit with Keff, not so much with the other parameters. I know Anurag changed Keff and Anisotropy. Not sure what to say about these yet. We want to change the channel soil but I don't know if we want to change the SSURGO soils. I would say we don't want to change these yet but I'll also ask Erin and Anurag and see what they say. Thanks!

rogerlew commented 6 years ago

Okay, cool. What are the Poros, Suct, and Tauc abbreviations for?

rogerlew commented 6 years ago

For the waterbalance are these correct?

Precipitation (mm) -> P Streamflow (mm) -> Q Transpiration + Evaporation (mm) -> Ep + Er Percolation (mm) -> Dp Total Soil Water Storage (mm) -> Total Soil Water

This is the legend in the .wat files J=julian day, Y=simulation year P= precipitation
RM=rainfall+irrigation+snowmelt Q=daily runoff over eff length, Ep=plant transpiration Es=soil evaporation, Er=residue evaporation Dp=deep percolation latqcc=lateral subsurface flow UpStrmQ=Runon added to OFE SubRIn=Subsurface runon added to OFE Total Soil Water=Unfrozen water in soil profile frozwt=Frozen water in soil profile Snow Water=Water in surface snow QOFE=daily runoff scaled to single OFE Tile=Tile drainage (mm) Irr=Irigation (mm) Area=Area that depths apply over (m^2)

marianadobre commented 6 years ago

Where do we have these values? Are these calculated internally? Because they aren't in the soil files.

Poros = Porosity

Suct = Wetting front suction

Tauc = Critical hydraulic shear

On Thu, Mar 29, 2018 at 9:15 PM, Roger Lew notifications@github.com wrote:

Okay, cool. What are the Poros, Suct, and Tauc abbreviations for?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rogerlew/wepppy/issues/37#issuecomment-377440967, or mute the thread https://github.com/notifications/unsubscribe-auth/Aj7cLtolN2vPjJWsVALRgVRpN_BjJV0Vks5tjbFFgaJpZM4TA1Dm .

marianadobre commented 6 years ago

Yes, but to:

Transpiration + Evaporation (mm) -> Es + Er I would also add Ep=plant transpiration

Mariana

On Thu, Mar 29, 2018 at 9:59 PM, Roger Lew notifications@github.com wrote:

For the waterbalance are these correct?

Precipitation (mm) -> P Streamflow (mm) -> Q Transpiration + Evaporation (mm) -> Es + Er Percolation (mm) -> Dp Total Soil Water Storage (mm) -> Total Soil Water

This is the legend in the .wat files J=julian day, Y=simulation year P= precipitation RM=rainfall+irrigation+snowmelt Q=daily runoff over eff length, Ep=plant transpiration Es=soil evaporation, Er=residue evaporation Dp=deep percolation latqcc=lateral subsurface flow UpStrmQ=Runon added to OFE SubRIn=Subsurface runon added to OFE Total Soil Water=Unfrozen water in soil profile frozwt=Frozen water in soil profile Snow Water=Water in surface snow QOFE=daily runoff scaled to single OFE Tile=Tile drainage (mm) Irr=Irigation (mm) Area=Area that depths apply over (m^2)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rogerlew/wepppy/issues/37#issuecomment-377445395, or mute the thread https://github.com/notifications/unsubscribe-auth/Aj7cLsRc_dfhsDJeDulAbAvtqVQ9vYMRks5tjbuZgaJpZM4TA1Dm .

rogerlew commented 6 years ago

Thanks. Based on Bill's feedback and what Anurag wanted with the channel waterbalance I'm thinking this is what we should do for the results and observed data. It should be a superset of what is in your stats.docx

(on main page)

<HEADING> WEPP Results
<LINK> Watershed Loss Report
<LINK> Return Periods Report
<LINK> Frequency Flooding Report

<HEADING> Water Balance Reports
<LINK> Average Annual Report
<LINK> Yearly Report
<LINK> Daily Runoff/Lateral Flow/Baseflow Graph

<HEADING> Model Fit

Yearly Streamflow Yields
    <INPUT> Specify Daily Streamflow Timeseries
    <TABLE> Monthly Descriptives
        cols: January, Feb..., Annual Average
        rows: Model mean, Model std, Observed mean, Observed std

    <TABLE> Fit Statistics
        cols: NSE, ^2, Dv, RMSE
        rows: Hillslopes, Channels

    <LINK>Yearly Streamflow Yields

Yearly Sediment Yields
    ...

Yearly Soluble Reactive Phosphorus Yields
    ...

Yearly Particulate Phosphorus Yields
    ...

Yearly Total Phosphorus Yields
    ...

 ~~~

WEPP Results
<PAGE> Watershed Loss Report
    data: loss_pw0

    <TABLE> Outlet
        rows:
            Total contributing area to outlet   
            Precipitation volume in contributing area   
            Avg. Ann. irrigation volume in contributing area    
            Avg. Ann. water discharge from outlet   
            Avg. Ann. total hillslope soil loss     
            Avg. Ann. total channel soil loss   
            Avg. Ann. sediment discharge from outlet    
            Avg. Ann. Sed. delivery per unit area of watershed  
            Sediment Delivery Ratio for Watershed
            Avg. Ann. Phosphorus discharge from outlet  
            Avg. Ann. P. delivery per unit area of watershed

    <TABLE> Hillslopes
        cols: WeppID, TopazID, Landuse, Soil, Length, Area, Runoff, Subrunoff, 
              Baseflow Vol., Soil Loss, Sediment Dep., Sediment Yield, Solub. 
              React. P., Particulate P., Total P.
        rows: subs

    <TABLE> Channels
        cols: WeppID, TopazID, Length, Area, Discharge Vol., Sediment Yield, 
              Soil Loss, Upland Charge, Subsuface Vol., Flow P., 
              Solub. React. P., Particulate P., Total P.
        rows: channels

<PAGE> Return Periods
    data: ebe_pw0
    <TABLE> Precipitation Depth        
        cols: Year, Month, Day, Precipitation, Runoff, Peak, Sediment Yield, 
              Soluble Reactive P, Particulate P, Total P
        rows: reoccurence intervals

    <TABLE> Runoff
        ...

    <TABLE> Peak Runoff
        ...

    <TABLE> Sediment Yield
        ...

    <TABLE> Soluble Reactive P
        ...

    <TABLE> Particulate P
        ...

    <TABLE> Total P
        ...

<PAGE> <TABLE> Frequency Flood
    data: ebe_pw0
    cols: Precipitation, Runoff, Peak Runoff, Sediment Yield
    rows: Recurrence Interval

Water Balance Reports
    <PAGE> Average Annual
        <TABLE> Hillslopes
            data: H{}.wat.dat
            cols: Precipitation, Streamflow, Transp + Evap, Percolation, Storage
            rows: hillslopes

        <TABLE> Channels
            data: chanwb.out
            cols: Precipitation, Streamflow, Transp + Evap, Percolation, Storage
            rows: channels

    <PAGE> Yearly
        <TABLE> Hillslopes
            data: H{}.wat.dat
            cols: Precipitation, Streamflow, Transp + Evap, Percolation, Storage
            rows: hillslopes, years OR years, hillslopes

        <TABLE> Channels
            data: chanwb.out
            cols: Precipitation, Streamflow, Transp + Evap, Percolation, Storage
            rows: channels, years OR years, channels

    <PAGE> 
        <PLOT> Daily Runoff/Lateral Flow/Baseflow

Observed

    <PAGE> Yearly Streamflow Yields
        <TABLE> Yearly Streamflow Yields
            cols: Observed (data: user), 
                  Predicted Hillslope (data: H{}.wat.dat), 
                  Predicted Channel (data: chanwb.out)
            rows: Water Year
        <PLOT> Yearly Water Yields
        <PLOT> Daily Streamflow vs. Observed Streamflow

    <PAGE> Yearly Sediment Yields
        ...

    <PAGE> Yearly Soluble Reactive Phosphorus Yields
        ...

    <PAGE> Yearly Particulate Phosphorus Yields
        ...

    <PAGE> Yearly Total Phosphorus Yields
        ...
rogerlew commented 6 years ago

@marianadobre For the daily runoff/lateral flow/baseflow plot how is Erin calculating baseflow? Could you send or upload his perl script?

runoff -> Q lateral flow -> latqcc baseflow -> ???

marianadobre commented 6 years ago

All these look great. Will look at them more today.

On Fri, Mar 30, 2018 at 4:08 PM, Roger Lew notifications@github.com wrote:

Thanks. Based on Bill's feedback and what Anurag wanted with the channel waterbalance I'm thinking this is what we should do for the results and observed data. It should be a superset of what is in your stats.docx

(on main page)

WEPP Results Watershed Loss Report Return Periods Report Frequency Flooding Report Water Balance Reports Average Annual Report Yearly Report Daily Runoff/Lateral Flow/Baseflow Graph Model Fit Yearly Streamflow Yields Specify Daily Streamflow Timeseries Monthly Descriptives cols: January, Feb..., Annual Average rows: Model mean, Model std, Observed mean, Observed std
Fit Statistics cols: NSE, ^2, Dv, RMSE rows: Hillslopes, Channels Yearly Streamflow Yields Yearly Sediment Yields ... Yearly Soluble Reactive Phosphorus Yields ... Yearly Particulate Phosphorus Yields ... Yearly Total Phosphorus Yields ... ~~~ WEPP Results Watershed Loss Report data: loss_pw0
Outlet rows: Total contributing area to outlet Precipitation volume in contributing area Avg. Ann. irrigation volume in contributing area Avg. Ann. water discharge from outlet Avg. Ann. total hillslope soil loss Avg. Ann. total channel soil loss Avg. Ann. sediment discharge from outlet Avg. Ann. Sed. delivery per unit area of watershed Sediment Delivery Ratio for Watershed Avg. Ann. Phosphorus discharge from outlet Avg. Ann. P. delivery per unit area of watershed
Hillslopes cols: WeppID, TopazID, Landuse, Soil, Length, Area, Runoff, Subrunoff, Baseflow Vol., Soil Loss, Sediment Dep., Sediment Yield, Solub. React. P., Particulate P., Total P. rows: subs
Channels cols: WeppID, TopazID, Length, Area, Discharge Vol., Sediment Yield, Soil Loss, Upland Charge, Subsuface Vol., Flow P., Solub. React. P., Particulate P., Total P. rows: channels Return Periods data: ebe_pw0
Precipitation Depth cols: Year, Month, Day, Precipitation, Runoff, Peak, Sediment Yield, Soluble Reactive P, Particulate P, Total P rows: reoccurence intervals
Runoff ...
Peak Runoff ...
Sediment Yield ...
Soluble Reactive P ...
Particulate P ...
Total P ...
Frequency Flood data: ebe_pw0 cols: Precipitation, Runoff, Peak Runoff, Sediment Yield rows: Recurrence Interval Water Balance Reports Average Annual
Hillslopes data: H{}.wat.dat cols: Precipitation, Streamflow, Transp + Evap, Percolation, Storage rows: hillslopes
Channels data: chanwb.out cols: Precipitation, Streamflow, Transp + Evap, Percolation, Storage rows: channels Yearly
Hillslopes data: H{}.wat.dat cols: Precipitation, Streamflow, Transp + Evap, Percolation, Storage rows: hillslopes, years OR years, hillslopes
Channels data: chanwb.out cols: Precipitation, Streamflow, Transp + Evap, Percolation, Storage rows: channels, years OR years, channels Daily Runoff/Lateral Flow/Baseflow Observed Yearly Streamflow Yields
Yearly Streamflow Yields cols: Observed (data: user), Predicted Hillslope (data: H{}.wat.dat), Predicted Channel (data: chanwb.out) rows: Water Year Yearly Water Yields Daily Streamflow vs. Observed Streamflow Yearly Sediment Yields ... Yearly Soluble Reactive Phosphorus Yields ... Yearly Particulate Phosphorus Yields ... Yearly Total Phosphorus Yields ... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
marianadobre commented 6 years ago

I just saw the comment on baseflow.

Baseflow is calculated in that excel I sent you, in the WaterBalance spreadsheet. Basically, we calculate reservoir volume, baseflow, and aquifer loss all at the same time based on initial values we provide in that groundwater coefficient text file. After calculating the baseflow, we are using solver to maximize the NSE. I can show you this tomorrow with the excel file.

rogerlew commented 6 years ago

I linked Erin's perl script into the post-processing and implemented the calculations in the excel.

There is a link in the export section to pull the totalwatsed with the added variables.

screenshot from 2018-04-04 22-33-29

The runoff/lateral/baseflow graph is plotting the values from totalwatsed. It is excluding the first 2 years.

screenshot from 2018-04-04 21-55-54

The code that does the excel calculation is here: https://github.com/rogerlew/wepppy/blob/master/wepppy/wepp/out/totalwatsed.py

Are these correct from the gwcoeff.txt? initial_res_volume -> gwstorage baseflow_coeff ->Baseflow coefficient aquifer_coeff -> Deep seepage coefficient

TODO: Need to add the % ratios to the waterbalance reports Need to build control and nodb subclass for calculating the model fit statistics

What is the solver for maximizing NSE for? Maybe we can talk about that tomorrow?

marianadobre commented 6 years ago

The stats should only be available when we are using DAYMET as only then we can actually compare the stremflow with observed. So maybe after the user clicks Daymet for the climate, there can be a button asking them to provide daily streamflow data (mm). Then we read the first and last date in the file and generate a daymet climate for the first year - 2 to last year. For ex, if the user has observed streamflow data between 1990-2006, we will create the climate file from 1988 – 2006. We will also run the model for 17+2 = 19 years. Then in the sediment calculations, we discard the first two years and calculate stats for the 17 years.

Perhaps we can alter the number of years on the interface by default and run it for n + 2 years but show results for the n years. The only problem will be to show the results from the outlet, which will have those two years incorporated. However, the streamflow should still be good.

Yes, the values from gwcoeff are correct.

Solver is an ad-in function in excel that minimizez or maximizez the value of a variable (in this case maximizing the NSE) by changing other variables (baseflow and aquifer coeff). We don’t really needed to do this if it is hard to do in Python. We can just use the default values we have in gwcoeff. Yes, we can talk about this at 2 PM.

On Wed, Apr 4, 2018 at 10:40 PM, Roger Lew notifications@github.com wrote:

I linked Erin's perl script into the post-processing and implemented the calculations in the excel.

There is a link in the export section to pull the totalwatsed with the added variables.

[image: screenshot from 2018-04-04 22-33-29] https://user-images.githubusercontent.com/3652906/38348827-0ecf85b2-3859-11e8-8bbc-15209adebe5d.png

The runoff/lateral/baseflow graph is plotting the values from totalwatsed. It is excluding the first 2 years.

[image: screenshot from 2018-04-04 21-55-54] https://user-images.githubusercontent.com/3652906/38348836-1407083e-3859-11e8-98b5-13bb69f200f1.png

The code that does the excel calculation is here: https://github.com/rogerlew/wepppy/blob/master/wepppy/ wepp/out/totalwatsed.py

Are these correct from the gwcoeff.txt? initial_res_volume -> gwstorage baseflow_coeff ->Baseflow coefficient aquifer_coeff -> Deep seepage coefficient

TODO: Need to add the % ratios to the waterbalance reports Need to build control and nodb subclass for calculating the model fit statistics

What is the solver for maximizing NSE for? Maybe we can talk about that tomorrow?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rogerlew/wepppy/issues/37#issuecomment-378825885, or mute the thread https://github.com/notifications/unsubscribe-auth/Aj7cLt5FZvwOCeNLqicH-s6L6zb-OEAtks5tla5AgaJpZM4TA1Dm .

rogerlew commented 6 years ago

The observed model fit statistics are implemented.

https://wepp1.nkn.uidaho.edu/weppcloud/runs/Blackwood_forStats/lt/

https://wepp1.nkn.uidaho.edu/weppcloud/runs/Blackwood_forStats/report/observed/

rogerlew commented 6 years ago

I also revised the climates. If you pull to your VM it will break your test projects. I can walk you through the changes to migrate those to the new system.

marianadobre commented 6 years ago

These are awesome! The results are a bit off so I will double check the values but I like how they are implemented.

I have a few questions and suggestion so let me know when you are available to talk. I have a seminar starting at 3:30 for one hour. I can talk on skype/meet either before or after, anytime.

Thanks!

On Mon, Apr 9, 2018 at 1:33 AM, Roger Lew notifications@github.com wrote:

I also revised the climates. If you pull to your VM it will break your test projects. I can walk you through the changes to migrate those to the new system.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rogerlew/wepppy/issues/37#issuecomment-379676146, or mute the thread https://github.com/notifications/unsubscribe-auth/Aj7cLlpwvVrwhtUmM7-_G78jI4bwYeU1ks5tmxy9gaJpZM4TA1Dm .

  • © Githubissues.
  • Githubissues is a development platform for aggregating issues.