Increasing temporal resolution

[ilonah22]

Here are some initial steps to take toward increasing temporal resolution of our StormVol and SimpleLM methods:

• [ ] Workflow to create dataset with predicted flow, error, and rating column

Use regression equation to calculate error against flow:

• [ ] Go through precip workflow using some timestep (event?, week?, month, etc.) to get estimated flow using monthly regression equation to obtain coefficients
• [ ] Pick a single basin to work with initially, each basin has its own coefficients:
  • [ ] Create a script with hardwired basin coefficients to test method
  • [ ] Take predicted vs. observed flow and calculate 1.0 - percent error

[rburghol]

@COBrogan we created this issue today to house the higher resolution method we've been noodling on. I recall you writing an equation for the error assessment somewhere but cannot for the life of me recall where. When you have a minute if you could take a quick look or just drop an idea that would be awesome. Thanks!

[rburghol]

@ilonah22 thanks for setting this up. Personally, since the storm method is already divided up by events, this seems like the easiest way to begin. Just add the predicted flow, error and rating column to that dataframe and then we can try to fit it into next steps of the workflow. That make sense?

[ilonah22]

@rburghol Sounds good.

[COBrogan]

I recall you writing an equation for the error assessment

@rburghol it was pretty basic, but it was part of our large agenda #https://github.com/HARPgroup/HARParchive/issues/1355. Basically, I had thought we could decide error as the residual/error ("predicted" - observed) value for a given week/storm event normalized by the observed value. This would give us a rating that is essentially the percent error for that day: $rating = 1 - \frac{abs(Q{obs} - Q{model})}{Q_{obs}}$

This would give us an effective rating that ranges from negative infinity to 1. Now, as you know, percent errors tend to blow up at smaller observed values. One thing we may want to be conscious of is that ratings will likely be lower for smaller flow weeks/events and larger for larger flow weeks/events. This difference needs to be taken in scope of the percent error calculation so that we aren't thinking "Our methods perform better for large events" without looking in detail at the regressions. Hopefully that makes sense...

[rburghol]

@COBrogan Thanks a bunch, I failed to look in the most obvious place!

I also appreciate your thoughts, it had not really occurred to me that small flows would likely have enormous hairs, but I see that you're very correct.

However, I think you were implying that as long as we understand the ratings as a means to compare amongst methods during a single slice of time, and not as a means of evaluating variations within a single method, we should be good.

[COBrogan]

However, I think you were implying that as long as we understand the ratings as a means to compare amongst methods during a single slice of time, and not as a means of evaluating variations within a single method, we should be good.

Yes, exactly that! This pct error rating will be useful as a comparison against methods for an event/week, but broader inferences about the variation in any single method would require further analysis.

[rburghol]

@COBrogan @ilonah22 do either of you see any reason why we shouldn't set the minimum value to zero, rather than to allow large negative values? Or I guess rather, what pros and cons do you all see to that approach?

[COBrogan]

I'd prefer to keep the negatives at first. My pros and cons of setting minimum value to zero: Pros:

1. Easier to understand the overall rating system
2. Eliminates cases where error is greater than 100%, which are likely the cases we are not as interested in
3. Focuses decisions on cases where residual are within 100% of the observed value, thereby setting a threshold above which error is considered unacceptable
4. Eliminates cases where we pick a "best-fit" dataset even though all error is very large (e.g. greater than 100%).

Cons:

1. Setting minimum value to 0 establishes a threshold of 100% error, which for the weekly lm method means that a predicted flow value of 2.1 cfs is unacceptable for a flow of 1 cfs. However, a predicted flow of 1999 cfs is acceptable if observed flow is 1000 cfs. This may be acceptable if we consider 1.1 cfs an unacceptable error for an observed low flow of 1 cfs.
2. By setting the threshold, we will be using the background data more often as there may be weeks/events where error is greater than 100% for all datasets. This leads to a situation where error using NLDAS2 may be the greatest for the week/event, but we still use it anyways as it is our background dataset

I think this is a good idea, but maybe we go from -1 to 1? That way we are effectively saying 200% error and above is not acceptable, which gives us a little more wiggle room on low flows. Of course, for higher flows 200% error is a lot so I can see why 0 - 1 is better for large flows.

[rburghol]

@COBrogan your example where we erroneously default to NLDAS2 makes perfect sense -- I vote. We keep the negatives for now!

[ilonah22]

I just pushed a file with the predicted flow functions I am working on, so far it has two functions to add predicted flow into a data frame with month as an input. One of them includes the error and rating, but I just wanted to try and see how I could add them all together before I tried to do anything too complex.

predictedFlow.R

[ilonah22]

@rburghol On Tuesday Connor and I talked about putting the predicted values into our data in the stormEvents_LM step and he agreed with you that it seemed like the best place, even though there is already a lot happening in that step.

Right now, that script is writing out the json into the /stats folder, and the monthly ratings into the /out folder. We decided the plan should be to write out another csv into the /out folder which includes the predicted flow with the original precip and flow data, since the data is being manipulated, it should not go back into the /precip or /flow folders. Then, separately, we can calculate errors and ratings using that data.

• Also, the plan is to keep units in MG

[rburghol]

@ilonah22 thanks for the update! Sounds like a good plan… Look forward to seeing some examples when we get together next.

[ilonah22]

Before yesterday's meeting, I pushed up a version of the stormEventsLM_cmd.R that I am working on to a new branch of the meta_model/scripts/met called predicted. Even though the current commit version I have in there is not what we expect the final product to be, it's there in case you want to take a closer look at anything.

stormEventsLM_cmd

The main change I will be making to that command this week is to predict flow based on stormStats rather than comp_data. Also, yesterday @rburghol briefly mentioned units and how inches might be easier to visualize, so, if that is a change we want to make, I can try to do that as I go through the command, just let me know.