search

DOI-BOR / PyForecast

PyForecast is a statistical modeling tool used by Reclamation water managers and reservoir operators to train and build predictive models for seasonal inflows and streamflows. PyForecast allows users to make current water-year forecasts using models developed with the program.
Other
28 stars 12 forks source link

Model skill metrics #20

Open jslanini opened 5 years ago

jslanini commented 5 years ago

I suggest replacing Nash Sutcliffe Error with Mallow's Cp as a metric of skill. Nash Sutcliffe should be identical to r-squared in the case of linear regression. Also, all of our metrics are based on squared error. This punishes models with outliers and skews model selection to those that fit really big years.

Also, would it be useful to include mean absolute error, which is similar to RMSE, but weights errors equally. If a forecaster is not worried about the outliers or high water years, it might be nice to select based on non-squared error. For example, we are in a drought year and snowpack is at record lows. We would want to avoid weighting models that fit for the really high water year.

tjrocha commented 5 years ago

Ok. Doing this seems easy enough with how the code handles the metric computations and metric-based model selection. Great job on making it modular @kevinfol! I'll go ahead and work this into the pn-development branch. I have some dedicated time next week for PyForecast work.

tjrocha commented 5 years ago

An implementable solution for the Cp-metric in the link shown below would require that we first evaluate the MSE of a model using all the candidate parameters. This looks like it would require a bit of work to implement since the change to the code would have to happen early on in the feature selection process and not (as I initially thought) something we could just tack on to the Metrics.py code file.

https://newonlinecourses.science.psu.edu/stat501/node/330/