Open jslanini opened 5 years ago
Ok. Doing this seems easy enough with how the code handles the metric computations and metric-based model selection. Great job on making it modular @kevinfol! I'll go ahead and work this into the pn-development branch. I have some dedicated time next week for PyForecast work.
An implementable solution for the Cp-metric in the link shown below would require that we first evaluate the MSE of a model using all the candidate parameters. This looks like it would require a bit of work to implement since the change to the code would have to happen early on in the feature selection process and not (as I initially thought) something we could just tack on to the Metrics.py code file.
I suggest replacing Nash Sutcliffe Error with Mallow's Cp as a metric of skill. Nash Sutcliffe should be identical to r-squared in the case of linear regression. Also, all of our metrics are based on squared error. This punishes models with outliers and skews model selection to those that fit really big years.
Also, would it be useful to include mean absolute error, which is similar to RMSE, but weights errors equally. If a forecaster is not worried about the outliers or high water years, it might be nice to select based on non-squared error. For example, we are in a drought year and snowpack is at record lows. We would want to avoid weighting models that fit for the really high water year.