search

openeemeter / caltrack

Shared repository for documentation and testing of CalTRACK methods
http://docs.caltrack.org
Creative Commons Zero v1.0 Universal
56 stars 14 forks source link

Metrics for assessing model fit for a portfolio of hourly-modeled buildings #99

Closed eliotcrowe closed 6 years ago

eliotcrowe commented 6 years ago

85 At a previous meeting we saw OpenEE's work on weighted portfolio CV(RMSE), and I wanted to jot down some LBNL thinking on that. If RMSE is squared before adding up the values, then the output is a statistically valid metric, an aggregated CV(RMSE). Two cautions though, when interpreting the results:

  1. A guidance value of <25% has been suggested for CV(RMSE) for a single building, but there is no evidence that the SAME guidance value is appropriate for an aggregated value for a portfolio. The metric may be appropriate but setting an acceptable level for that metric needs more work.
  2. When discussing, we should be clear that this is a model fitness metric, not a quantification of savings uncertainty

We're happy to chat more on this (though I'm not the right person to get deeper in the weeds myself)

image

hshaban commented 6 years ago

Thanks @eliotcrowe. I think we're on the same page - here are some of the conclusions that were buried in the 50 or so threads in #71. in general, we think the aggregate CVRMSE might be a decent metric for planning and trend analysis, but we definitely can't determine the actual values of appropriate thresholds without more testing, because they would vary depending on many different factors.

CVRMSE is not savings uncertainty - CVRMSE is a measure of model fitness and can be used to evaluate how closely a model represents a building's energy use. Fractional savings uncertainty (FSU) is the bounds within which savings are expected to lie (e.g. Savings = 100 kWh +/- 10%). Lower FSU means higher confidence in your results. We are recommending portfolio FSU as the main metric to evaluate portfolio-level results in certain applications. CVRMSE can be used to screen individual buildings prior to an EE intervention, and aggregates of CVRMSE at the portfolio-level may serve as proxies for uncertainty (only the trends, not the actual values). This could be useful in some cases for planning, because calculating FSU requires knowledge of the actual or estimated savings.

hshaban commented 6 years ago

Closing this - Hourly model metrics will be further investigated in #92