steevschmidt
commented
5 years ago Notes from working group meeting 10/23/19:
- Ethan expressed concern about including part of the baseline period in the reporting period. Steve suggested that any impacts would cancel themselves out (but provided no proof).
- Jon suggested this approach would make sense if seasonality were a bigger factor in energy use than the savings from a given intervention. Steve concurred.
- Ethan also suggested this issue is more of a "PA contract" issue than a CalTRACK methods issue. Steve asked if current methods would require using a 12 month baseline model to calculate savings for a 1 month reporting period, and I believe the answer was yes. If so, this assumes that a 12 month model will be equally accurate for any month in any season; HEA believes this assumption is faulty, at least for residential buildings.
philngo-recurve
Prerequisites
Article reference number in CalTRACK documentation (optional): 1.4.3, 1.4.4 & 2.1.3.1.2. (Also, see original issue #93, closed during CalTRACK transition from OEE to EM2.)
Description
Background: Because incremental savings are affected by seasonal trends, and daily and monthly CalTRACK models rely on annual baselines, raw incremental savings numbers can vary dramatically from one season to the next.
For aggregators depending on P4P payments, this instability causes hardship in forecasting yields and managing cash flows. Summer savings may rise, only to fall again during winter. That makes settlement/cash flows unstable and causes great consternation between parties.
This instability of savings appears to be particularly true for high energy residential buildings (see related issue #123), as shown in this portfolio-wide savings chart with CalTRACK results over time:
Many households have events that increase or decrease energy use during different times of the year: holiday lighting and/or visitors, pool pump runtimes, school years, athletic schedules, regular & irregular vacations, seasonal grow lights, etc. As a result, a home's non-HVAC (base) energy use in February may be very different from January or July.
Issue: Current CalTRACK methods (and the CPUC?) require a 12 month baseline period but allow shorter reporting periods. These shorter reporting periods can produce inaccurate savings whenever normal seasonal or monthly events are shifted into or out of the short reporting period (e.g. Christmas vacation taken in January one year and December the next). These inaccurate results often "even out" when the reporting period reaches a full 12 months, but since CalTRACK is used to pay monthly savings these short term irregularities can significantly impact incremental savings calculations.
HEA avoids this destabilizing issue by always using a 12 month reporting period. This "rolling 12 month" reporting period will overlap with the baseline period initially, and later can be separated by it by several years (which is useful to monitor savings persistence), as shown in the diagram below showing hypothetical savings for one very well-behaved home:
By covering all seasons, holidays, and vacation periods across a 12 month reporting period this approach provides an apples-to-apples comparison to the fixed 12 month baseline period.
Validation: In April 2018 we ran a simple test comparing default CalTRACK savings results to CalTRACK savings results when the reporting period was extended backward (into the baseline period) to cover a full 12 months. We ran this test on a cohort of 84 homes in PG&E's territory. As a result of this single change, savings results changed by an average of 20% for natural gas and 10% for electricity. This test was easy to perform and we recommend others run similar tests.
Requested CalTRACK change: To reduce dramatic variations in incremental savings calculations for monthly and daily models, update methods to always use reporting periods that cover 12 months in order to calculate annualized savings from a fixed 12 month baseline period.
Proposed test methodology
Perhaps we could use a form of cross-validation testing:
Acceptance Criteria
The 12 different savings calculations for each project using the rolling 12 month reporting periods should show much better consistency and less instability than those using shorter reporting periods. This should be demonstrated across a variety of building types and portfolios without any negative impacts.