Should CalTRACK v1 monthly methods use a fixed or variable degree day base for calculating site-level savings?

[matthewgee]

The degree day base used in CalTRACK monthly methods was discussed initially in meeting 8 and then again in meetings 16 and 17 in the context of the Beta Test. The initial agreement was to use a variable degree day base with a grid search and adjusted R-squared selection method. However, challenges in getting outputs to agree led the beta testers to agree to modify this to a fixed degree day base in pursuit of output consistency. That is currently reflected in the monthly methods specification.

Since VDD is standard practice in industry and the fixed basis was selected in pursuit of the short-term goal of rooting out other more serious issues in the specification, OEE proposes that the v1 monthly specification should go back to using variable a degree day basis.

[jbackusm]

Actually, EnergySavvy did some analysis investigating balance-point optimization for the monthly models--see the results here.

The takeaway was that it didn't seem to matter (for this dataset, for monthly models) whether we used fixed balance points or not, so it made sense to choose the simpler option.

If we choose to go back to variable balance points, we will need some way to mitigate the tendency toward extreme values, or somehow handle the extreme values. For example, we could fall back from the extreme values to a default fixed balance point, but I would again claim in that case that the end results would not be significantly different from using fixed balance points from the start.

[mcgeeyoung]

@jbackusm It sounds like for monthly, the complexity of VDD isn't worth the benefit. Let's put that in the final spec. And revisit for Daily. OK with you? @jfarland?

[jfarland]

@mcgeeyoung @jbackusm I think that sticking with fixed degree base temperatures makes sense for this purpose. Agree @gkagnew?

[jbackusm]

I agree for the purposes of monthly modeling, at least with the PG&E AHUP dataset--it might be that VBDD is more important for monthly modeling on other datasets, but this is all we have to work with for now...

[matthewgee]

I agree with the above. If the AHUC data is representative generally of the kinds of projects that CalTRACK use cases will support (which is a reasonable near-term assumption), then there is little evidence that, for monthly methods, having a fixed degree day base instead of VDD makes a substantive difference in portfolio-level savings, while it demonstrably does lead to added complexity on implementation that can lead to substantive differences in portfolio-level savings. Given that, sticking with a fixed degree day base for the monthly spec is a reasonable and defensible choice.

Of course, we will want to revisit this for the daily methods.

@gkagnew, it would be great if you could chime in on this before we close this issue since this is one that you raised as important for reconsideration and that you have particular insight into that we could all benefit from.

[gkagnew]

I tend to agree that the evidence put forward by ES indicates that for this data the VDD does not make much difference. However, as I have said multiple times, optimal VDD selection will only tend to show substantial improvements the further "out of sample" you are predicting, in this case predicting post year consumption under very different weather conditions from the pre-data on which the model was estimated. It is always risky to make decisions on 1 in 2 weather year data as we have no way of knowing whether next years data will be a 1 in 10 year, or worse.

To add to the complication, my experience indicates that even a good weather normalization process (on, say, data with a well support cooling trend) may not be enough to address all weather-related differences, and that this may be the most important aspect of the comparison group. That is, some portion of the non-weather, non-program pre-post change we look to the comparison group to address, is likely to be weather effects not controlled for in these simple models, but mostly addressed by the comparison group estimated using the same limited model.

So, in short, putting into place a monthly fixed DD for measuring pre-post change as a proxy for savings sounds extremely risky to me. However, I am willing to concede that under the circumstances this appears to be a necessary first step in a process that will seek to mitigate those risks with more granular data, VDDs and comparison groups in the near future.

[matthewgee]

This is a great summary @gkagnew. Essentially, but having v1 be fixed DD, we are allowing for a savings measure that exposes stakeholders to higher weather-related measurement risks.

First, our sensitivity analysis was not a true measure of observed weather risk exposure because we used only one (relatively mile) weather scenario to perform the comparison. Likely, the difference between VDD and fixed DD would be larger in extreme weather scenarios, and those differences could significantly increase or decrease measured savings relative to the normal year, indicating that one method is doing a poorer job of weather normalization than the other because the goal of weather normalization is to provide relatively stable estimates of savings conditional on weather given a particular intervention.

Second, part of weather risk in savings measurement is actually unobserved weather. This effect can be large, and because it's unobserved, by definition it's loaded into the error term. The standard way to reduce variance in your savings estimator to due unobserved but systemically correlated factors is through a well-defined comparison group. By omitting this from v1 of the monthly method, we are exposing users of this method for savings measurement to additional weather-related quantification risk that they will need to either take into account in some other way (post-estimation adjustments, pricing, program design, etc) or be willing to take on in its entirety.

I think as long as we are absolutely clear about the quantification risks that users of a particular versioned method are taking on by using it, and we have a plan for addressing some or all of those concerns in future versions (e.g. daily methods with VDD), I am okay with releasing a version of monthly methods that have not accounted for all quantification risks, weather-related or otherwise, given some of the other tradeoffs discussed above.

[matthewgee]

Based on the discussion above, I'd like to propose we close this issue for CalTRACK v1 monthly methods with the following fixed degree day base temperatures use in calculating HDD and CDD:

HDD base temp: 60 F CDD base temp: 70 F