Gross savings adjustments from future participants?

[jbackusm]

From UMP chapter 8, the second stage in the two-stage approach effectively refers to adjustments that come from cross-sectional analysis of the comparison group (future participants in our case). Have we discussed at all how to do this in such a way as to correct for systematic differences in the treatment and comparison groups? For example, 1:1 matching criteria, or an aggregate model of the changes in comparison-group usage patterns depending on home characteristics or baseline usage?

In our current implementation of comparison groups at EnergySavvy, we split comparison groups geographically by weather station, and we actually use random forests to model the necessary adjustment as a function of baseline stage-one model coefficients, as well as time elapsed and weather observed in the post-treatment period so far.

[matthewgee]

Good question. Since the determination by PG&E in the P4P use case was to not use comparison groups for calculating payable savings, we first have to do a second stage without a comparison group adjustment.

However, to estimate the savings estimation risk that PG&E and the aggregators might face by not having a comparison group, and in support of the software feedback use case, we'll want to do a separate analysis that includes future participants as the comparison group.

We discussed several approaches to doing this, mostly ranked by their replicability, and I think the consensus was that an aggregate model by geography where CalTrack could publish adjustment factors would be the most replicable.

Your approach sounds totally sensible; conditioning by geography then taking an ensemble approach to estimation.

What do you guys think about just one of us coming up with the adjustment factors by geography and sharing those on the repo so that everyone else can apply them. I think that will simplify things the most and it will replicate what the Technical Working Group discussed as the long-term solution for public factor adjustments. Thoughts?

[jbackusm]

That seems like a reasonable approach, though there might be some important details that haven't yet been discussed:

• Should these adjustment factors correspond to only a single point in time, i.e. 12 or 24 months after program start? Or should we have adjustments corresponding to every possible post-treatment period (maybe coarsened by month)?
• In the latter case, how do we assign "fake treatment dates" to the comparison premises? Should we do it randomly, or use the full comparison population (by geographical region) for each treatment date?
• Alternatively, would we prefer a 1:1 or many:1 matching approach based on the pre-treatment usages? How would we specify that matching? Also, note that we may not get the full benefit of matching if we're just applying a single aggregate adjustment in each geographical region.
• Should we at least explore the extent to which the adjustment varies within each geographical region, for example by annual pre-treatment usage? Since we have use cases in which we want to ensure accuracy for smaller subsets of the population (e.g. all projects from a single contractor), this seems important.
• Finally, I would be somewhat concerned that we may get "lucky", and the adjustment may turn out to be small for the particular timeframe and population we're considering. Should we also explore other timeframes, for example 2012 vs. 2013, 2013 vs. 2014, 2014 vs. 2015, etc? In each step we could use all premises that lack a project in the 2-year window.

[jbackusm]

One more basic question: are we assuming these adjustment "factors" to be additive or multiplicative on the baseline NAC? In the UMP they're certainly additive, but I worry about an additive approach when we're constrained to just a single adjustment for an entire geographic region, without regard to overall amount of consumption at the site. Leaves a lot of room for error if the current and future participants just happen to be different in terms of usage profiles.

[mcgeeyoung]

Since future participants are not part of use case, closing issue.