Key questions for comparison groups R&D

[mariano-recurve]

1. On what basis do comparison groups get constructed? a. What criteria do we use to define the composition of a comparison group? b. How are existing comparison group members disqualified and removed from a comparison group?
2. How large does a comparison group need to be? a. What metrics do we use to evaluate the validity of a comparison group?
3. What methodology do we use to include non-participants in a comparison group? a. Random sampling? Stratified sampling? Weighted sampling?
4. How are comparison group “savings” calculated? a. How are these results applied to treated customers?
5. Should there be a formal process for comparison group approval? a. What are the criteria for accepting the validity of a comparison group?

[mblasnik]

proposed Q0 -- What is the goal of the analysis?
I see better comparison groups as a limited tool in addressing most evaluation questions of interest affected by the pandemic. a) If the goal is to measure savings actually achieved only during a specific timeframe that includes the pandemic (with no interest in energy savings in the future) then an ideal comparison group would indeed help answer the question. b) But if the goal is to estimate the lifetime impacts of treatments that are expected to last more than one year/season, then the ideal comparison group is unlikely to answer this question. We would need a way to estimate how savings will change when occupancy and lifestyle patterns transition to the new normal and also estimates of when this all happens.

For example, if the treatment is a more efficient motor or controller for commercial elevators, a perfect comparison group might find that the energy savings are only one tenth what were expected (because the elevators are mostly unused). But those results are unlikely to provide a good estimate of the lifetime or any future energy savings.