Closed DrDAJump closed 1 year ago
steevschmidt
commented
5 years ago I agree the CalTRACK "model of choice" might vary by building type, and will probably change over time as we gain experience.
For example, HEA has found a daily LOESS regression model appears to produce better results for many homes.
lwebster101
commented
5 years ago I agree that additional model types should be included, especially since there are many that have been vetted for use with commercial facilities.
mcgeeyoung
commented
5 years ago David, this is an interesting observation. The testing was done in CalTRACK 2.0 on residential and non-residential buildings. There's a general consensus that CalTRACK methods might not be right for some buildings. In fact, we spent 25% of CalTRACK 2.0 working on this issue of project qualification and tested the methods on more than 50 million meters representing all of the data of the 4 IOUs thanks to a data set provided by the CEC. Some of the results are found in the CalTRACK methodological appendix, which suggested that there are some buildings for which custom or non-NMEC methods should be used. In other cases, like schools, there is some seasonality that needs to be addressed, which tends to be solved pretty well with how the hourly methods are implemented.
There is a broader point that needs to be understood. CalTRACK is not designed to be "all of the methods." It is a particular approach to calculating avoided energy use that is broadly applicable to most buildings, but not all. Where it isn't, other (custom) methods should be used. But for its main audience, the core values of consistency and specificity are more important.
lwebster101
commented
5 years ago Hello- Allowing only this model type seems too limited for commercial applications since it is more complex than others being successfully applied. Perhaps some delineation between residential and commercial applications could be helpful.
I am unable to attend today but wanted to share my thoughts on this.
Thank you, Lia
Thank you, Lia Webster
720.470.9297
Please excuse any typos.
On May 21, 2019, at 9:52 PM, McGee Young notifications@github.com<mailto:notifications@github.com> wrote:
David, this is an interesting observation. The testing was done in CalTRACK 2.0 on residential and non-residential buildings. There's a general consensus that CalTRACK methods might not be right for some buildings. In fact, we spent 25% of CalTRACK 2.0 working on this issue of project qualification and tested the methods on more than 50 million meters representing all of the data of the 4 IOUs thanks to a data set provided by the CEC. Some of the results are found in the CalTRACK methodological appendix, which suggested that there are some buildings for which custom or non-NMEC methods should be used. In other cases, like schools, there is some seasonality that needs to be addressed, which tends to be solved pretty well with how the hourly methods are implemented.
There is a broader point that needs to be understood. CalTRACK is not designed to be "all of the methods." It is a particular approach to calculating avoided energy use that is broadly applicable to most buildings, but not all. Where it isn't, other (custom) methods should be used. But for its main audience, the core values of consistency and specificity are more important.
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/energy-market-methods/caltrack/issues/128?email_source=notifications&email_token=ALT4BCYRX5U5FRQQWOGEDETPWSROFA5CNFSM4HM6WCKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODV5VFWA#issuecomment-494621400, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALT4BC46P3T6IC4L5M3LRBTPWSROFANCNFSM4HM6WCKA.
DrDAJump
commented
5 years ago It seems as if there is an overall assumption in CalTRACK about the use case for meter-based savings will be aggregation. In aggregation, every individual meter is analyzed using the same modeling algorithm. Results are obtained across the population, but individual sites may have extremely good or very poor outcomes (negative savings, high site-level uncertainties, etc.). This strategy may be improved by pre-screening the population. In population-based approaches, no attempt is made to adjust any individual model for an individual meter data set. I can see that the modeling algorithm selected should be the most accurate (lowest random and bias error) across the population, and this may be how CalTRACK arrived at its hourly model.
I come to this mainly from applying advanced models with interval data from individual non-residential buildings. We prescreen the buildings and seek to develop accurate models. This often means using different modeling algorithms and making interventions in the modeling process - to account for different operation periods where the relationship between energy use and independent variables are different. We document why the operation periods are different using information from the building (systems are shut down over colder months, schools have vacation weeks, etc.). We identify NREs in the data and determine their cause - again based on building information. Our method requires knowledge of the building to make justifiable adjustments and analyses using the meter data. Interaction with the building is typical of non-res projects in general. This contrasts with aggregation methods, which do not require individual building knowledge (or much less), nor do the aggregation methods make interventions on any one model in the population.
If CalTRACK’s use case is about aggregation only, it should make this clear. If the claim is that it can be used for site-level M&V (res and non-res), it must acknowledge its models are one of many options and focus on the model acceptance criteria. I read the technical appendix cited for the CalTRACK monthly-hourly method and see it was tested on residential data; there is no mention of non-res data (again, please show me where the non-res testing results are).
Another issue, maybe for the next round: the ASHRAE FSU formula applied with hourly data has been shown to be uninformative by recent LBNL research (PG&E, https://www.etcc-ca.com/reports/energy-management-and-information-system-software-testing-methodology-and-protocol). Site-level uncertainties estimated using this formula for hourly models are underestimated. The CalTRACK methods use this formula to propagate site uncertainties to aggregate totals. The aggregate uncertainty will be underestimated for hourly models.
As an example, here is a building energy use profile for a school. A daily time interval is used. You can see that the school has many out of session periods where the energy use is low. These periods correspond with periods listed on the school calendar. Two different modeling analysis were performed, one without accounting for different operation periods, and one that did account for them. You can see how much better the model that accounted for the operation periods follows the data. The GOF statistics are also provided.
Its worth mentioning that if we modeled these types of facilities that have distinct operation periods with only one modeling algorithm, and included no modeling interventions, they would not qualify under the CalTRACK process. However they are fine for site-level NMEC approaches.
Without Operation Periods Considered
With Operation Periods Considered
Goodness of Fit Metrics
lwebster101
commented
5 years ago Having site-level methods available and inclusive within CalTRACK is critical if it is to be successfully applied to commercial buildings. These projects often require substantial project-level accuracy, and commercial sites follow seasonal rather than monthly behavior schemes.
The WG email sent 10/22/19 provided updates on all of the open issues except #128. What is the status on this important issue? (Please respond via GitHub as I am unable to attend the WG meeting today.) Thank you
philngo-recurve
commented
1 year ago Closing stale issue in preparation for new working group
Prerequisites
Description
The discussion and testing behind the CalTRACK hourly model seems to be based largely on residential applications and data. For non-residential applications, generally the energy use patterns are more predictable using any number of modeling algorithms. We've had success starting with OLS regressions, ASHRAE's inverse modeling toolkit (also known as change-point models), and LBNL's time-of-week and temperature (TOWT) algorithm, applied at both hourly and daily time intervals. New methods continue to be developed (e.g. LBNL's gradient boosting machine algorithm). From a non-res perspective, I don't see the basis for requiring 12 TOWT hourly models for each month of the year (as described in 3.7.5), including the previous and following months, weighted at 0.5. For non-res, that seems arbitrary (I see in 3.7.5.1. that a 365 day model may be used if certain thresholds are met). I don't see that this has been tested with non-res data. Please correct me if I missed the issue where this has been explained.
I can see an approach of using different operation periods, rather than different months, for separate modeling. For example educational facilities have in-session and out-of-session operation periods throughout the year. Separate models may be developed for these periods, or perhaps indicator variables used, and these techniques improve overall model goodness-of-fit.
Existing public domain modeling algorithms (ASHRAE IMT, LBNL TOWT, LBNL GBM) have been tested extensively and results published (and cited on CalTRACK). I don't think there is a 'one size fits all' algorithm for both res and non-res buildings, though perhaps a majority (of non-res bldgs) in California are regularly scheduled and temperature sensitive, which may point to algorithms using time of week and temperature as independent variables.
The CalTRACK descriptions should accommodate different modeling approaches for different building types. This means adding to the existing modeling methods described in CalTRACK beyond the single algorithms based on monthly, daily, or hourly.
Because non-res buildings are more 'predictable' than res, the algorithms used to model them have some advantages. This is an issue for GRID, but the point is that techniques are emerging for identifying and quantifying non-routine events within the modeling algorithm. Some techniques use indicator variables - which are additional independent variables in OLS, change-point or TOWT models, another method was introduced by LBNL using its GBM algorithm (https://github.com/LBNL-ETA/nre). My point is: modeling algorithm selection should not be based on goodness-of-fit criteria alone.
Proposed test methodology
Another issue (#117) for this round of CalTRACK suggests a fixed-effects model (based on month) and testing methodology. I support this suggestion and propose we run comparative tests for existing modeling algorithms (TOWT, change-point, GBM, fixed-effects), for non-res buildings. The goodness of fit criteria should be expanded to include MAPE (monthly average percent error), and NMBE.
Acceptance Criteria
I'm proposing the text of CalTRACK , after comparative testing, allow established modeling approaches in addition to those listed, with perhaps some information or guidance on the advantages of using a particular algorithm for a specific site-level Option C application.