Decision Proposal 195 - Candidate Usage End Points

[CDR-API-Stream]

This decision proposal contains a recommendation for the candidate URIs and end points for Usage Data. This proposal has been developed with the aid of AEMO.

The decision proposal is embedded below: Decision Proposal 195 - Candidate Usage End Points.pdf

This consultation will be open for feedback until the 24th September 2021.

[SelenaLiuEA]

Hi All,

EnergyAustralia has prepared the attached slide pack to facilitate discussion on the issue of providing Historical Metering Data. The pack outlines scenarios, issues and possible solution options with some discussion of the pros and cons for each option.

We invite any feedback from stakeholders, including their preferred option.

Thank you

EnergyAustralia - Option Analysis - CDR - Historical Data Issue 3 August 2021.pdf

[RobHale-RAB]

Thanks for sharing this information @SelenaLiuEA - https://github.com/ConsumerDataStandardsAustralia/standards/issues/195#issuecomment-891507241 - some interesting challenges here and you've done well to describe them. Just reflecting on whether we encountered something similar in banking and whether we could save some pain for you. I am not familiar with the energy industry but would suggest the following may be useful when considering the options:

1. Can we get some data on how often customers switch energy retailers? If we understand the data here, it might make a difference and I suspect some use cases could work with 12-24 months worth of data. My banking data from 2+ years ago is not really considered relevant today and I wonder if energy data is similar.
2. Option 4 might appear to be a compromise but it also seems the easiest to understand and therefore communicate. Point 1 above might mean it still provides value to many consumers.
3. In banking, once I end my financial institution relationship, I no longer have an active customer ID, I cannot authenticate with that bank as a DH and therefore I cannot retrieve my data from them. Are we saying we would like energy to be different? I'm not sure I understand why that would or should be the case so it would be good to have a clear and agreed response to this.
4. Try not to over-engineer the solution in order to meet every conceivable outcome - I think we made this mistake in banking with joint account ownership structures. There is value to be had in starting simple and adding features and depth over time. Option 4 might work for 75% of consumers so don't discount that as a starting point.
5. Keep the consumer consent process as simple as possible - don't expect them to recall prior retailers. In fact, I'd caution against even giving them the option as that will create a more complex UX for everyone.
6. Retrieving energy data associated with prior addresses for the same consumer might result in a skewed energy data profile so I would question the relevance of that data to current consumption models in some cases.

[momentumrb]

Many thanks for putting these options together Selina. As mentioned by robhale-rab above Momentum Energy is also of the view that Option 4 delivers the simplest and most programmatic approach. It is important to understand and not place to much emphasis on the past consumption of a property when trying to determine the most appropriate energy product for a consumer going forward. Forecasting the future energy loads is far more important as appliance changes and number of occupants can make a significant difference to the energy loads going forward. The following questions should be considered when determining future residential energy loads :

• will people be at home all day in the future?

• will there be the same number of occupants (maybe a new born) living at the property, ?

• is the appliance mix likely to change in the near future?

• are the occupants planning any extended holidays going forward?

• are any energy efficiency improvements or solar installations planned?

  Industry should not be required to build and develop complex high cost systems and processes required to address the other options suggested in the EA paper particularly when their value is questionable and the likely use of this CDR facility is unknown.

Thanks for providing the facility to comment.

[jsaxon-red]

From our perspective Option 4 from the EA Options Analysis remains the preferred approach.

Fundamentally, this is due to Option 4 being required as a foundation even if another option was required, given that non-consumption history would not be shared (i.e. Billing or Payment history) and in such cases a Consumer/Retailer consent is required for each Retailer a given Consumer may have had in the past.

In addition, it seems reasonable that at the very least Option 4 should be available in addition to any other Options, should a different approach be selected.

Finally, Option 4 does not require a new consumer transfer structure and protocol, which to date seems to have been descibed at a very high level with a notable number of concerns and issues that would need to be addressed before a fully considered implementation could be agrred upon, which furthers the risk that no reasonable solution is found or that the implementation of CDR as a whole is delayed further

As such, we believe a prudent approach is to implement Optioon 4 as described, with further consideration possible at a later date if it becomes evident that consumer needs are not reasonabely met.

[PratibhaOrigin]

Thank you Energy Australia. The paper sets out some clear issues and options in relation to historical meter data.

As highlighted in the paper, the energy sector is unique in that the identifier of a consumer’s consumption data is a meter – which is not unique to the person themselves but the property at which they are consuming energy. The current energy retailer is the only person in the market today who can match a meter number (National Meter Identifier or NMI for electricity) or account with the authorised consumer at the account. That is, a meter will continue to record consumption regardless of who is consuming on the meter. The market cannot differentiate between 1 ) consumers who are using on the meter, 2) change of occupancy nor 3) know whether there has been a change in usage (ie installation of solar, batteries or an increase in residents in the property).

Given the set-up of the energy sector – Origin supports Option 4. Access to meter data should be limited to a current active account with the current retailer. This option will ensure consumers only receive meter data relevant to their consumption and premises. This options minimises both security and privacy risks of incorrect meter data being provided to a consumer.

Option 3 We have some concerns with Option 3 – this is where AEMO would record customer movement in MSATS, and allow current Retailer to send requested Historical Meter Data belonging to previous Retailer(s), where customer consents. We have privacy and liability concerns with this option as it is requiring a retailer to pass on meter data for a consumer for a period for which it has no record as to whether the meter data is relevant to them.

Outstanding questions with this option: • The information that AEMO would capture under this option? It is proposed to capture any personal information? Or only change of occupancy? • Who is liable if the incorrect meter data has been provided? If AEMO only records whether there have been change in occupant and not customer details – AEMO won’t know who the customer is. The current retailers will not be able to verify either as it is not their customer. Further, the previous retailer has not been consulted as to whether the customer was actually at the premises. • How would AEMO capture Domestic Violence flagged accounts or accounts that have special instructions for data not to be shared?
• How would it work for aggregated accounts where there are mixture of meters and sites on the same account? How would the consumer request the meter data relevant to just one meter or would it need to be the multiple meters that are on the premises? How will AEMO know? • How will AEMO know when there is a change in tenant? The account is transferred from occupant to another? • How are manually read or self reads to be dealt with? Will this information be available with AEMO as well or only previous retailer?

The energy sector has high levels of moves, updates to contact and account details and consumer churn (change retailers) daily. There can be delays with the information being transferred from AEMO to retailer (these delays occur currently with everyday transactions). There is always a risk of a timing mismatch between that data update and processing a consumer data request. The result could be a consumer receiving consumption data for a property that is not actually theirs.

Further, the release of raw interval data (type 4) can provide significant amount of intelligence about a premises. The times that a person leaves the premise or the days no one is home. The inadvertent release of this data to a non-occupier or consumer of the energy can have privacy and security risks for a consumer.

Options 1 and 2 have their own complexities that would need to be worked through. How authentication could occur across multiple retailers and what happens if consumer details have changed and authentication cannot occur across multiple retailers.

The use cases and needs for historical meter data needs to be worked through. We should not over complicate the CDR framework for a small number of use cases – over complication is costly for consumers.

[gurchan-AGL]

Thank you @SelenaLiuEA and the team at Energy Australia in putting together the options analysis.

AGL Energy supports Option 4 as it represents the most programmatic approach.

[MGiampiccoloSimplyEnergy]

Thanks for putting this together Selena,

Simply Energy also supports Option 4. We agree that this is the simplest approach and raises the fewest number of risks.

In addition to the questions raised above, if Treasury were to seek to implement Option 3 they should consider the precedent this would set for other CDR industries. For example, should telco providers also be required to make available their past customers data usage history through the CDR?

[CDR-API-Stream]

Thanks to EA for their very valuable submission. The feedback would seem to indicate the that the approach defined in option 4 (to limit usage data to the current retailer relationship) is the best path, at least for now. The DSB concurs with this position based on current information.

If MSATS was modified to allow for AEMO to be able to confidently determined length of ownership of a NMI then option 3 would become a possibility and may be able to provide consumers with a better experience. This could be assessed in the future in response to the evolution of MSATS, however.

[CDR-API-Stream]

Note that we have just published the decision proposal for this consultation (in the issue description).

[PratibhaOrigin]

Thanks for the opportunity to provide feedback on this topic. Few queries /comments below –

1) GET /energy/electricity/servicepoints/{servicePointIdentifier}/usage

Will {servicePointIdentifier} be a NMI when the retailer calls this API ?

2) Are the 2 parameters - {servicePointIdentifier} and {servicePointId} same or different?

• GET /energy/electricity/servicepoints/{servicePointIdentifier}/usage • GET /energy/electricity/servicepoints/{servicePointId}

3) For the API - /energy/electricity/servicepoints/usage – In the response payload , when AEMO responding , servicePointIds (Array of specific servicePointIds to obtain usage for) will be instead array of NMI IDs.Is this correct ?

4) For the Meter Read Common Type in the response payload , when AEMO responding , servicePointId will be instead NMI ID. Is this correct ?

5) On the historic usage data , just confirming based on above discussion –Only the usage data the customer shares with current retailer is only in scope and any historic data with previous retailers not applicable for day 1. Is this correct ?

[EnergyAustraliaBA]

Hi. Further feedback from EnergyAustralia as follows:

1. The proposed data structure doesn’t included demandCharges. We request the inclusion of ‘demandCharges’ in the API calls made to AEMO.

• The Metering Data payloads defined to date for mass-market will not support the accurate calculation of Demand charges. Demand charges are substantially different from and are measured in different units than usage charges. It is very common for C&I customers to have network cost components that contain Demand charges.

• For mass-market customers, network tariffs also increasingly include a Demand component as well as usage and supply charges. As mass-market bills are often bundled, the Demand charges are increasingly included in retail charges. This will need to be resolved before the standards are published in final form and before the implementation timeframe commences.

• Another complexity of C&I Metering Data is that many large electricity users have CT (current transformer) metering, that allows the electricity current to be stepped down, so it is not too high to pass through the meter. The customer’s actual usage is determined by applying a multiplier to the indirectly metered value, which also is not reflected in the current payloads.

2. ‘unitofMeasure’: is limited to kWh however C&I meters also record reactive energy (KVARH) which forms part of the components for bills

3. ‘basicRead’ meter read data: This is a positive value when there is consumption. This will be a negative value if export into the network. Perhaps update the description similar to the ‘aggregateValue’ which mentions “if negative it means net export”.

4. Unclear how the structure works for an E1B1E2 NMI Configuration (3 data stream suffixes at 1 NMI).

[johnAEMO]

AEMO would like to accept the offer of an extension for our response to allow for the Public Holiday on Friday 24th September. We will make our submission by cob Monday 27th.

[SareaCoates]

ESB thanks the DSB and EA for the work on this issue (Decision Proposal 195) and the useful discussion in this forum.

ESB does not believe that the proposed Option 4 - where consumers lose access to their historic meter data whenever they switch retailers - meets the policy intent of CDR, as proposed in Treasury’s draft Rules amendments.
Critically, it does not meet growing consumer needs in the energy market transition, as identified in ESB’s Post 2025 Market Design, where consumers face increasingly complex choices in energy services and increasingly multiple service providers. Limiting access to data needed by many parties through only one service provider creates unnecessary friction on competition and innovation as the market rapidly diversifies.

ESB strongly supports Option 3 as the only option proposed that meets the policy intent and priority use cases.

ESB recognises Option 3 faces technical gaps, with the need to revise existing data and switching processes to identify when account holders change. However these issues are able to be resolved, as supported in AEMO’s submission to Treasury’s Rule consultation, and ESB support these energy reforms as a priority.

ESB recognises Treasury supports resolving this concern, with the draft Rules proposing to allow AEMO to provide historic meter data for the same consumer at the same premises. If Option 4 is progressed without clear intent to support Option 3 as soon as AEMO resolves data issues, then data standards, processes and retailer systems will have to be updated, creating potential barriers and inefficiencies. The policy position that losing data access when switching is unacceptable should be clear at the outset and part of planning.

Ongoing access to meter data across retailers is critical for CDR energy benefits. A minimum of 12 months of usage data is critical for many priority CDR use cases – such as comparisons of personalised benefits from different retail plans, DER systems, efficiency options or new equipment. Strong seasonality of energy use due to space heating impacts means any advice based on less than 12 months of data can be highly misleading and lead to poor outcomes.

Without data across retailers, consumers who switch retailers will lose access to adequate advice on alternative services for 12 months. This creates friction on competition and disadvantages active consumers. As active consumers are the most likely users of CDR, this may have material impacts on CDR energy benefits overall.

Switching rates vary over time and region but based on AEMO data are between 10-20% of small consumers per year, or 2.5 to 4 million households and small businesses. Importantly these are also more active consumers who are more likely to be CDR users. A customer who switch have a 1.5 times higher chance of switching again the following year than an average customer (Quantium Research 2019 for DISER). They are likely to also have a higher propensity to investigate other energy services – such as DER, EVs and home upgrades. If these consumers lose effective access to CDR every time they switch it is likely CDR will lose significant value. The larger proportion of passive consumers may never seek a service using CDR - approximately 63% of energy consumers have not switched in over 5 years.

ESB also does not support alternative approaches proposed which increase consumer friction and reduce usability: for example where consumers must respond to multiple retailer authorisation processes, with ADR’s or AEMO coordinating requests to multiple retailers. This complicates development of ADR services and confuses consumers, likely reducing use of CDR.

These arguments are further expanded in ESB’s public submission to Treasury’s draft Rules. Cheers.

[johnAEMO]

Thank you for the opportunity to respond to the above Decision Proposal Our response is included in the attached document: Decision Proposal 195 - AEMO Response v1.0.docx

[PratibhaOrigin]

In addition to above feedback provided by Origin, some further inputs from our C&I teams -

1) Reads are only being provided in kWh unit of measure, does this mean reads on Q & K streams (kvarh) are not included? The NEM12 MDFF spec has other unit of measurements that can be provided. These should be catered as per the link to MDFF 2) Can we please get confirmation on below - if a site has multiple meters, the current “Meter Read” object only caters to one meter per service point. Does this mean that multiple read objects will be needed to be created for the same service point? 3) Basic reads will be exclusive of any meter multipliers, this has also not been considered.

[CDR-API-Stream]

We left this consultation open for a couple of extra days at the request of the community but we will now close the thread for feedback.

Thanks for all of the feedback provided. We will respond in due course as we develop the final decision for the Chair.

[CDR-API-Stream]

In response to the feedback from @PratibhaOrigin:

1. GET /energy/electricity/servicepoints/{servicePointIdentifier}/usage Will {servicePointIdentifier} be a NMI when the retailer calls this API ?

This is correct. When retailer calls any CDR API exposed by AEMO, all occurrences of servicePointID (both in request and response) should be populated with NMI instead of the servicePointID

2. Are the 2 parameters - {servicePointIdentifier} and {servicePointId} same or different? • GET /energy/electricity/servicepoints/{servicePointIdentifier}/usage • GET /energy/electricity/servicepoints/{servicePointId}

This is correct and a good pickup. We will update the standards to make it consistent.

3. For the API - /energy/electricity/servicepoints/usage – In the response payload , when AEMO responding , servicePointIds (Array of specific servicePointIds to obtain usage for) will be instead array of NMI IDs.Is this correct ?

Note that AEMO will not expose this specific endpoint as it does have any consumer context. The retailer is expected to use other AEMO exposed endpoints to construct the desire payload. For example, if an ADR requests usage data for all service points associated with the consumer, the retailer would identify the list of NMIs associated with the consumer and provide the list of NMIs in the request payload to the ‘POST/energy/electricity/servicepoints/usage’ for which usage data is required.

4. For the Meter Read Common Type in the response payload , when AEMO responding , servicePointId will be instead NMI ID. Is this correct ?

This is correct  

5. On the historic usage data , just confirming based on above discussion –Only the usage data the customer shares with current retailer is only in scope and any historic data with previous retailers not applicable for day 1. Is this correct ?

Yes. As mentioned in our previous response, based on feedback it seems the best path for now is to limit usage data to the current retailer relationship for now. This does not exclude other options from being revisited in the future if circumstances change.

[CDR-API-Stream]

In response to feedback from EnergyAustralia:

Hi. Further feedback from EnergyAustralia as follows:

1. The proposed data structure doesn’t included demandCharges. We request the inclusion of ‘demandCharges’ in the API calls made to AEMO. • The Metering Data payloads defined to date for mass-market will not support the accurate calculation of Demand charges. Demand charges are substantially different from and are measured in different units than usage charges. It is very common for C&I customers to have network cost components that contain Demand charges. • For mass-market customers, network tariffs also increasingly include a Demand component as well as usage and supply charges. As mass-market bills are often bundled, the Demand charges are increasingly included in retail charges. This will need to be resolved before the standards are published in final form and before the implementation timeframe commences. • Another complexity of C&I Metering Data is that many large electricity users have CT (current transformer) metering, that allows the electricity current to be stepped down, so it is not too high to pass through the meter. The customer’s actual usage is determined by applying a multiplier to the indirectly metered value, which also is not reflected in the current payloads.

As this feedback is closely related to the data held by AEMO with have sought feedback from them on this issue. AEMO have provided the below response re the above points:

“AEMO don’t receive the demandCharges either, so we cannot provide them. We have a demand field we have removed from standing data as it is optional, we don’t use it and it has a very low population and accuracy. However, demand can be calculated from usage data provided by this API over key time periods (eg Summer) to form the basis of a demand charge.

Large customers do have a stepping down concept as described. The meter reading is corrected by applying the multiplier to form the actual consumption before it gets to our records and it is this figure which will be provided in our API response. The multiplier is not required in the usage payload as it has already been applied by the meter readers before providing the data to us.”

2. ‘unitofMeasure’: is limited to kWh however C&I meters also record reactive energy (KVARH) which forms part of the components for bills

This feedback aligns with AEMO’s feedback to update the unit of measure as per MDFF Specification NEM12 NEM13 v2.1 ; Appendix B to cater for C&I consumers. We will update the standards accordingly.

3. ‘basicRead’ meter read data: This is a positive value when there is consumption. This will be a negative value if export into the network. Perhaps update the description similar to the ‘aggregateValue’ which mentions “if negative it means net export”.

The above feedback is not clear. The description of both the ‘value’ attribute within basicRead object and the 'aggregateValue' field both currently mention “if negative it means export”.

4. Unclear how the structure works for an E1B1E2 NMI Configuration (3 data stream suffixes at 1 NMI).

The scenario of 1 NMI with multiple suffixes/registers can be represented by providing multiple instances of the meter read common type within the reads array of the usage data payload. The servicePointID would be the same in each instance of the object within the array with the suffix’s changing accordingly

[CDR-API-Stream]

In response to feedback from @johnAEMO:

• Overall API structure – we note that the payload appears in a “denormalised” form where potentially every day of meter readings includes a repeat of the various “header” fields – meterID, registerId etc. We suggest that if the payload was “normalised” these header fields would appear as little as once in the payload followed by multiple arrays of daily meter readings making for a much smaller payload size.

The common meter read type object represents meter reading for a single day and have been designed with a view to be read/processed individually if required. Note that the schema presented in this consultation has been derived following numerous discussions with the industry (including AEMO) and as such is optimised accordingly.

o page 4, after ServicePointId we note that NMI is not included yet it is included in the NMI Standing Data Payload. Should it be included here also?

NMI has been excluded from this payload on purpose. It is only passed in the standing data payload to help establish the servicePointID, which is the CDR specific id for all subsequent payloads requests. Note that the servicePointID attribute will be populated with the NMI for any request response interaction between a retailer and AEMO.

o page 4, in order to improve the usefulness of the usage data (our experience with price comparisons and other use cases has shown that it is critical to understand the constituent elements of the usage payload at least to a level of consumption, export and controlled load), we would recommend the inclusion of the controlledLoad flag from the NMI Standing Data Payload. While this flag is available from that API’s payload, it is only available for the current meter configuration and may not facilitate the identification of controlled load in earlier meter configurations. We also note that because this inclusion is from Standing Data, it would increase the overall response time of this API.

We can include the controlledLoad attribute and align it to the standing data definition.

o page 4, unitOfMeasure. Given the change of scope to include large and commercial and industrial customers, this description should include KVAR and others from being MDFF Specification NEM12 NEM13 v2.1 ; Appendix B. This field remains a string.

We can incorporate this change noting that this aligns with the feedback from EA.

o page 4, insert at the value level in Basic reads and at the interval length level of Interval reads, a new field “directionIndicator” a Mandatory field, an enum with allowable values I or E for Import or Export from the perspective of the grid – an I is Import to the grid, an E is Export from the grid

This attribute is not required. The direction of the reads (i.e. export or import) is indicated by the positive or negative read value. This helps with cross sectoral consistency (the same pattern is used for similar use cases in banking, for e.g. credit or debit in transaction history)

o page 4, value. Just to clarify the Type. As a Number, this will be a decimal not an integer.

The number type in JSON caters for both integers and decimals

o page 5, readIntervalLength, we would recommend is Mandatory

Sure, we will make the change as requested

o page 5, intervalReads, the description needs to be modified to read “…beginning at the first interval after midnight of the…. (for example 00:00 to 00:30 would be the first reading in a 30 minute Interval)”

We will update the description as requested

[CDR-API-Stream]

Please find attached the final decision of the Chair: Decision 195 - Candidate Usage Data End Points.pdf