Decision Proposal 109 - NMI Standing Data Payloads

[CDR-API-Stream]

The decision proposal for URIs and payloads for NMI Standing Data is attached below: Decision Proposal 109 - NMI Standing Data Payloads.pdf

Feedback is very welcome on this proposal. Consultation has been extended and this thread will remain open for feedback until the 19th of June.

As previously discussed in various forums, the practice of the DSB is to respond to feedback incrementally before the consultation is complete to promote an interactive style of consultation. Participants are therefore encouraged to provide feedback earlier in the consultation process so that the community can work together to a consensus outcome.

[gavin-csiro]

CSIRO Land and Water developed the Water and Electricity Supply and Consumption Markup Language in 2015. It contains a conceptual model and a physical model of data to be shared. https://www.wescml.org/ The format was developed in consultation with Victorian electricty and water distributors. It was designed for use in individual and aggregate data sets. Aggregate data for multiple states is available in AURIN and CSIRO Energy's NEAR project.

[CDR-API-Stream]

@gavin-csiro, thanks for providing the link to the WESC. The proposal for consultation is based on the data currently held with AEMO. It would be great to get your insight as to how well the proposal and the WESC align and if there is space to leverage the WESC.

On first read it would seem that the WESC will also be useful in defining consumption structures to allow for future extension to other utility types. Is this a valid assumption?

[mannharleen]

The pdf mentions "Feedback Conclusion Date: 12th May 2020". Is'nt it 12th June ?

[SelenaLiuEA]

Good afternooon all,

A few initial comments/questions:

1. We would expect to see a Request payload for both the High level Service Point list and Service Point Detailed data list (including a time period), but this is marked n/a.

2. Re the Response Payloads for both the High level Service Point list and Service Point Detailed data list, there is one start and end date item which seems to relate to the effective date of the service point. In the CATS tables (CATS_METER_REGISTER, CATS_NMI_DATA, CATS_NMI_DATA_STREAM, etc), the individual items of data e.g. classification of small vs large, will have their own start and end dates. These individual data effective dates would seem to be relevant i.e. they will tell you when a meter installation type was installed at a NMI. Can DSB please comment on this issue.

3. Payloads relate to NMIs, but how will this reconcile with the Customer when multiple customers may be linked to a NMI over time, or where a customer has multiple NMIs over time? i.e. how will the data be connected with the customer making the CDR request? (This is probably broader than this proposal). Does this link into AEMO's MSATs standing data review?

Thank you Selena

[CDR-API-Stream]

The pdf mentions "Feedback Conclusion Date: 12th May 2020". Is'nt it 12th June ?

Yes. That is correct. Apologies for the typo

[CDR-API-Stream]

@SelenaLiuEA, thanks for your feedback. Responses to your three questions are posted separately below so they can be quoted and responded to if needed.

[CDR-API-Stream]

In response to @SelenaLiuEA:

1. We would expect to see a Request payload for both the High level Service Point list and Service Point Detailed data list (including a time period), but this is marked n/a.

That is correct. Both of the proposed end points are GET requests which means that there is no request payload. This is standard industry practice for read only APIs (like these) for a variety of reasons. Query parameters are used for filter the response.

In this case, no query parameters were specified for time period because it was not perceived that a time dimension was relevant. Only service points currently associated with a customer would be returned, not service points associated in the past but now no longer associated. Is this an invalid assumption?

[CDR-API-Stream]

In response to @SelenaLiuEA:

2. Re the Response Payloads for both the High level Service Point list and Service Point Detailed data list, there is one start and end date item which seems to relate to the effective date of the service point. In the CATS tables (CATS_METER_REGISTER, CATS_NMI_DATA, CATS_NMI_DATA_STREAM, etc), the individual items of data e.g. classification of small vs large, will have their own start and end dates. These individual data effective dates would seem to be relevant i.e. they will tell you when a meter installation type was installed at a NMI. Can DSB please comment on this issue.

This is a good observation, apologies for not commenting on this directly in the proposal. The separate tables contained in CATS all have update dates for individual records. These date records are relevant when accessing the data in a table based structure. In the CDR, we are not providing separate end points for each CATS table, we are representing the data at the entity level with nested data. This means that data from multiple tables in CATS are being merged into a single response.

In this context it was perceived that multiple date/time stamps would be less useful and potentially confusing. The single set of date/time stamps should therefore be understood as an amalgamation of all of the relevant underlying tables and records.

Specifically:

• creationDateTime would either represent the earliest creation date of the records presented, or the creationDateTime for the base NMI record
• lastUpdateDateTime would contain the most recent update time for any of the records presented in the payload

[CDR-API-Stream]

In response to @SelenaLiuEA:

3. Payloads relate to NMIs, but how will this reconcile with the Customer when multiple customers may be linked to a NMI over time, or where a customer has multiple NMIs over time? i.e. how will the data be connected with the customer making the CDR request? (This is probably broader than this proposal). Does this link into AEMO's MSATs standing data review?

This does not link to the MSATs standing data review in any way. The current proposal is based only on the data currently held by AEMO. The proposal will be modified if AEMO data changes.

The data will be connected to the CDR consent request by the identity of the consumer establishing consent. This can be described as follows:

• Authentication (through a mechanism yet to be established) will occur
• Once authentication is complete and consent established the identity of the customer will be known to the retailer
• The retailer will know which NMIs the customer currently has access to (with no expectation of any historic association)
• The end point providing the list of service points will contain this list as provided by the retailer

Some specific points to make explicitly clear:

• These end points allow for a customer to currently own multiple NMIs
• These end points have no concept of "over time". They only represent data as it stands at the moment the API was called

If any of the statements above are invalid or should be revisited it would be very helpful to receive that feedback.

[SelenaLiuEA]

Thanks for your responses James.

Regarding:

In response to @SelenaLiuEA: ... In this context it was perceived that multiple date/time stamps would be less useful and potentially confusing. The single set of date/time stamps should therefore be understood as an amalgamation of all of the relevant underlying tables and records.

Some further thoughts:

Generally, we see the purpose of including NMI Standing data in the CDR, as helping to interpret other data set responses particularly metering data. And, potentially, assisting Accredited Data Recipients in deciding what data to request; and providing some basic information about the site.

We understand that the Data Standards Body is proposing to take the approach of time stamping on an amalgamated basis, instead of having multiple date/time stamps for individual data points. While we agree it’s more simple and less confusing, we point out that it may be useful to timestamp individual data points.

For instance, for Service Point Detailed Data one item is “Installation type” which basically refers to the meter type. If the current “Installation type” is returned in a response payload, it might be useful to know when the current Installation type was updated i.e. when the data is valid from or the “from date”. This information would help to interpret metering data, i.e. current metering data would be expected to change from the “from date” (e.g. readings move from accumulated to interval). Moreover, it can be deduced that a different meter was in place before then and an ADR may choose to not request that data. Absent the “from date”, the information might have less utility in interpreting other datasets.

As another example, the “Threshold” as defined in the National Energy Retail Regulations helps to define what type of user a customer is by consumption level. It would be useful to know the “from date” to confirm assumptions about the customer i.e. usage increased, and then on X date the customer switched from a Low to Medium threshold. A similar point can be made for “Classification” as Business and Residential.

In our examples, if changes to “Installation type”, “Threshold” and “Classification” can be deduced from metering data alone, then there might be a question as to whether it should be included in NMI standing data in the first place. Our view is that these fields are useful though, if only to define the current site.

Separately we note that “validPeriod” has a "toDate" label which is described as “The known end date for this service point. If absent then no end date has been set”. We understand that if there is an end date or abolishment date for a NMI, then this data is filled after the NMI is abolished i.e. it will always be absent until that time.

[SelenaLiuEA]

Further, regarding your response to my 3rd question around how data will be linked to the customer:

In response to @SelenaLiuEA: ...

The data will be connected to the CDR consent request by the identity of the consumer establishing consent. This can be described as follows:

• Authentication (through a mechanism yet to be established) will occur
• Once authentication is complete and consent established the identity of the customer will be known to the retailer
• The retailer will know which NMIs the customer currently has access to (with no expectation of any historic association)
• The end point providing the list of service points will contain this list as provided by the retailer

Thank you for this clarification. We note a couple of other points to complete our understanding which might be useful to others. First, the ADR will ask the customer for their current retailer, and then the retailer will be contacted for authentication by AEMO (the ACCC is considering how this part will work). In this authentication process, the retailer will know the identity of the customer and will be relied on to link the customer to their current NMIs, this will not be done by AEMO.

[mannharleen]

Feedback from Origin Energy on DP-109

There are three main buckets of concerns that we would like to raise here:

1. Understanding of Authentication, Authorization & Consent for NMI Standing data: Given that a customer may have been historically lived on multiple properties (multiple NMIs) with multiple Retailers (Data Holders). How is the authorization managed across Data Holders? How does the Gateway know which Retailer to go for Authorization? How can the consent be limited to a certain time period?

2. New terms and fields being introduced like ServicePointId: What is the value or purpose of introduction of a new field service Point ID on top of the existing NMI. Currently all data element within the internal system are based on the NMI. This would lead to complex change to internal system and will require to introduce an additional mapping between NMI and Service Point ID. Is the Service Point ID only known to AEMO or do the Retailers need to be aware of this mapping to NMI as well?

3. Understanding of the request-response flow between Customer, ADR, DH and Gateway: This is very different from Banking flows due to involvement of AEMO as the Gateway that does not have information related to the customer context.

Section wise detailed comments/queries follow:

Section on DP	Comment
None - General comments	Authentication, Authorization & Consent: What if the customer was in multiple sites supplied by different retailers? How would the authentication and authorization work? When an ADR makes a request for NMI standing data related to the customer, should it be assumed that the customer wants to share all past NMIs' data with the ADR? How can the customer revoke access/consent? How does the gateway (AEMO) know which customer belongs to which retailer and what NMIs? Will there be a mapping managed by AEMO that the retailers will have to update? What happens when the mapping goes out of sync?
	Error handling: What are the error and exception codes that are likely to occur for the 2 requests (apart from the technical errors) What scenarios are to be handled and how?
	Data quality: A number of fields like network tariff code, unit of measure, time of day, etc have been called out as String type. What level of data governance will be in place to ensure the values are appropriate. Who and how will this be validated
	Does this apply to all customers? Or only small market customers? Is it only electricity? Or do we also need to consider other products, fuels etc? This could impact how we consider the overall solution to provide data across our system environments.
Alignment To MSATS Conventions	Misalignment to naming convention will lead to additional overhead to all existing systems, people and process in processing the data. This decision should be re-assessed.
Request Payload /energy/electricity/servicepoints	What is the basis of identifying the service points to the site or customers? For example a customer could have multiple property like residential & investment property or shopping complex with multiple tenants
Response payloads for  /energy/electricity/servicepoints	What is the value or purpose of introduction of a new field service Point ID on top of the existing NMI. Currently all data element within the internal system are based on the NMI. This would lead to complex change to internal system and will require to introduce an additional mapping between NMI and Service Point ID Who owns and manages the mapping between service-point-id and NMI? Is the service-point-id something the retailer needs to be aware of?
	Service Point Status of off-market. Is this CDR framework applicable for off-Market meters. How would AEMO provide off-market related details? Do retailers have the obligation to publish any off-market meter data to AEMO market. Considering Standing data is expected to be serviced directly by AEMO (special meters installed for Solar or Demand/Response)?
	Request to provide additional information on isGenerator field. Is this representing even distributed generation like Solar panels? i.            If not can additional field be introduced to provide more details around the DERR ii.            If yes, then can additional field be included to provide details on type of generators
	Last update date time - Requires additional details on what update time is being referred to here
	Jurisdiction Code - When would type NEM be applied?
	Should the response not be an array type? How else will pagination work?
Request Payload for /energy/electricity/servicepoints/{servicepointid}	Related participants: i.    Please confirm which of the related participants are mandatory to be returned? ii.    What is the benefit of these details from CDR perspective and objective?
	Location i. Please confirm if this referring to the site address as per AEMO or expected to be Postal Address of the site? As it could be different in certain circumstance.

[ghost]

Hi Ahmed from AGL Energy,

AGL's understanding of the rationale behind the introduction of the “servicePointIds” was an alternative to the earlier “site” proposal to address a concern around ambiguity for participants that may not be immersed in the electricity sector and the application of NMI.

1. It’s unclear to AGL how this new concept would address this ambiguity?
2. Would this concept be exposed to the end consumer? Today it’s typical for the NMI be included in the bill.
3. Which entity will be responsible for the mastering & maintenance of “servicePointIds” and what the role, if any, will the retailer play?

AGL also seeks clarity on the topic of authorization and consent as raised by other contributors. We believe that API security model need to be clarified well upfront, otherwise there is an increasing risk of having to refactor some of the API definitions.

[CDR-API-Stream]

Responses to the most recent submissions will be posted shortly. As some significant submissions have come in at the end of the consultation the consultation period will be extended to allow for participants to comment on these responses. The thread will be closed next Friday unless more dialogue is requested.

[CDR-API-Stream]

In response to @SelenaLiuEA:

Thank you for the additional feedback and comments. We will take these into consideration for the final draft. In general your clarifications as to your understanding of the proposal align to the intent of the DSB.

For the specific case of the validTo field, if this is not likely to be populated for any of the scenarios covered by the CDR then we will consider dropping the field and having a valid from field only.

[CDR-API-Stream]

In response to @mannharleen, the multiple items in the feedback from Origin Energy will be dealt with in individual responses below

[CDR-API-Stream]

In response to @mannharleen, comment on first main item:

1. Understanding of Authentication, Authorization & Consent for NMI Standing data: Given that a customer may have been historically lived on multiple properties (multiple NMIs) with multiple Retailers (Data Holders). How is the authorization managed across Data Holders? How does the Gateway know which Retailer to go for Authorization? How can the consent be limited to a certain time period?

For the context of the use of the URIs and payloads proposed in this consultation the specific context should be understood to be:

1. A customer has authenticated and provided consent (via a mechanism yet to be finalised).
2. The specific NMIs that they currently have ownership of is requested by the ADR
3. The current (not historic) details of the NMIs that the customer currently has ownership of are returned

If there is a perception that the ability to retrieve historic NMI data is of value this would be good to understand as that would be new feedback. If there is value in historic data then this could take the form of historic data of the NMI data (ie. the data for NMI was different in the past) or in the for of historic ownership (ie. the customer used to own a NMI in the past but they no longer have ownership.

[CDR-API-Stream]

In response to @mannharleen, comment on second main item:

2. New terms and fields being introduced like ServicePointId: What is the value or purpose of introduction of a new field service Point ID on top of the existing NMI. Currently all data element within the internal system are based on the NMI. This would lead to complex change to internal system and will require to introduce an additional mapping between NMI and Service Point ID. Is the Service Point ID only known to AEMO or do the Retailers need to be aware of this mapping to NMI as well?

The only new piece of data introduced is the servicePointId so this will be addressed directly (all other fields are understood to be existing data in MSATS, although the field names may differ slightly). Providing an explanation for the inclusion is of the servicePointId field specifically would be worthwhile as it arises from an underlying approach applied widely in the CDR standards.

The servicePointId was introduced in accordance with the current approach of the CDR to pre-existing entity identifiers. Where an important identifier, with it’s own meaning and lifecycle, is used within an industry there is a preference to instead use a contextualised ID within the CDR standards to avoid the pre-existing ID being included in URI paths and request objects. URI parameters are often stored in web logs for intermediate infrastructure and pre-existing IDs often have privacy or data leakage implications that mean they need to be given extra levels of protection.

This also allows for future flexibility of the standards if the meaning of these identifiers (which are usually managed outside of the scope of the CDR) changes over time.

To avoid these concerns the CDR standards adopted an entity ID standard that is expected to be used for all sectors. In the Banking sector it was used to identify accounts and payments even where unique identifiers were already present.

You can review the standards applying to IDs of this nature in the ID permanence section of the CDR standards.

In response to the final query, the servicePoint ID field would be specific to a single ADR-Customer-Retailer relationship and is only relevant to the ADR and the Gateway. The customer and retailer never need to know this ID as it is specifically, by design, not able to be used outside of the context of CDR API request.

[CDR-API-Stream]

In response to @mannharleen, comment on third main item:

3. Understanding of the request-response flow between Customer, ADR, DH and Gateway: This is very different from Banking flows due to involvement of AEMO as the Gateway that does not have information related to the customer context.

From the perspective of this proposal (the URI and payloads for NMI standing data) the inclusion of a gateway actually makes no difference for the URI and payload definitions. The request made by the ADR will still be terminated by a resource server that will need to validate that the request can be executed in the context of a previously established consent. The existence of a gateway should be invisible to the ADR.

There will, of course, be implementation differences but for the purposes of this specific consultation it is the understanding of the DSB that these concerns are not relevant. If we have misunderstood this situation then we would very much appreciate feedback on the specific technical situations where the introduction of a gateway impact this proposal so we can address them.

[CDR-API-Stream]

In response to @mannharleen, on the topic of Authentication, Authorization & Consent:

1. What if the customer was in multiple sites supplied by different retailers? How would the authentication and authorization work?

Each consent is expected to be in the context of a single retailer so only the NMIs known by the single current retailer to be owned by the customer should be provided in a response.

2. When an ADR makes a request for NMI standing data related to the customer, should it be assumed that the customer wants to share all past NMIs' data with the ADR?

No, as responded to here, only the currently owned NMIs are expected to be returned. Expanding to historic ownership is something that can be considered if this will be of value.

3. How can the customer revoke access/consent?

That issue is not being addressed in this proposal or consultation. It will first be dealt with via the ACCC consultation on authentication models.

4. How does the gateway (AEMO) know which customer belongs to which retailer and what NMIs? Will there be a mapping managed by AEMO that the retailers will have to update? What happens when the mapping goes out of sync?

That issue is not being addressed in this proposal or consultation. It will first be dealt with via the ACCC consultation on authentication models.

[CDR-API-Stream]

In response to @mannharleen, on the topic of error handling:

1. What are the error and exception codes that are likely to occur for the 2 requests (apart from the technical errors)

This is an excellent question. Suggestions for likely error scenarios would be very welcome.

2. What scenarios are to be handled and how?

As stated above, feedback on error scenarios would be welcome. In addition, it is assumed that the existing error handling mechanisms for the CDR standards will apply as these were developed generically and are not expected to be specific to any single sector.

See the HTTP Response Codes section of the current standards to see how error response are currently handled.

Also, it is worth noting that a series of consultations on the handling of errors in the CDR standards are currently underway. These consultations are not considered sector specific and the outcomes will apply to all sectors.

[CDR-API-Stream]

In response to @mannharleen, on the topic of data quality:

1. A number of fields like network tariff code, unit of measure, time of day, etc have been called out as String type. What level of data governance will be in place to ensure the values are appropriate. Who and how will this be validated

These fields will not be validated via the API contract. The URIs in this proposal are designed to retrieve data that currently exists. These APIs can therefore only present the data to the level of quality that currently exists. Where specific types are available and known these have been specified but if the underlying data is free-form text then no additional validation is able to be applied.

[CDR-API-Stream]

In response to @mannharleen, on the general comment below:

Does this apply to all customers? Or only small market customers? Is it only electricity? Or do we also need to consider other products, fuels etc? This could impact how we consider the overall solution to provide data across our system environments.

It applies to all customers designated as interpreted by the ACCC rules and according to the implementation schedule to be specified by the ACCC. It is beyond the scope of this consultation to delve into these topics. These are question more appropriately asked of the ACCC.

[CDR-API-Stream]

In response to @mannharleen, on topic of alignment To MSATS Conventions

Misalignment to naming convention will lead to additional overhead to all existing systems, people and process in processing the data. This decision should be re-assessed.

This decision will not be re-assessed. It is fundamental to the principles of the CDR standards development process to establish a cross-sector regime. If there are specific fields that have been transformed in an inappropriate or needlessly complex way we would be happy to consider a different approach.

[CDR-API-Stream]

In response to @mannharleen, in response to the following feedback:

What is the basis of identifying the service points to the site or customers? For example a customer could have multiple property like residential & investment property or shopping complex with multiple tenants

This is up to the retailer.

What is the value or purpose of introduction of a new field service Point ID on top of the existing NMI. Currently all data element within the internal system are based on the NMI. This would lead to complex change to internal system and will require to introduce an additional mapping between NMI and Service Point ID

Who owns and manages the mapping between service-point-id and NMI?

Is the service-point-id something the retailer needs to be aware of?

These questions have been addressed here

Service Point Status of off-market. Is this CDR framework applicable for off-Market meters. How would AEMO provide off-market related details?

This has been included as it is a current component of the MSATS data definitions so it would be a reflection of currently held data (if it exists).

Do retailers have the obligation to publish any off-market meter data to AEMO market. Considering Standing data is expected to be serviced directly by AEMO (special meters installed for Solar or Demand/Response)?

The data holder designated for this data set is AEMO so it is currently expected that AEMO will service the data for these APIs directly although the Retailer would need to confirm customer ownership of NMIs as AEMO is not aware of these relationships.

Request to provide additional information on isGenerator field. Is this representing even distributed generation like Solar panels?

i.            If not can additional field be introduced to provide more details around the DERR

ii.            If yes, then can additional field be included to provide details on type of generators

This field is taken directly from the MSATS data so its interpretation would be aligned to the underlying field held by AEMO. If there is an additional field currently held on NMIs then this can be included. Data that does not currently exist cannot be added through the CDR standards specification process, however.

Last update date time - Requires additional details on what update time is being referred to here

This has been responded to here

Jurisdiction Code - When would type NEM be applied?

This field aligns to the equivalent field in MSATS. It’s interpretation and use would be as defined in the procedures managing that data.

Should the response not be an array type? How else will pagination work?

This is an absolutely awesome observation and I am embarrassed that I got this wrong.

[CDR-API-Stream]

In response to @mannharleen, in response to the following feedback:

Related participants:

i.    Please confirm which of the related participants are mandatory to be returned?

ii.    What is the benefit of these details from CDR perspective and objective?

All related participants held by AEMO in MSATS must be returned. This has been included as data as it is held by AEMO and designated by the Treasury. In addition, there has been no feedback to date indicating that it is data with significantly poor quality or specific privacy issues.

Beyond these considerations the specific benefit has not really been considered. It is up to the ADRs to find innovative uses for the designated data.

Location

i. Please confirm if this referring to the site address as per AEMO or expected to be Postal Address of the site? As it could be different in certain circumstance.

The site address held by AEMO. Customer postal address will be dealt with under the Customer data set.

[CDR-API-Stream]

In response to @AElharouny:

AGL's understanding of the rationale behind the introduction of the “servicePointIds” was an alternative to the earlier “site” proposal to address a concern around ambiguity for participants that may not be immersed in the electricity sector and the application of NMI.

The term Service Point was introduced for this reason. The specific field servicePointId was introduced due to the reasons explained here

1. It’s unclear to AGL how this new concept would address this ambiguity?

For existing industry participants this is probably understandable. Coming with relatively new eyes to the industry sector a number of members of the DSB team experienced this ambiguity first hand. The NMI is an ID number not the actual entity and most members of the general public would use the term meter to identify a service point (which would be technically inaccurate).

2. Would this concept be exposed to the end consumer? Today it’s typical for the NMI be included in the bill.

No. It is only used for the definition of the APIs in the CDR standards. The language presented to customers is entirely up to the ADRs that would use the APIs to create new experiences.

3. Which entity will be responsible for the mastering & maintenance of “servicePointIds” and what the role, if any, will the retailer play?

This was addressed here. Only the ADR and the terminating gateway (ie. AEMO) need to be concerned with the concept of a service point or the servicePointId.

AGL also seeks clarity on the topic of authorization and consent as raised by other contributors. We believe that API security model need to be clarified well upfront, otherwise there is an increasing risk of having to refactor some of the API definitions.

This is not in the scope of this consultation and will first be addressed consultations published by the ACCC.

[CDR-API-Stream]

This consultation is now being closed.

[CDR-API-Stream]

An additional submission from AEMO has just been received for this consultation and is attached below: Decision Proposal 109 - AEMO response 2.0.pdf