search

ConsumerDataStandardsAustralia / standards

Work space for data standards development in Australia under the Consumer Data Right regime
Other
321 stars 56 forks source link

Decision Proposal 103 - Electricity End Points #103

Closed CDR-API-Stream closed 4 years ago

CDR-API-Stream commented 4 years ago

With the recent publishing by the Treasury of the in-principle data sets for the electricity sector it is now possible to begin the process of design and consultation of the data standards for electricity.

To begin this process a decision proposal describing the URIs for a hypothetical set of end points for the electricity sector has been created. This proposal is attached below: Decision Proposal 103 - Electricity End Points.pdf

Feedback is now open for this proposal. Feedback is planned to be closed on 3rd April 2020.

This is first significant proposal for the electricity sector. It would be appreciated if the existence of this consultation were communicated to stakeholders that would like be interested in participating.

For stakeholders that are new to this community please sign up for email notifications here.

DannyDuong commented 4 years ago

We have some issues with the GET /energy/plans end points that we would like to bring to your attention.

Attached is a short document that describe the issue: DELWP Response to Decision.Proposal.103.-.Electricity.End.Points.pdf

Please contact me as required

dziga97 commented 4 years ago

While tariff types are covered off in "Obtain the details for a specific account that the current customer has with the retailer" in order for usage readings to be meaningful they will need to be associated with a tariff type - as Time of Use and Flat Tariff types are fundamentally different. This may be too granular a detail for this stage of the API development but needs to be incorporated in to the data structure for fundamental comparison purposes.

CDR-API-Stream commented 4 years ago

@dziga97, thanks for the feedback. This information will be very helpful for the development of the specific payloads that are accessible through the end points described in this consultation.

Would you see the tariff type in this case being an attribute of usage data (ie. accessible under the /energy/electricity/site/{siteId}/usage end point) or via the tailored tariff data (ie. the end point /energy/accounts/{accountId})?

Also, if there is any existing data structure or model that you are aware related to this that you could put us to that would be helpful as well.

CDR-API-Stream commented 4 years ago

@DannyDuong, thanks very much for your feedback. It is an important clarification to make. The assumption that the DSB is proceeding with is aligned to Option 2 in you document.

Essentially the assumed approach is that:

This approach aligns to the following principles for standards development:

The potential for two significant gateways exposing this data, being the existing EME and VEC platforms does raise some specific questions relevant to this consultation, however, that it would be valuable to obtain an opinion on from the community. Specifically:

  1. Are tariffs for electricity geographically partitioned? For instance, should the tariffs for a single retailer be expected to be different between states
  2. If tariffs are geographically partitioned what are the boundaries for this partitioning? Would this be according to state/territory boundaries, postcodes or some other arbitrary delineation?
  3. Should such partitioning be handled within the payloads (i.e. the response separates pricing by geography in the payload) or in the URI structure?
jonmilne commented 4 years ago

@CDR-API-Stream As the product owner for EME, thought I'd jump in on this one.

In relation to this point: Each gateway would advertise a base URL per retailer that they support so that the consumer can choose the retailers they would like to obtain data in relation to. Is this envisaged to be the case where are an individual retailer exposes an endpoint? Otherwise, if it were EME and VEC then should this perhaps be a query parameter rather than a distinct URL? The consideration here is that new entrants can (and do) pop up at any time and there might be more configuration work required to establish a new base URL for a new retailer each time.

In response to the 3 questions posed:

  1. Yes, tariffs vary both interstate and intrastate. This is because the retailer will typically set a tariff based on the underlying network charges that are set by the distributor in the locations they service.
  2. The boundary is typically the distribution network, that is not always aligned to a state/territory boundary. For example: in South Australia, there is only one 1 distributor that covers the majority of the state but in NSW there are multiple distributors that each cover a slice of NSW - for metro, outer urban and rural areas. Postcode alone is not perfect either, as there are postcodes that are serviced by >1 distributor (particularly in NSW/ACT). In EME we use a construct called 'supply area'. A supply area has a 1:1 mapping to a distributor, and contains a collection of postcodes that are serviced by that distributor i.e. their network. There is a 1:many relationship between postcodes and supply areas.
  3. Based on the above, the partitioning is probably best handled within the payload. That way if a user wanted to collect all tariff data for a given retailer across all states/territories, then this can be done more efficiently. But it may also be worth considering partitioning further with an additional level below state/territory for more granularity for those who need it e.g. partition by state first, and then in those cases where it is relevant (i.e. NSW) option to further partition by supply area/distributor.

I should also add - there are many factors that ultimately decide a retail tariff and an individual retailer may have a suite of variable tariffs across a range of products they offer in a given state/territory and/or supply area/distribution network.

CDR-API-Stream commented 4 years ago

@jonmilne, that is extremely helpful feedback.

Based on your comments related to tariff boundaries it would seem that it would be appropriate to defer this to payload as you describe. We will take that into account when developing the draft payloads for general tariff information.

Regarding the new entrant question, taking a holistic view across the CDR, when a new entrant is introduced they will need to be set up in the CDR register so that an accredited data recipient can present a list of retailers to a customer with logos, etc. This effort will not be alleviated by choices made regarding the URI structure for general tariff information. In addition, implementing the retailer as a query parameter would expose the use of a gateway to the recipient which would be against principle 6 listed above.

The implementation cost for a gateway of exposing many disparate end points is understood, however, and needs to be considered.

As an alternative, it would be possible for a gateway to implement the end points with retailer as a path parameter instead of a query parameter. For example: GET https://api.acmegateway.org/public/{retailer}/cds-au/v1/energy/plans

The {retailer} portion of the URI could be considered a variable by the gateway and allow for differentiation of retailers. This approach would be aligned to the existing CDR standards and would not require additional burden on any potential gateway implementation (noting that the maintenance of CDR Register meta data is a topic that will need to be addressed at a later date for the regime as a whole).

jonmilne commented 4 years ago

@CDR-API-Stream Appreciate the holistic view of participants across the CDR needing consideration here - and it wasn't something I had taken into account in my response, so thanks for bringing it to front of mind for me.

Using a path parameter as you describe makes sense and I think it could be a happy medium. Without having reviewed the existing standards in depth, one question would be: does this approach also provide a unique path parameter (or omission of one) that allows the user to make a single request for all product data, from all retailers (notwithstanding pagination may result in multiple calls to the endpoint to facilitate this)?
This is the most common use case for comparator services, where they would seek to acquire the full suite of products across the market (and perhaps within a particular state/territory depending on their service offering).

CDR-API-Stream commented 4 years ago

The standards currently allow for a single call to obtain all products for a single Data Holder with the ability to request only changes since the previous time information was requested (denoted by a provided timestamp).

In the context of electricity this does mean that a comparator site would call once per Retailer.

The need for multiple API calls has previously been of minimal concern when compared to concerns around payload size which is why pagination has been included in the standards also. If all data was available on a single API call it would still need to be paginated to moderate payload size so many calls would still result to get all of the data anyway.

jonmilne commented 4 years ago

I guess my consideration was more around a user needing to know who all the retailers in the market are first, in order to form the 'per retailer' requests. If it is a public API with no auth required, then there wouldn't otherwise be a need for a user to query the CDR register, or have prior knowledge of a market's participants, to fetch all products.

But I guess that's a problem for participant dev teams to solve :)

dziga97 commented 4 years ago

@dziga97, thanks for the feedback. This information will be very helpful for the development of the specific payloads that are accessible through the end points described in this consultation.

Would you see the tariff type in this case being an attribute of usage data (ie. accessible under the /energy/electricity/site/{siteId}/usage end point) or via the tailored tariff data (ie. the end point /energy/accounts/{accountId})?

Also, if there is any existing data structure or model that you are aware related to this that you could put us to that would be helpful as well.

The definitive tariff type should be part of the '/energy/accounts/{accountId}' - this information is currently in MSATS and sometimes hard to reverse engineer from current usage data. For comparison purposes comparison sites would only consider offers of equivalent tariff type for particular customers.

jonmilne commented 4 years ago

@dziga97 for clarity, when you say 'tariff type' do you mean type of tariff with respect to product configuration i.e. single rate, time of use/flexible or the network tariff that is applicable to the site itself?

If it is the former, then there is no reason a comparison site shouldn't present all alternative products of varying configurations that the consumer might be eligible to acquire. Provided no other barriers exist (and there may be some in certain cases with respect to metering config), a consumer on a single rate tariff could potentially acquire an alternative product with a time-of-use tariff or vice versa. With interval metering data, the calculations for price x usage become 'tariff type' agnostic. Granted, without interval metering data this becomes more difficult, but not impossible.

This in itself isn't a discussion for standards development, but I think it is important that use cases are considered appropriately and holistically.

DannyDuong commented 4 years ago

General comment This document covers quite a few end-points.

Would it be more helpful if we prefix our discussion with the dataset or end-point that we are addressing? As some terminology can have slightly different meaning between datasets. e.g. Tariff type

dziga97 commented 4 years ago

@jonmilne I think the product config tariff is important but I know from experience running the Transformer switching product at CHOICE that sometimes retailers would not switch customers to particular offers after doing an MSATS lookup about the site - I can't recall the tech details.

Also we found that currently it is difficult and time consuming for customers to actually switch tariff types - and it can be difficult mathematically to compare a TOU with single rate tariff - point taken on interval metering data though. Rule of thumb with current offers seems to be that single tariffs are more competitive than TOU though according to our research on about 20,000 bills.

Point taken though that all potential offers should be included in the API

On Tue, Mar 17, 2020 at 1:33 PM jonmilne notifications@github.com wrote:

@dziga97 https://github.com/dziga97 for clarity, when you say 'tariff type' do you mean type of tariff with respect to product configuration i.e. single rate, time of use/flexible or the network tariff that is applicable to the site itself?

If it is the former, then there is no reason a comparison site shouldn't present all alternative products of varying configurations that the consumer might be eligible to acquire. Provided no other barriers exist (and there may be some in certain cases with respect to metering config), a consumer on a single rate tariff could potentially acquire an alternative product with a time-of-use tariff or vice versa. With interval metering data, the calculations for price x usage become 'tariff type' agnostic. Granted, without interval metering data this becomes more difficult, but not impossible.

This in itself isn't a discussion for standards development, but I think it is important that use cases are considered appropriately and holistically.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ConsumerDataStandardsAustralia/standards/issues/103#issuecomment-599846447, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF3FRO25OHW5O4KIDGDLZ63RH3OPVANCNFSM4K5LLBWA .

CDR-API-Stream commented 4 years ago

Due to the recent developments across the Australian society, and in response to participant feedback received out of band, the DSB has decided to extend this consultation for at least another week.

Kat-AGL commented 4 years ago

Hi all Please find attached AGL's submission on electricity end points. We appreciate the additional time provided to stakeholders, unfortunately the business is prioritising COVID-19 related work and is unlikely to be able to consider this matter in any more detail in the next week. Please take care of yourselves and others, Kat AGL submission - electricity endpoint URIs - 27 March 2020.pdf

SelenaLiuEA commented 4 years ago

Hi All EnergyAustralia appreciates the extended time frame to comment on this issue. Is it possible to post a diagram of the proposed Entity Relationship Assumptions on this Stream to facilitate industry feedback (i.e. the assumptions listed on pages 2 & 3 of the Decision Proposal)? The diagram should show the linkages between the concepts and what relationships are one to one and/or one to many. Thank you Selena

mrOrigin commented 4 years ago

Hello All, Origin Energy has reviewed the document and we submit the attached comments, concerns and questions for the project team. Regards Paul Magee & The Origin Team :)

Origin Energy - Electricity End Point URIs comments Final.docx

commbankoss commented 4 years ago

Given the sensitivity of hardship data, the CDR Rules and Standards should not require data holders to provide hardship data content that is more expansive than their existing regulatory obligations.

We further recommend that any decisions regarding the inclusion of hardship data in the CDR should not be made until further consultation is undertaken to consider the relevant privacy and security issues.

At a minimum, consultation should cover the risks and impacts to vulnerable consumers and be inclusive of industry partners, consumer groups, and advocacy organisations.

We support AGL's recommendation that hardship data should not be considered by Data61 until such time that the Government publishes the full scope of designated data-sets.

SelenaLiuEA commented 4 years ago

Hi All, please find attached EnergyAustralia's submission on this Decision Proposal. Regards, Selena EnergyAustralia - Github consultation - API end points 3 April 2020.pdf

CDR-API-Stream commented 4 years ago

Thanks everyone for the strong participation in this consultation. The DSB will review the responses and will provide additional commentary here.

A workshop to work through the implications of this end point structure and the underlying data sets is being organised for the week starting 27th April.

In addition, a series of issues will be raised to allow for the underlying payloads to be worked through and the proposals for these issues will be iteratively published.

CDR-API-Stream commented 4 years ago

The feedback provided has now been reviewed by the DSB. The feedback was extensive and provided by multiple stakeholders which is very much appreciated. It has highlighted a number of specific areas where we can, as an emerging community working on the standards, focus our attention.

A series of separate posts will be made below addressing the feedback categorised, as much as possible, by topic. We will then again open this thread for comment for another week so that participants can clarify their feedback if our interpretation appears to be incorrect or the key points misunderstood.

CDR-API-Stream commented 4 years ago

The purpose of standards consultation

A number of respondents indicated that the purpose of the DSB consultation process was unclear, that it should be paused and that technical working groups should be established. This indicates that we have failed to articulate well enough the purpose and intent of these consultations in the electricity sector.

In response to this observation this post will elaborate on the intent and purpose of this consultations and the ones that will follow:

Consultation Intent

In the absence of a finalised designation instrument and rules for the electricity sector no binding standards can be made for the sector. This means that the consultation we are currently conducting is not to make final binding decisions but to explore the complexity of the data that is likely to be designated and uncover the areas of complexity that may not be obvious, even to industry experts. The feedback obtained to date is an already an indication of this. The subsequent posts on this issue thread identify a number of areas where it is clear that further consultation is needed before a firm standard can be contemplated.

The previous experience of the DSB is that this preliminary work of understanding the problem space cannot be compressed and will take an extended period of time. It is also the experience of the CDR regime as a whole that the consultations conducted on the technical implications of various assumptions can highlight policy issues that all parties were previously unaware of. This can be valuable input into the process of defining both the designation instrument and the rules (although both of these artefacts are the responsibility of other agencies and outside of the scope of the DSB).

As a result the Data Standards Chair has asked that the consultation process continue but with due consideration to the constraints and pressures created by Australia’s response the current health crisis.

The goal of these consultations at this stage is to build a hypothesis of the standards that can then be tested against the emerging rules and designation instrument by all participants and amended accordingly.

Working Groups

A number of respondents suggested that working groups be initiated to start developing the standards. To clarify, that is the intent of these GitHub based consultations. The participants responding on GitHub are the technical working group for the standards. Involvement is voluntary and, beyond an expectation that the published rules of engagement are adhered to, open to anyone with interest.

This approach has evolved from the geographically distributed nature of the stakeholders involved in the CDR regime and the need for all feedback to be equally considered without a bias to participants based in the major cities. As a result, all consultation and work is conducted by the DSB online and in public. Recommendations made to the Chair will only be based on feedback that has been publicly exposed for comment. While the DSB is happy to receive feedback via email to cdr-data61@csiro.au we will request permission to publish the feedback, or at least the salient points, on GitHub before it can be considered.

Any organisations that wish to influence and contribute to the standards are therefore encouraged to identify appropriate team members in their organisations to monitor and contribute to the consultations on GitHub, in accordance with their internal communication policies.

Workshops

Feedback also suggested that workshops would be a good idea. It is part of the process of the DSB to hold workshops on specific topics as well as information session specifically for questions and clarifications. It is our intention to do that for the electricity sector. The current requirements for social distancing have, of course, made a physical workshop impossible so we have been working on how to conduct such a collaborative workshop online.

Our first workshop has been scheduled for the 29th April and registration is now open at this link.

Registration for the workshop is requested due to the practicalities of running online collaboration (tool licensing and setup for instance) as well as to ensure that a broad cross section of participants is aware of the workshop and planning to attend. In the past workshops have filled up quickly so please register early.

Note that no decisions will be taken in this, or any future workshop. The results of the workshop will be brought back to GitHub for open comment and feedback prior to any recommendations being made to the Chair.

Work plan

Requests for an understanding of the forward plan for consultation have also been received. The difficulty in proposing a forward plan is that the areas requiring the most consultation are not yet known. The creation of a plan was dependent on this first consultation.

Now that feedback has been received, however, a series of consultation items will be pre-staged on GitHub to indicate what topics are likely to be considered in the future and in what order. As more feedback is received, workshops are conducted and the problem spaces are better understood specific milestone dates for standard drafts will be set.

The first tranche of these issues will be raised after these responses are posted.

CDR-API-Stream commented 4 years ago

Hardship

A number of respondents provided feedback that hardship data is sensitive and should not be included in the consultation process.

Hardship data has been included in this consultation, and will be included in future consultations, for the following reasons:

CDR-API-Stream commented 4 years ago

Additional Account Holders

The assumption presented that each account held with a retailer was owned and accessible by a single customer, as presented in the proposal, is clearly incorrect based on the feedback provided.

The feedback, in summary, has been understood to indicate that:

This is very helpful feedback and indicates that this topic warrants specific focus and consultation.

On a side note, the feedback that an account in banking and electricity sectors are completely different concepts is understood. The approach presented in this proposal for electricity accounts is considerably different to the approach adopted in the banking sector for CDR as a result of this difference.

CDR-API-Stream commented 4 years ago

Entity Diagram

Feedback that a diagram describing entity relationships would be helpful was received. It was decided to describe the assumptions in this first consultation using language only for accessibility reasons. Entity relationship notation can sometimes be hard to consume for non-technical people.

The need for a visual representation of the assumptions is understood, however, and this will be included in future proposals and the scheduled workshop.

It is noted that the response from Energy Australia included such a diagram which was very useful and could be referred to by other participants.

CDR-API-Stream commented 4 years ago

Tariff Data

The presentment of general and specific tariff data is an area that the initial assumptions made in this proposal may not hold. This was highlighted in multiple responses.

This is another area where specific focus and consultation is likely to be useful.

CDR-API-Stream commented 4 years ago

Site vs NMI vs ??

The NMI (National Metering Identifier) as an identification number that represents a metering location or site. It is not, in itself the name of the site, only the identifier used to uniquely identify the site.

In the technical standards the API definitions will need to include NMI (as a unique identifier) but it will also need to identify the overall entity that the NMI identifies. Feedback was provided that this is not a meter (as there can be more than one meter at a location identified by a NMI). It is also not a physical location per se, as there can be multiple NMI identified points at a single physical address.

The term site was used in the proposal as an initial suggestion but this seemed to be assumed to be synonymous with location or address in feedback.

One respondent suggested the term servicepoint as an alternative which is an option to be considered.

The naming of this entity appears to be a specific issue that will need to be addressed through further consultation.

CDR-API-Stream commented 4 years ago

Technical Feedback

A number of respondents provided specific technical feedback. This kind of feedback is extremely welcome as it helps drive the specifics of the standards forward.

A number of items of technical feedback are therefore responded to below.

Technical feedback from Energy Australia:

On page 6 of the Proposal there are references to end points like “GET /energy/accounts/{accountId}/payment” and “GET /energy/electricity/site”. Path resources should be in plural form for all resources e.g. payments instead of payment.

In this case the path represents a singular attribute of the entity and not an entity collection. payment in this case represents the payment method that a customer has nominated for a specific account not a collection of payments.

It may be that the term payment is confusing and should be replaced with a more specific name such as paymentmethod. Suggestions are welcome.

The payment URI on page 8 of the Proposal is ‘/energy/accounts/{accountId}/payment’. However, the description only refers to payment method. Should this refer to payment details as well or should there be a separate URI for payment details? Further, the description for the URI ‘/energy/accounts/billing’ mentions payments and charges, which would appear to be more appropriate for the URI ‘/energy/accounts/{accountId}/payment’.

Payments are proposed to be included along with charges under the billing path as described in the description of that end point.

In relation to the URI ‘/energy/accounts/{accountId}/billing’ on page 9 of the Proposal, is “accountId” considered a PII data point (applied to other path parameters)? If so, please avoid (use synthetics) to avoid logging at termination SSL/TLS termination points

An Account ID (or customer number) is not considered personally identifiable information in most contexts or industries. That said, it is often sensitive.

In the context of the CDR, to ensure all possible sensitivities are addressed, a set of standards for ID permanence is defined in the ID Permanence Section of the high level CDR standards.

It would be expected that accountId in the above path would align with these requirements and would be a token that allows identification of the account without being an account identifier that is useful in other channels and contexts.

The URI ‘/energy/electricity/sites’ on page 10 of the Proposal has a title of ‘Obtain a list of electricity meters’”. Sites (or NMIs as defined by the Proposal) and meters are not technically the same.

Noted, thank you. This will be addressed in future iterations.

The URI ’ /energy/electricity/sites/{siteId}’ on page 10 of the Proposal has a description that includes ‘This end point provides detailed information for a specific meter’. Again, a site can have multiple meters so this should be changed to reflect this possible arrangement.

Noted, thank you. This will be addressed in future iterations. The fact that a NMI identified location may have multiple meters is identified in the assumptions.

Technical feedback from AGL:

  • Response/ Request Structures of the API calls need to be standardized for design level discussions.
  • Need to add security authentication standards for API Calls and data access.
  • Are we adding API standards around testing, documentation, automation, versioning etc ?

The high level standards have already been defined for the CDR and these will be applied to each sector as they have already been applied to the banking sector.

An information security profile has also been defined for the CDR. This profile is built upon common technology standards and approaches and was designed with the intention of being cross-sector so that recipient implementations could be leveraged as the CDR expands. It is therefore expected that this profile will also be applicable to the electricity sector, however, specific considerations related to authentication may warrant amendment. These issues will be the subject of future consultations.

Is there a decision register for architecture or design choices that are being reviewed and agreed upon Yes. The standards issue tracker (ie. the site this response is being posted in) is the decision register for the CDR standards. All decisions taken by the Data Standards Chair are documented in this issue tracker.

Do the API standards cover 3rd party to AEMO and AEMO to retailer?

The current assumption for consultation is that the DSB will be defining the standards for the sharing of data between an Accredited Data Recipient and the designated Gateway (ie. AEMO).

Is there any further documentation around what Energy industry assumptions have been made and haw have these assumptions been validated?

No. This consultation is the validation process, however.

CDR-API-Stream commented 4 years ago

No final decision will be presented to the chair for this issue. End point structures will be further consulted on in the context of the specific data cluster consultations and these will result in standards decisions. The feedback provided to this consultation will be incorporated into the specific data cluster proposals.