search

ConsumerDataStandardsAustralia / standards-maintenance

This repository houses the interactions, consultations and work management to support the maintenance of baselined components of the Consumer Data Right API Standards and Information Security profile.
41 stars 9 forks source link

Energy / Get DER for Service Point - allow for no data #526

Closed OpalRussAEMO closed 1 year ago

OpalRussAEMO commented 2 years ago

Description

For Get DER for Service Point (incl. SR version), for a HTTP 200 response the schema requires the data payload to include EnergyDerDetailResponse.

Where the request is valid, i.e. from the current retailer & the servicePointId is valid, but no DER data is available we need the ability to return a 200 response code without including DER details.

Area Affected

https://consumerdatastandardsaustralia.github.io/standards/#get-der-for-service-point-sr & https://consumerdatastandardsaustralia.github.io/standards/#get-der-for-service-point /Responses/200 schema

Change Proposed

Update schema to allow for either an empty data object (where request is valid but no DER data) or EnergyDERDetailResponse

DSB Proposed Solution

The current DSB proposal for this issue is in this comment

johnAEMO commented 2 years ago

Current analysis indicates that 75-80% of NMIs will NOT have a DER record.

The response for Get DER for Specific Service Points (SR) is in the form of an array so empty array elements are acceptable. However the response for Get DER for Service Point (SR) is not in the form of an array so an empty response where mandatory fields exist is not currently acceptable.

Our preferred position is either:

The first option has the benefit of providing a 200 success code so would be easily included in the total of all successful responses but is not in line with current standards.

Happy to discuss.

biza-io commented 2 years ago

Biza.io supports the introduction of a 404 Not Found error response with a new error code of DER Information Unavailable or similar. We note also that it will be necessary to consider error conditions for endpoints which accept a POST payload and consistency with other endpoints would introduce a 422 Unprocessable Entity error response.

perlboy commented 2 years ago

I believe there was some question around why 422 for POST endpoints was suggested. This suggestion was simply an alignment with what was understood to be the accepted behaviour for other bulk endpoints that were unable to return 404s in a Zendesk article: Enhanced error handling API differences.

Happy to go with an alternative but would then ask why other endpoints behaviours wouldn't be aligned?

OpalRussAEMO commented 2 years ago

As per discussion at yesterday's maintenance meeting, commenting to mention that we must have a solution to this issue in place for November. We cannot produce schema compliant responses for this scenario which will represent about 80+% of requests for DER data requests for single service points.

CDR-API-Stream commented 2 years ago

The DSB and AEMO have consulted on this issue and would like to recommend an alternative solution of defining default values for mandatory fields in the EnergyDerRecord object. This would allow a schema compliant 200 response to Get DER for Service Point (incl. SR version) API requests where servicepointid’s do not have a DER record.

The specific fields and proposed defaults are as follows:

The remaining fields do not require a default as they are either optional or an appropriate schema compliant value can be provided.

As there are no significant changes required to the standards this would not require to be categorised as urgent with the expectation of being implemented for November 15th.

The DSB welcomes feedback on the above proposal.

perlboy commented 2 years ago

The above proposal seems to be prioritising convenience for timeline reasons rather than sticking to a number of Standards and internet norms. In particular Technical Principal 1:

The API standards will adhere to RESTful API concepts where possible and sensible to do so. In particular the concepts of statelessness and resource orientation will be followed.

A 0 value explicitly defines a state for a DER record and would be a divergent behaviour for communicating a not found record both within the Standards and global internet norms. It also means there is ambiguity regarding whether a data quality issue exists or the record is indeed unavailable as well as whether that unavailability is temporary.

I question the justification for not aligning to what is universally accepted as a (potentially temporary) Not Found by way of http status 404.

Additionally, referring to Outcome Principal 5:

The standards will strive for consistency in patterns, structure, security mechanisms and user experience across sectors to facilitate the development of customer experiences and services that are able to integrate data from multiple sectors seamlessly and to reduce the cost of customer education for new sectors.

The analogy here is Get Transaction Detail which often returns 404 because many transactions have no such detail as communicated by isDetailAvailable. While the proposal was a specific error message an Invalid Resource or Unavailable Resource could suffice as well.

I'm intrigued what reason there could be, other than time, to not align with the most frequently occurring error behaviour (404) on the internet but also wary of setting a precedent that the way to communicate an unknown value is a bunch of zeros. This isn't a fixed width file format and zero values is one of the biggest data quality issues ADRs are encountering right now (ie. loose spec leads to poor quality outcomes).

Is the reason for this alternate proposal purely driven by the delivery date? I don't grasp what the technical win is for going down this path.

CDR-API-Stream commented 2 years ago

Hi @perlboy, we did consider these patterns but there are reasons we didn't suggest applying them.

The 404/422 pattern have been applied to situations where the entity defined by the ID itself is available or not (for instance AccountID where an account is no longer available for sharing). In this regard we already support the UnavailableServicePoint and InvalidServicePoint errors for DER data. If we simply applied these to the DER end point the client would not be able to differentiate between 'The Service Point is unavailable' and 'The Service Point is fine, there's just no data for DER'.

Essentially, the resource is available, but a subset of data related it just happens to be empty which is a nuance but it is a nuance with meaning. This is effectively the model we follow for bank account balances and bank transactions in that if the subset of data requested is empty then a zero value (for balance) or an empty array (for transactions) is expected.

The other issue we discussed internally with presenting an error response is that we felt it would make life unreasonably difficult for Energy Retailers trying to facilitate the Get Bulk DER end point. As this end point, when called by the ADR, doesn't have any IDs a 404/422 response would be inappropriate and could not be passed on to the ADR. This means the Energy Retailer would have to process the 422 when they call the Get DER For Specific Service Points (SR) end point resulting in multiple calls to AEMO to fulfil the request. Also, if a service point results in a 422, the status could not be cached for later invocations as DER data could appear at any time (ie. a customer can install solar panels at any time).

We also discussed the isDetailAvailable pattern that we use for bank transaction detail. We did not seek to adopt this as there is no other end point where this flag could be presented with the same scope. This means that making the flag useful would require another scope to be included in the disclosure consent which the customer may not approve.

As a result of all of this we felt that the proposed position is a trade off between the the various design principles, including the ones you quote.

perlboy commented 2 years ago

Hi @perlboy, we did consider these patterns but there are reasons we didn't suggest applying them. The 404/422 pattern have been applied to situations where the entity defined by the ID itself is available or not (for instance AccountID where an account is no longer available for sharing). In this regard we already support the UnavailableServicePoint and InvalidServicePoint errors for DER data. If we simply applied these to the DER end point the client would not be able to differentiate between 'The Service Point is unavailable' and 'The Service Point is fine, there's just no data for DER'.

I agree which is why the original suggestion was to introduce DER Information Unavailable error. As a temporary workaround to achieve November though we could temporarily use one of these error responses as it is already defined.

Essentially, the resource is available, but a subset of data related it just happens to be empty which is a nuance but it is a nuance with meaning.

So I feel like the requirements here are morphing. The original thread started out as "no DER data is available" but now we are talking about a "subset" of data. From what I understand a whole DER record is often missing because only a very small number of NMIs actually have DER records. Based on dashboard data there's only 2.6mil DER sites which is a minority versus total NMIs (>15mil?) which makes me think we should focus more on the DER record missing problem.

I'm going to assume the focus is on the entire DER record being missing however if it's just the subset and it is known this DER record exists the problem is easily solved by switching the attributes in question (approvedCapacity, availablePhasesCount, installedPhasesCount) to Optional and we'd be done with null provided.

Deliberately making it schema compliant by injecting 0 has historically been frowned upon by the DSB/ACCC so I'm not sure why this situation is different and it also doesn't allow differentiation between attributes missing and DER record not there at all.

This is effectively the model we follow for bank account balances and bank transactions in that if the subset of data requested is empty then a zero value (for balance) or an empty array (for transactions) is expected.

I don't believe this is true. The Standards don't state that a zero value is assumed for current balance only for available balance and in fact it's "assumed to be zero or positive": image

The net effect of this is that a consumer isn't incorrectly informed of an overdrawn account (ie. the bank needs to be explicit about it) which is quite important. The same goes for creditLimit and amortisedLimit where the assumption of 0 is advantageous as far as consumer protection and effective communication by a data recipient.

The empty array for transactions is accurate and if Get DER for Service Point was an array response it'd "all be fine" because it could be omitted but, it isn't...

The other issue we discussed internally with presenting an error response is that we felt it would make life unreasonably difficult for Energy Retailers trying to facilitate the Get Bulk DER end point. As this end point, when called by the ADR, doesn't have any IDs a 404/422 response would be inappropriate and could not be passed on to the ADR. This means the Energy Retailer would have to process the 422 when they call the Get DER For Specific Service Points (SR) end point resulting in multiple calls to AEMO to fulfil the request. Also, if a service point results in a 422, the status could not be cached for later invocations as DER data could appear at any time (ie. a customer can install solar panels at any time).

Yep, so the team internally at Biza.io discussed this and we agreed omitting the unavailable DER from the array would be the cleanest. Like I said happy to go with an alternative but this fresh proposal of deliberately producing stubbed DER data for missing records seems to be another divergent pattern which is only increasing the complexity for implementers.

We also discussed the isDetailAvailable pattern that we use for bank transaction detail. We did not seek to adopt this as there is no other end point where this flag could be presented with the same scope. This means that making the flag useful would require another scope to be included in the disclosure consent which the customer may not approve.

My focus on Transaction Detail was purely about its 404 behaviours. There's no List DER Records functionality although this would be useful if DER was ever not 1:1 with a NMI / Service Point, ie. DER historical changes.

As a result of all of this we felt that the proposed position is a trade off between the the various design principles, including the ones you quote.

As I stated before:

This isn't a fixed width file format and zero values is one of the biggest data quality issues ADRs are encountering right now (ie. loose spec leads to poor quality outcomes).

I think it's important to highlight and be able to differentiate whether the DER record exists but isn't complete versus the DER record not being present at all. By way of trying to demonstrate this challenge for a Consumer (cause that's what really matters here right?) let's talk about a fairly obvious use case - solar panel assessments.

Solar4Us

Solar4Us is a solar panel finance business and an Accredited Data Recipient. They utilise the CDR to gain an understanding of a particular sites installed configuration. They use information obtained from CDR to provide a customised quote.

When there is no solar things are easy however when they are assessing upgrade opportunities they must also assess whether the potential customer has an existing installation, whether that installation can be reused in some way or whether there is deinstallation costs associated with stripping out the old panels, cabling and inverter.

Beyond discovering service point information, which tells them metering installation, they also request DER information so they can make an assessment of the presence of a solar installation and what type of installation it is. Consequently they must assess, ideally without asking the Consumer, who may have no understanding of what they have installed, the following: 1) Is there any related equipment present 1) Does the location have an existing inverter 2) Does the location have existing solar panels (it is possibly to have an inverter without solar panels)

404 Scenario with fields changed to Optional

When accessing Get DER Detail the response is one of three values:

  1. 404 not found, no DER devices found at all ("Clean site")
  2. 200 Ok with DER details including approvedCapacity, availablePhasesCount, installedPhasesCount
  3. 200 Ok with DER details of null for approvedCapacity, availablePhasesCount, installedPhasesCount

The ADR is now able to:

  1. State that the customer doesn't have any solar detected
  2. Continue as per normal with an understanding of what the existing solar installation is along with additional phases are available (useful if you just addon to an existing install)
  3. State the customer has solar but can't determine current capacity and phase count and allow manual input

Scenario with fields zero'd

When accessing Get DER Detail the response is one of two values:

  1. 200 Ok with DER details including approvedCapacity, availablePhasesCount, installedPhasesCount
  2. 200 Ok with DER details of 0 for approvedCapacity, availablePhasesCount, installedPhasesCount

The ADR is now able to:

  1. Continue as per normal with an understanding of what the existing solar installation is along with additional phases are available (useful if you just addon to an existing install)
  2. Ask the customer if they have any solar at all then:
    1. Ask them if they have solar and if they do;
    2. Have them manually input the details

Given the ADR wasn't able to determine if there was an actual DER record (ie. "Something there") or if it was a data quality issue they've just introduced friction into the process and this is likely to come with a high drop off rate. In either case the potential benefit to the Consumer is impacted due to a baked in behaviour.

Conclusion

Having written this now I wonder if the proposal should actually be to introduce a 404 and allow null for approvedCapacity, availablePhasesCount, installedPhasesCount if both these are real scenarios.

CDR-API-Stream commented 2 years ago

The DSB proposal was discussed during the MI call on 31st August 2022.

Below are the key points:

If no further feedback is provided the DSB will recommend the proposal to the Chair for approval.

CDR-API-Stream commented 1 year ago

This change has been staged: https://github.com/ConsumerDataStandardsAustralia/standards-staging/pull/232/commits/0ca761e7219b6d660f47c72adecea358169442d6