search

ConsumerDataStandardsAustralia / standards-maintenance

This repository houses the interactions, consultations and work management to support the maintenance of baselined components of the Consumer Data Right API Standards and Information Security profile.
41 stars 9 forks source link

Supporting HTTP Status 429 passthrough from Secondary Data Holder #651

Open perlboy opened 4 months ago

perlboy commented 4 months ago

Description

The Secondary Data Holder (AEMO) datasets sometimes are unable to be provided within a suitable period of time. This results in timeouts at the data holder side and often results in the ADR immediately re-requesting the data. This leads onto a new timeout and a layering of duplicate requests. As energy adoption has picked up there are now scenarios where hundreds of NMIs are brought in scope for a single ADR request.

Intention and Value of Change

While a longer term solution is desired (notably asynchronous retrieval) a short term solution mooted by industry participants is to specifically introduce the ability of AEMO to return a 429 Too Many Requests with a Retry-After communicating a forecast availability of the data being ready. The use of Retry-After is a little bit odd but provides a mechanism for AEMO to provide such a forecast, something they know based on the number of NMIs and usage period requested.

Area Affected

Supported error messages for Secondary Data Holder.

Change Proposed

Introduce 429 as a formally supported response from SDH/AEMO, required it to be passed through by Data Holders and required it to be honoured by Data Recipients.

johnAEMO commented 3 months ago

Based on the conversations today - this one could be a candidate for an asynch model. If we have to change the standards after this pilot, we might as well go straight to asynch!

CDR-API-Stream commented 3 months ago

The DSB is progressing discussions with AEMO/Biza to determine an approach to conduct a trial to test the use of 429 Retry-After pattern in sharing large volume of data. The intent is to help determine the effectiveness of the pattern and determine the required standards changes for recommendation once the trail is concluded.

We will update this CR with further details on the trial when available.