search

ConsumerDataStandardsAustralia / standards-maintenance

This repository houses the interactions, consultations and work management to support the maintenance of baselined components of the Consumer Data Right API Standards and Information Security profile.
41 stars 9 forks source link

Confirm Register API 2022 release dates #465

Closed CDR-API-Stream closed 2 years ago

CDR-API-Stream commented 2 years ago

Description

As part of V1.15.0, new versions of the Register APIs were defined to accommodate multiple sectors in the regime. As part of this change, Future Dated Obligations were tentatively set for 30th Aug 2022 with further consultation required with the ACCC to confirm this obligation date

This issue has been raised to facilitate this further consultation

Area Affected

Future Dated Obligations - Register APIs

Change Proposed

Define the future dated obligation for Register API versionss delivered in 2022

perlboy commented 2 years ago

Guidance to energy sector at this point is indicating a June(ish) testing initiation. Not having a functional SSA until end of August provides 4 weeks before the currently mandated compliance date. Suggest the ACCC align its obligations with the intended start time of its testing. Vendors currently serving other markets may be unable (or unwilling) to "hack" an SSA in to support the Regulators desires.

ACCC-CDR commented 2 years ago

The instigator of the Register API changes in v1.15.0 (#424 and #425) was to ensure the Register would support multiple sectors. The ACCC proposes that the Future Dated Obligations for the v1.15.0 Register API changes be updated to 15 November 2022. This would align the obligation date with the commencement date for consumer data sharing within the Energy sector, which is effectively the date the CDR becomes multi-sector.

It is important to distinguish between obligation dates and release dates. While these dates need to be coordinated for non-backwards compatible changes, this is not the case in this instance and it would be preferable to separate these concepts in future GitHub issues. While the ACCC is proposing that the obligation date be moved to 15 November, the Register API changes may be released earlier and the ACCC will keep the community informed of future releases to the Register.

In regards to testing, the ACCC offers tools to participants to support the development and testing of their solutions. The enhancement of these tools to support the energy sector have planned release dates for Q2 of 2022.

For further information on these tools please refer to the following pages:

  1. https://www.cdr.gov.au/for-providers/participant-tooling

  2. https://www.cdr.gov.au/for-providers/participant-tooling/consumer-data-right-sandbox

For these reasons, the ACCC does not see testing as a reason to move the obligation date forward. Additionally, the ACCC proposes moving the obligation date backwards to the 15th November 2022 to maintain alignment with the consumer data sharing obligation date for Energy.

perlboy commented 2 years ago

It is important to distinguish between obligation dates and release dates. While these dates need to be coordinated for non-backwards compatible changes, this is not the case in this instance and it would be preferable to separate these concepts in future GitHub issues. While the ACCC is proposing that the obligation date be moved to 15 November, the Register API changes may be released earlier and the ACCC will keep the community informed of future releases to the Register.

The bolded statement above appears to be factually incorrect in the following areas: 1) GetDataRecipientsStatus, which had it's structure fundamentally changed by attribute name changes, had x-v set to Mandatory when previously it was Optional. If implemented as specified this will immediately cause all implementations not specifying x-v to be rejected. Currently there is no transitionary state specified for the Register to retain historical behaviour. 2) GetDataRecipientsStatus, which had it's structure fundamentally changed by attribute name changes, had x-v set to Mandatory when previously it was Optional. If implemented as specified this will immediately cause all implementations not specifying x-v to be rejected. Currently there is no transitionary state specified for the Register to retain historical behaviour. 3) GetSoftwareStatementAssertion, which had scopes added, had x-v set to Mandatory when previously it was Optional. If implemented as specified this will immediately cause all implementations not specifying x-v to be rejected. Currently there is no transitionary state specified for the Register to retain historical behaviour. There is also precedence available regarding SSA modifications with the addition of profile and openid scopes being set as a FDO in 2021 4) cdr-register:read was added as a required scope (the Standards language is a plural "must be authenticated and authorised with the following scopes") for Recipients to authenticate with.

I can confirm the following with regards to the existing ecosystem:

In regards to testing, the ACCC offers tools to participants to support the development and testing of their solutions. The enhancement of these tools to support the energy sector have planned release dates for Q2 of 2022. For further information on these tools please refer to the following pages:

  1. https://www.cdr.gov.au/for-providers/participant-tooling
  2. https://www.cdr.gov.au/for-providers/participant-tooling/consumer-data-right-sandbox

Firstly, if there are "planned release dates" can the ACCC please publish them. Given the use of taxpayer funds it is unclear why there isn't transparency on this.

Beyond this all these pages say is "anticipated release in the coming months" and it is unclear if this Sandbox will: 1) Support multi mode behaviour of both new and old Register behaviours concurrently - which, as stated, is impossible if x-v is mandated as specified 2) Be tested for interoperability between participants rather than be a closed loop of a single vendors implementation 3) Not be buggy itself. There is some precedence here with the Register having a broken Admin Metrics implementation for nearly 3 months and the ACCC CTS, even now, failing to meet the Register specification in both how it manages the removal of software products and its availability even with a loaded Test Plan. Forgive those of us who perhaps have a relatively low confidence in the Regulators ability to deliver software that is actually compliant.

Somewhat related to the Sandbox is the question of when the CTS will support the new Register behaviours. That seems relatively important to have some comms on too so that Energy organisations are tested with the launch date state.

For these reasons, the ACCC does not see testing as a reason to move the obligation date forward. Additionally, the ACCC proposes moving the obligation date backwards to the 15th November 2022 to maintain alignment with the consumer data sharing obligation date for Energy.

It seems there should probably be two (possibly three) obligation dates: 1) When ACCC will introduce a multi-mode Register that works with both old and new clients (in essence not enforcing x-v header and returning old payloads when they are absent or specified as x-v=1). It seems logical to expect this by August 2022. 2) When Holders must upgrade their installations by, probably before energy to ensure ADRs don't have embarrassing interoperability issues on launch day. November 2022 would align with the prescribed Holder upgrade schedule. 3) (Possibly) A deprecation date for Version 1. February 2023 would align here.

CDR-API-Stream commented 2 years ago

@perlboy Thanks for raising these issues.

Regarding Points 1,2 and 3, this highlights that flipping the x-v header value to Mandatory is mutually exclusive to being able to keep banking and energy versions of the APIs available in parallel. We require a default value version value to be maintained for the banking sector which is no longer possible for a mandatory x-v header..

I see two impacts from these points:

  1. x-v can't be mandatory if we are to maintain no impact on the banking sector
  2. If we keep x-v optional during this transition period, a default value for the version would need to be specified when the x-v header is not supplied. Initial thinking is this should align to the minimum supported value of the API.

The intent is to maintain a transitionary period between both sets of APIs and this needs to be addressed.

Regarding Point 4, this is a defect of the scopes assignment. The intent has been that original versions of the APIs use the authorisation scope cdr-register:bank:read while the new versions proposed to use the more generic scope cdr-register:read. This would allow clients to change the scope used when they transition to the new version of the API. A union of the two scopes was not intended. Issue #498 has been raised to track this issue

CDR-API-Stream commented 2 years ago

Issue #501 has been raised to track the concerns raised with x-v headers moving to mandatory

CDR-API-Stream commented 2 years ago

To simplify tracking, based on the feedback we have received this issue has been broken into 3 separate issues, all covered through maintenance iteration 10.

  1. Define the future dated obligation for Register API versions delivered in 2022 - to be covered through this issue
  2. Address the concern that a union of cdr-register:read and cdr-register:bank:read scopes being required for new Register API versions. - To be tracked under issue #498
  3. Address the concern that mandating x-v headers for the new versions of the Register APIs impacts banking implementations. - To be tracked under issue #501

Proposal: Defining the future dated obligation for Register API versions delivered in 2022

Collaboration on this issue through maintenance iteration 10, has resulted in the following DSB proposal, expanding off @ACCC-CDR's initial proposal.

The future-dated obligation binding dates for the following Register API versions will move to 15th November 2022 to align with the Energy rollout.

API Name Endpoint Method Version
Get Data Holder Brands /{industry}/data-holders/brands GET V2
Get Software Statement Assertion (SSA) /{industry}/data-recipients/ brands/{dataRecipientBrandId}/ software-products/{softwareProductId}/ssa GET V3
Get Software Products Statuses /{industry}/data-recipients/ brands/software-products/status GET V2
Get Data Recipient Statuses /{industry}/data-recipients/status GET V2
Get Data Recipients /{industry}/data-recipients GET V3
Get Data Holder Statuses /{industry}/data-holders/status GET V1

As described in issue #424 and #425, these APIs are expected to be used by participants entering the Energy sector. Other participants will be free to migrate to the latest versions however the previous versions will remain in effect to avoid impact to the Banking sector.

Actual release timings on the CDR Register and any related CTS impacts are in the ACCC's domain and will not be articulated through the standards.

CDR-API-Stream commented 2 years ago

This change was incorporated into release v1.17.0.

Please refer to Decision 237 for further details.

Concern was raised by industry that this change does not articulate when these APIs will be available in testing or released to production. The actual release timings on the CDR Register and any related Conformance Test Suite (CTS) impacts are in the ACCC's domain and will not be articulated through the standards. The standards seek to define what the logical latest date is for the ACCC to release these APIs and not impact the obligations of the participants. The community has requested the ACCC make their release schedule known.