Update Register API to return separate PRD and status/outage endpoint

[jxeeno]

Description

It is not currently possible to programatically discover a list of Energy CDR Product Reference Data (PRD) endpoints programatically through CDR-supported endpoints such as the Register API. This change request proposes a change to the Register API to allow for the programatic discovery of PRD endpoints for Energy CDR.

As outlined in discussions here and here, this is due the following limitations:

• AER / the "Victorian agency" are the data holder for Energy CDR PRD (Product Reference Data), whereas the provider is the data holder for consumer data requests.
• The GetDataHolderBrandsSummary endpoint was implemented in #444 as a means for the Register API to allow unauthenticated users to discover data holder endpoints. However, the publicBaseUri field returns different values depending on the provider's status for consumer data requests which makes this solution incomplete.
  • Where a provider is active, the publicBaseUri field is populated with the provider's endpoint for responding to status and outages requests.
  • Where a provider is inactive, the publicBaseUri field will return the PRD endpoint provided by AER/the "Victorian agency" on the cdr.energymadeeasy.gov.au host.
• Under Energy CDR rules, the providers are not data holders for the PRD and are not required to implement product reference data endpoints. Although, they may do so if they wish. As confirmed in Noting Paper 248, this remains the status quo.

There are workarounds which have been noted:

• AER publishes a list of endpoints in a PDF file. However, this is not a CDR API, there are no clear means to access this data programatically, it is not in a standard format, and there is no clear update frequency.
• Community maintained datasources.json file is a temporary workaround. There is no obligation for the community to keep this file up-to-date and is not an official source of CDR data.

Area Affected

This CR affects the GetDataHolderBrandsSummary endpoint of the Register API.

Change Proposed

This CR proposes a change to the Register API's GetDataHolderBrandsSummary endpoint to separately return PRD endpoints provided by AER / the "Victorian agency" and outages/status endpoints provided by the providers.

Where available, the GetDataHolderBrandsSummary endpoint will return separate *BaseUri fields for the purposes of serving Product Reference Data (PRD) and Status/Outage endpoints.

The proposed endpoint will return the following data:

Name	Type	Required	Description
brandName	String	true	Name of the data holder brand hosting the public endpoints
brandId	String	true	ID of the data holder brand hosting the public endpoints.  To be used to uniquely identify the record and not to be reused
publicBaseUri	UriString	true	Public endpoints base URL (Specifically, for the Outage and Status endpoints)
productReferenceDataBaseUri	UriString	true	Product Reference Data endpoints base URL
logoUri	UriString	true	Data Holder Brand logo URL
industries	[string]	true	The industries the Data Holder Brand belongs to. Please note that the CDR Register entity model is constrained to one industry per brand which is planned to be relaxed in the future.
lastUpdated	DateTime	true	Time when the record was last updated
abn	String	false	Australian Business Number for the organisation
acn	String	false	Australian Company Number for the organisation
arbn	String	false	Australian Registered Body Number. ARBNs are issued to registrable Australian bodies and foreign companies

As an example:

{
    "data": [{
        "dataHolderBrandId": "244d8a80-3828-ed11-a832-000d3a8830d6",
        "brandName": "Origin Energy",
        "industries": [
            "energy"
        ],
        "logoUri": "https://res.cloudinary.com/originenergy/image/upload/v1667947270/CDR/origin-energy-logo.png",
        "publicBaseUri": "https://public.mydata.cdr.originenergy.com.au",
        "productReferenceDataBaseUri": "https://cdr.energymadeeasy.gov.au/origin",
        "abn": "30000051696",
        "acn": "000051696",
        "lastUpdated": "2022-12-01T10:36:22Z"
    }]
}

Discussions so far

1. In the discussion of Noting Paper 248, it was mentioned that the proposed changes in this CR were discussed. However, "[it was] decided against pursuing this route due to the complexity of change required for the Register and existing participants for a situation that is probably unique to the energy sector".
2. In the discussion of Noting Paper 248, @CDR-API-Stream suggested a CR in standards maintenance is the most appropriate place to raise this. In absence of other CRs relating to this issue, this CR was raised to move this conversation forward.
3. Origin, AGL and @perlboy (see here) appear to be broadly supportive separation of endpoints for PRD and status/outages.

Delivery of V2 GetDataHolderBrandsSummary API

Separate to this CR, there is a separate CR to implement V2 of the GetDataHolderBrandSummary API (#518).

The implementation of V2 as proposed does not resolve the outstanding issues of PRD endpoint discovery.

Other options

Discussion of other options are welcome.

[jxeeno]

On the point raised by the Register relating to the uniqueness to the energy sector:

decided against pursuing this route due to the complexity of change required for the Register and existing participants for a situation that is probably unique to the energy sector

I don't believe this issue will necessarily be unique to the energy sector. There are similar arrangements which exist in other industries such as health insurance and superannuation where some product data is held by the Private Health Ombudsman or APRA.

[CDRPhoenix]

Might also want to add a field like the following to the [GetDataHolderBrandsSummary](https://consumerdatastandardsaustralia.github.io/standards/index.html#get-data-holder-brands-summary) endpoint brand | string | optional | Used in the query string to filter results on the brand field. If absent defaults to include all -- | -- | -- | -- The current banking and energy PRD endpoints contains this field and when combined with a separate PRD endpoint, it opens up the possibility of service providers hosting and managing PRD data for multiple Data Holders on a single URL

[jxeeno]

I note that product-comparator-demo has been updated to use a new override.json file instead of providing the PRD as part of the URL in the datasources.json field. Since the file isn't an officially supported method of PRD endpoint discovery, DSB are entitled to make these changes without consultation. However, this has broken any implementations that rely on datasources.json.

Previous advice to use datasources.json for discovery are now also invalid.

I think this issue requires fairly urgent attention. AER as the dataholder only provides endpoints in a PDF file (i.e. not programatically discoverable). ACCC Register API provides incomplete list of PRD endpoints. Community/DSB maintained datasources file is subject to arbitrary changes as required by the team.

EnergyAustralia are now also active data holders, meaning all publicly available machine readable datasources are out of date again. This makes usage and automated discovery of energy PRD impossible.

[jxeeno]

Closing as the proposed amendments to the Rules by Treasury outlines the preferred path forward is to require energy retailer data holders to "forward" the requests to data holder of the PRD (i.e. AER/the Victorian agency).

[biza-io]

We request this be reopened as we do not believe the proposed Rules amendments are in the best interests of the ecosystem. The proposal to split Public vs. Product base uri is likely the most cost effective and architecturally sound solution to achieving the objective of providing discovery capability for Energy product data.

[perlboy]

Given #637, requesting this be added to the backlog for MI19.