ConsumerDataStandardsAustralia / standards-maintenance

This repository houses the interactions, consultations and work management to support the maintenance of baselined components of the Consumer Data Right API Standards and Information Security profile.
41 stars 9 forks source link

EnergyDerRecord - mandatory values not available in AEMO's DER register #524

Closed OpalRussAEMO closed 1 year ago

OpalRussAEMO commented 2 years ago

Description

For the EnergyDerRecord schema, NominalRatedCapacity is mandatory and nominalStorageCapacity is conditionally required where the equipment type is storage.

A data issue has been identified in AEMO's DER register where 100k+ records do not have nominal rated capacity populated and there are some storage equipment records that do not have nominalStorageCapacity populated. These values are optional when establishing DER records in AEMO's DER register.

Edit 14/9/22

The same issue as above also applies to inverterDeviceCapacity. Where inverterDeviceCapacity is not populated, the NominalRatedCapacity for the device is used to determine this value. In the case where both inverterDeviceCapacity and NominalRatedCapacity are null, a value cannot be provided for inverterDeviceCapacity.

Area Affected

Energy APIs Get DER for... EnergyDerRecord schema

https://consumerdatastandardsaustralia.github.io/standards/#schemacdr-energy-secondary-data-holder-apienergyderrecord

Change Proposed

Interim solution -

Standards change - change the two fields to optional to allow for data to be provided for these service points.

perlboy commented 2 years ago

As I understand it nominalRatedCapacity can't legitimately be 0 (ie. no DER device would have a 0 kVA rating?) so a 0 value would be "ok". For nominalStorageCapacity, this can legitimately be 0 value (ie. a STORAGE device without a battery)? If so, defaulting to a 0 value may be misleading, perhaps -1 would be more appropriate.

In both cases though it seems your record is actually invalid (?) and previous guidance (on addresses) was to omit the record entirely.

Finally, since AEMO is a Secondary Data Holder, is obligated to provide this information and has the dataset elsewhere, I presume it will lodge a rectification schedule with the ACCC and rectify the faulty records on an agreed timeframe?

OpalRussAEMO commented 2 years ago

Updated description - not a migration issue. These values are optional when establishing DER records in the DER register.

johnAEMO commented 1 year ago

The team has just finalised analysis of the values in these 2 fields:

In both cases we would recommend that, where we find a null value we will pass on a "0" (zero) in the payoad as a workaround. In both cases we would expect this to impact a very small number of actual DER records and rather than holding back the entire record for a data quality issue with one field, we see value in providing the balance of the record even if there are a very small number where a field has a data quality issue.
A value of "0" is not a legitimate value for either nominalRatedCapacity or nominalStorageCapacity (a storage device must have a capacity >0) Should this cause an issue to the ADR, it can be treated as a data quality issue and handled accordingly (forwarded to the LNSP for rectification). We are expecting the resultant impact of this to be very small.

Happy to discuss

perlboy commented 1 year ago

A value of "0" is not a legitimate value for either nominalRatedCapacity or nominalStorageCapacity (a storage device must have a capacity >0)

The WEM DER Technical Specification specifies that 0 is a valid value for both these values. Is the specification incorrect or am I reading the wrong document?

image

OpalRussAEMO commented 1 year ago

A value of "0" is not a legitimate value for either nominalRatedCapacity or nominalStorageCapacity (a storage device must have a capacity >0)

The WEM DER Technical Specification specifies that 0 is a valid value for both these values. Is the specification incorrect or am I reading the wrong document?

image

We have looked into this further, a value of 0 was allowed for under the WEM DER to cater for limited historic device information for around 8% of records. So the use of a 0 value for CDR would be consistent with the WEM DER approach.

OpalRussAEMO commented 1 year ago

Update: AEMO are planning a program of work to remediate DER data issues, mentioned by John above, with the assistance of distributors. This work is planned to be completed by May 2023.

In alignment with the WEM DER, AEMO will populate the missing values with 0 until the data is remediated. Individual requests to correct data in the DER (where a 0 value is received) can be made to DNSPs.

OpalRussAEMO commented 1 year ago

As mentioned in maintenance call, handling of null values for inverterDeviceCapacity is also required (updated issue description). Roughly 0.06% of DER records have no value for this field, where it is not available we attempt to derive from the nominal rated capacity. Where both values are not available, we will need to also provide a 0 value for inverterDeviceCapacity.

CDR-API-Stream commented 1 year ago

As discussed and agreed during the September 14th MI12 call, the description of the following fields will be updated to note a default of 0 will provided if value is not known:

The above proposal will be recommended to the Chair for approval.

CDR-API-Stream commented 1 year ago

The change has been staged: https://github.com/ConsumerDataStandardsAustralia/standards-staging/pull/231/commits/f0313eaf114676e11aae619e86da1cbb62e6aa42