Iteration 12 Holistic Feedback

elizabetharnold commented 2 years ago

This CR has been created to simplify the raising of minor changes, such as text corrections or description clarifications, that are not really material to the standards but do have real impact of readability and clarity.

Please raise any such suggestions that you would like included in Maintenance Iteration 12 and the DSB will review them. If we believe the suggestion are material we will raise a dedicated CR for them.

perlboy commented 2 years ago

A reminder that was lost in MI11, can the "effective date" used for oldest-date and newest-date for Get Invoices be clarified as issueDate (similar to the clarifications for Get Banking Transactions). This was stated as the intent by @HemangCDR in the Implementation call 21 July.

CDR-API-Stream commented 2 years ago

Sorry we missed that one @perlboy, we'll pick it up in this iteration

CDR-API-Stream commented 2 years ago

It was noted via a Zendesk ticket that some of the statements in the FDO section of the standards user may and must without capitalisation. This should be rectified as a simple formatting update.

CDR-API-Stream commented 2 years ago

The Get Data Holder Brands link is incorrect and the anchor needs to be updated (i.e. #get-data-holder-brands).

GetDataHolderBrands Link

Similarly, the Get Data Recipients link needs to point to the correct Register API anchor (i.e. #get-data-recipients).

nils-work commented 2 years ago

The items in the Register Dependency Schedule should be shown in ascending order as per the FDO table.

nils-work commented 2 years ago

There is a documentation typo in ResponseOpenIDProviderConfigMetadata, there are escaped quotes around private_key_jwt -

nils-work commented 2 years ago

Review the Future Dated Obligations table - consider removing or greying-out items past their due date.

nils-work commented 2 years ago

The Profile Scope Data Language link in the Future Dated Obligations table is currently broken and should point to Profile Scope and Standard Claims: Common.

nils-work commented 2 years ago

Some parameters in the following sections specify field types that are not aligned to the Common Field Types definitions -

The details have been acknowledged in the Known Issues list since v1.13.0 (Register APIs use different field type definitions).

nils-work commented 2 years ago

Response Headers in the Energy APIs section (under Get Generic Plans for example) show none for most header Descriptions.

These could be aligned to the convention used in the Banking APIs Response Headers (Get Accounts for example).

nils-work commented 2 years ago

There is a typo 'ofways' in the Standards README -

CDR-API-Stream commented 2 years ago

AEMO have stated that the description of validFromDate in EnergyServicePoint structure needs to be updated to the following to be more clear:

The start date from which this service point first became valid. For service points that became valid prior to 2 years, this date will be set to the request date minus 2 years.

perlboy commented 2 years ago

AEMO have stated that the description of validFromDate in EnergyServicePoint structure needs to be updated to the following to be more clear:

The start date from which this service point first became valid. For service points that became valid prior to 2 years, this date will be set to the request date minus 2 years.

Doesn't this fundamentally alter the meaning of this field? Does this mean that service points that are older than 2 years will now have a validFromDate that is constantly incrementing forward? I get the reasoning for historical data load but for perpetuity, this seems flawed.

perlboy commented 2 years ago

1.19.0 was just released but introduced open-status into Energy Account Detail as well? I don't believe this was part of https://github.com/ConsumerDataStandardsAustralia/standards/issues/260 ?

OpalRussAEMO commented 2 years ago

AEMO have stated that the description of validFromDate in EnergyServicePoint structure needs to be updated to the following to be more clear:

The start date from which this service point first became valid. For service points that became valid prior to 2 years, this date will be set to the request date minus 2 years.

Doesn't this fundamentally alter the meaning of this field? Does this mean that service points that are older than 2 years will now have a validFromDate that is constantly incrementing forward? I get the reasoning for historical data load but for perpetuity, this seems flawed.

I agree that's an issue. For background, we raised a question to clarify how this field should be populated and what a valid service point is. We have performance concerns if we need to retrieve records that are up to 20 years old to determine the original date a service point was created (which may or may not be a defaulted date), as we need 2 years of standing data to support usage APIs it was suggested that we could default the date if the service point was created more than 2 years prior.

Looking at previous consultation on this, it seems to have been suggested that the validFromDate would be the latest start date for service point details. So essentially validFromDate would be the effective date of the current set of service point details and the lastUpdateDateTime would be the date on which they were modified last. We would be happy to align with this option if preferable.

" “vaildFromDate”: the latest start date of the constituent data sets. “lastUpdateDateTime”: the latest time any of the constituent data sets has been changed – noting that this may be from a different data-set providing the “validFromDate”. "

perlboy commented 2 years ago

I agree that's an issue. For background, we raised a question to clarify how this field should be populated and what a valid service point is. We have performance concerns if we need to retrieve records that are up to 20 years old to determine the original date a service point was created (which may or may not be a defaulted date), as we need 2 years of standing data to support usage APIs it was suggested that we could default the date if the service point was created more than 2 years prior.

Hmmm, I would have thought this data would be in a materialised view and not retrieved live, if it is live it will fail an NFR very quickly in an O(n2) problem as demand increases. Ultimately this is an implementation concern, I don't think it's appropriate to be altering the Standard because this isn't being persisted by the source system.

As per above, I get if this was once off but the rolling forward thing means that most service points, cause most are older and unchanged >2 years, will essentially be considered to have a transient value, this could actually exacerbate load from Data Recipients.

At a minimum it basically guarantees that the range queries from a retailer won't align with what the Data Recipient is going to get back in this field.

Looking at previous consultation on this, it seems to have been suggested that the validFromDate would be the latest start date for service point details. So essentially validFromDate would be the effective date of the current set of service point details and the lastUpdateDateTime would be the date on which they were modified last. We would be happy to align with this option if preferable. _" “vaildFromDate”: the latest start date of the constituent data sets. “lastUpdateDateTime”: the latest time any of the constituent data sets has been changed – noting that this may be from a different data-set providing the “validFromDate”.

So the above was feedback from AEMO at the time but the DSB response was:

We defer to the AEMO's definition of this field under MSATS as this field has a direct corollary in the MSATS data.

What's interesting with that statement is that I can't actually find validFromDate in MSATS definitions, I admit I didn't look very long but does it actually exist?

Ultimately, from an implementation perspective I guess it doesn't matter ("a date is a date") but from a Data Recipient perspective how do they find out how old a meter is? That seems like a useful data point that I presumed was the purpose of this field?

OpalRussAEMO commented 2 years ago

So the above was feedback from AEMO at the time but the DSB response was:

We defer to the AEMO's definition of this field under MSATS as this field has a direct corollary in the MSATS data.

What's interesting with that statement is that I can't actually find validFromDate in MSATS definitions, I admit I didn't look very long but does it actually exist?

Ultimately, from an implementation perspective I guess it doesn't matter ("a date is a date") but from a Data Recipient perspective how do they find out how old a meter is? That seems like a useful data point that I presumed was the purpose of this field?

There is no such field in MSATS. We have effective from/to and updated/created dates on all standing data records which tell you what date range the information is effective for and when it was created / updated. You would not be able to tell how old a meter is by us providing a single date for when the service point was created in MSATS. Meters change over time and won't align with dates for when the service point was created or updated. Also service points are created in advance of supply being connected or metering being installed so the date on which it was created doesn't mean anything to consumers.

CDR-API-Stream commented 2 years ago

1.19.0 was just released but introduced open-status into Energy Account Detail as well? I don't believe this was part of ConsumerDataStandardsAustralia/standards#260 ?

Thanks @perlboy this has been identified and corrected in the 1.19.0 release.

CDR-API-Stream commented 2 years ago

Doesn't this fundamentally alter the meaning of this field? Does this mean that service points that are older than 2 years will now have a validFromDate that is constantly incrementing forward? I get the reasoning for historical data load but for perpetuity, this seems flawed.

This is valid feedback.

Looking at previous consultation on this, it seems to have been suggested that the validFromDate would be the latest start date for service point details. So essentially validFromDate would be the effective date of the current set of service point details and the lastUpdateDateTime would be the date on which they were modified last. We would be happy to align with this option if preferable.

This would be a better approach as it aligns with the original intent of the field. The description can be updated as below:

validFromDate: The latest start date from which the constituent data sets of this service point became valid