Closed CDR-API-Stream closed 2 years ago
Please find below the response on proposed structure.
Thanks, Kingson EnergyAustralia
If a site has two separate controlled load meter, there can be two different controlled load rates. As this controlled load section is not an array, we can specify only one controlled load rate. Can you please assess the proposed structure to allow us specifying rate for more than one controlled load (i.e Controlled Load 1 & Controlled Load 2).
Thanks, Kingson EnergyAustralia
@Kingson-EA thank you for your feedback.
- Add Start Date (MM-DD) and End Date (MM-DD) (similar to EnergyPlanTariffPeriod) to support seasonal rates for Controlled Load.
We can include the start and end date as optional attributes.
- Proposed Days for controlled load section is aligned with EnergyPlanTariffPeriod > timeOfUseRates section. Can we please also align Demand Section Days similar to this proposed enumerated value so that days representation are unique across. This would also allow specifying demand is applicable for Business days (currently allows weekdays and weekend options only).
As mentioned in the previous MI call, we will raise a separate CR to consult for consistent representation of ENUM values for 'days' across sectors and within energy endpoints
If a site has two separate controlled load meter, there can be two different controlled load rates. As this controlled load section is not an array, we can specify only one controlled load rate. Can you please assess the proposed structure to allow us specifying rate for more than one controlled load (i.e Controlled Load 1 & Controlled Load 2).
This is good feedback. Is it also worth considering including an optional meterId attribute to specify which meter the controlled load rates apply to? Alternatively, the meter information can be captured in a description attribute.
Below is the updated controlloedLoad schema incorporating feedback received throughout maintenance iteration 10. The DSB recommends the change to adopted.
{
"controlledLoad": [ //controlledLoad is an array
{
"displayName": "string",
"rateBlockUType": [ "singleRate | timeOfUseRates" ], //Attribute to specify rate type
"startDate": "DateString", //optional
"endDate": "DateString", //optional
"singleRate": {
"displayName": "string",
"description": "string",
"dailySupplyCharge": "AmountString",
"period": "string", //Formatted according to ISO 8601 Durations (excludes recurrence syntax)
"rates": [
{
"unitPrice": "AmountString",
"measureUnit": [ "KWH | KVA | KVAR| KVARH | KW | DAYS | METER | MONTH" ],
"volume": "number"
}
]
},
"timeOfUseRates": [ //Object to capture Time of Use based Controlled Load rates
{
"displayName": "string",
"description": "string",
"dailySupplyCharge": "AmountString",
"rates": [
{
"unitPrice": "AmountString",
"measureUnit": [ "KWH | KVA | KVAR| KVARH | KW | DAYS | METER | MONTH" ],
"volume": "number"
}
],
"timeOfUse": [
{
"days": ["SUNDAY, MONDAY, TUESDAY, WEDNESDAY, THURSDAY, FRIDAY, SATURDAY, BUSINESS_DAYS"],
"startTime": "string", //Start of the period in HHMM format using 24 hour clock format
"endTime": "string" //End of the period in HHMM format using 24 hour clock format
}
],
// I don't believe that SHOULDER is available in the market, but I've included it for consistency with EnergyPlanTariffPeriod
"type": [ "PEAK | OFF_PEAK | SHOULDER" ]
}
]
}
]
}
This proposal seeks to force the adoption of a non-standard time format beyond the "existing data sources" and inherits a lack of precision in the process.
A number of justifications for this were given from the call:
1) Another payload "at the same level" is using HHMM
tariffPeriod
. The same could be said about solarFeedInTariff
which actually uses TimeString
for a very similar requirement named literally the same way as the proposal.
2) Expansion of an existing payload. The specific timeOfUse
attribute in EnergyPlanTariffPeriod
is an anonymous schema (ie. it isn't a named schema) which, from an implementers perspective, means the "existing payload" statement is technically incorrect.
3) Implementation cost. This didn't seem to involve consulting actual implementers and, in the case of AER related API deliverables seem to relate specifically to AER implementation cost to implement a fairly generic transform. In fact, the desire to keep HHMM
seems to have led to an acceptance of a non-international format to avoid, literally, a one liner being used by the delivery personnel of Get Generic Plan. Since the DSB are such fans of Ruby that's time = Time.new; time.strftime("%k%M")
by the way. Conversely using the TimeString
format is just as easily parsed if desired to the magical format, while having the precision of its original data source dropped by choice.
4) The retailers already supply to AER in this format. This isn't technically true for this proposal because the proposed data isn't in the AER data now. Nonetheless, if we look at this more broadly, at least 2 energy retailers (and indeed any retailer using a pretty popular energy billing platform, ie. 100s) it is happening as a transformed extract from a data source which is already handling it in a well defined time format (specifically GraphQL timetz
). Implementers are highly unlikely to reparse AER extracts for the EnergyAccountDetail
instead retrieving it from the source is both architecturally and implementation cost wise preferred.
Further AER doesn't supply this specific addition (ie. it isn't existing data), the parsing of this value requires not only industry specific knowledge (eg. is this UTC? Is it location based? How does one actually use it) but also the specification of the data format and specification supplied to AER by retailers doesn't seem to be easily accessible (where is it actually published? Does someone know?).
As per the call, my opinion (and the general opinion of Biza.io and possibly any sane engineer who has ever had to do time math) would be to align, wherever possible, with CDR Data Types. These Data Types were specifically defined, hard fought, in many cases to the chagrin of purists who said "Use Native Types" (Me included). Not defaulting to these in essence diminishes the value produced by being divergent from natives in the first place and instead creates an implied EnergyTimeString
for JSON serialisers that will come with whatever the cascading effects involving added then removed timezones back and forth as they make their way through the web of transforms.
The only reasonable justification to do otherwise seems to be a like for like transform from an existing data source - this has tacit acceptance although no defined Standards principle (literally nothing in the principles explicitly supports this pathway). More broadly speaking this justification should be assessed as, with multiple industries in multiple formats, the implementation of defining endlessly variant ways of time, dates, decimals, currencies etc. with variable levels of precision introduces a significant technical debt problem for implementers of CDR itself and will inevitably result in transformation issues between participants.
Finally, on Principles, here's a list that seems to not be aligned:
Thank you for the feedback @perlboy.
There a couple of points worth reiterating/clarifying:
Uplifting time of use to be better while still being consistent across the payloads is an important discussion. As a result, and as mentioned during the MI call, the DSB will raise a CR to consult separately on how time is represented within EnergyPlanSchema. This comment will be updated with the link to the CR once raised. New issue #505 has been raised.
For this CR, however, we believe that the current proposed solution is more appropriate and will be recommending this to Chair for incorporation into 1.17.0.
Thank you for the feedback @perlboy. There a couple of points worth reiterating/clarifying:
- The proposed timeofuse structure for controlled load is derived directly from tariffPeriod as it provides more flexibility and leverages a pattern already in place within this payload.
- AER does hold information about time of use based controlled loads with rates and is expected to provide them.
Ok then I misunderstood what was stated on the call. This would suggest the original payloads did not provide coverage for the AER data to begin with.
Since it is apparent the Generic Plans APIs may be deficient for presentation of existing AER data it's worthwhile evaluating the examples more deeply.
Some examples below:
Uplifting time of use to be better while still being consistent across the payloads is an important discussion. As a result, and as mentioned during the MI call, the DSB will raise a CR to consult separately on how time is represented within EnergyPlanSchema. This comment will be updated with the link to the CR once raised.
Great.
For this CR, however, we believe that the current proposed solution is more appropriate and will be recommending this to Chair for incorporation into 1.17.0.
Sure although the proposed solution doesn't seem to cover all the use cases....
Given the feedback from @perlboy, we suggest the following two options to progress this CR:
Option 1 - Resolve this CR in current MI10 incorporating the solution recommended as per this comment in v1.17.0 of the standards. This could give a baseline to anyone already implementing this specific component. The baseline can be further refined (e.g. include more ENUM values) with separate CRs in future MIs to cater for any remaining use cases.
Option 2 - Carry over and resolve this CR in the next MI (which starts next week on 27th April) to discuss and address the above feedback.
Any feedback on the above two options would be appreciated. We can table this as part of the grooming session at the start of MI11.
So, I went digging into EME and there seems to be quite a few more things worth investigating. These are noteworthy particularly in the context of "existing data sources" but also in the handling of time, dates, days and currency for Energy endpoints. I've provided links to the individual JSONs used for this info. These all matter because beyond AER data Retailers are now expected to present contract data in response to Energy Account Details. Maybe this is the feeder for other items but I'll put it here for now because it does relate to the original change request on time of use tariffs and matching those to AER data.
startTime
and endTime
values from AER are in the format [H]HMM not HHMM . If this is the case then the Standards seem to be redefining the existing data to be zero padded. This is important because neither format is an RFC standard so parsers will be written to process it, it is a string so it is a transform during serialisation and timezone data appears to be inconsistent (see below)AmountString
which is 0.3809 but validating this will be challenging because AmountString
is unbounded (ie. 38.09 is a valid value).BUSINESS DAYS
and other times it is Business Days
. OpenAPI schema is case sensitive so I presume these will be uppercased and transformed.BUSINESS_DAYS
value in the day definition enum is actually unnecessary and in fact ambiguous. The enum therefore could be normalised to actual days of the week at which point recycling banking lastWeekDay
enum would be more consistent (ie. three letter days) with the Standards
days
in EME data seemed to be Business Days|Saturday|Sunday
which seems to actually mean, and indeed is presented on EME as, 7 days per week regardless of public holidays.LOCAL
in the timeDefinition
. This is relevant because it goes to the use of HHMM
rather than a timezone qualified CDR TimeString
and the fact that, at a minimum, there isn't any time definition field. LOCAL
is a commonly used timeDefinition
which I guess means "Local Time" but there seems to be situations where this conflicts non-AE[D]ST locations.postcode
especially since EnergyPlan
already includes/excludes postcodesBy way of examples, here is some info from the sources provided below:
tariffPeriod.touBlock
) :
endDate
is Mandatory in the Standards:
2021-01-01
and finished 2021-12-31
:
2020-07-01
and finished 2020-06-30
(yes, they finish before they start):
2021-07-01
and finishing 2022-06-25
:
timeDefinition
as AEST
:
timeDefinition
as LOCAL
:
LOCAL
but the description the consumer sees is AEST
. This is important because it is for 2000 postcode which has daylight savingsAEST
which is interesting because it is offered in an ACST
timezone (ie. Adelaide) and the shoulder periods seem to be aligned with local timehttps://api.energymadeeasy.gov.au/plans/dpids/RED207686MRE4 https://api.energymadeeasy.gov.au/plans/dpids/RED66349SBE2 https://api.energymadeeasy.gov.au/plans/dpids/SIM244354MRD1 https://api.energymadeeasy.gov.au/plans/dpids/SIM313052MRD1 https://api.energymadeeasy.gov.au/plans/dpids/ORI208128MRE6 https://api.energymadeeasy.gov.au/plans/dpids/AGL15222MRE14 https://api.energymadeeasy.gov.au/plans/dpids/ENE19179MRE27 https://api.energymadeeasy.gov.au/plans/dpids/DEN254925SRE1
The EME team in collaboration with the DSB provided the initial design for the ToU Controlled Load proposal, however, @perlboy has quite rightly highlighted the need for further consultation on this design to ensure that retailer products can be adequately represented by the Standards.
Acknowledging @perlboy's investigative efforts into the EME API, it's worth noting that this endpoint is specifically designed to service the EME UI, only exposes a limited amount of data for that purpose, and that this data should in no way be interpreted as the AER's CDR PRD. The AER is building CDR PRD endpoints that will expose EME and Victorian Energy Compare (VEC) product data transformed to conform with the Standards.
Some of @perlboy's insights from his investigation do warrant further discussion, but not here. I will tackle them in separate proposals.
Acknowledging @perlboy's investigative efforts into the EME API, it's worth noting that this endpoint is specifically designed to service the EME UI, only exposes a limited amount of data for that purpose, and that this data should in no way be interpreted as the AER's CDR PRD.
Thanks Joe and totally agree. To clarify our primary interest in this analysis actually relates to the Account Detail components that reuse the Product Data payloads in those payloads primarily because there is a data transform component related to things like "Business Days" (which week by week is challenging).
As per a previous reply:
Further AER [....] but also the specification of the data format and specification supplied to AER by retailers doesn't seem to be easily accessible (where is it actually published? Does someone know?).
I would have preferred to use the input format supplied by retailers to AER when analysing this but it didn't seem to be available - is that something you could provide a link to?
Unfortunately Stuart, the EME data specification document is only available to authorised energy retailers.
Unfortunately Stuart, the EME data specification document is only available to authorised energy retailers.
That's interesting because I would have thought as a Government agency unless there was some national security or confidentiality requirement such a document would be in the public interest especially since the justification for adoption of certain data types within the Standards is on the basis of existing data formats.
Below points summarise the above feedback that was discussed in today's MI11 call:
The proposed time of use type for controlled load does not currently cater for Solar Sponge:
Controlled loads that have non-specific time of use description such as "Generally available for a 6 hours period between 10pm and 7am."
Stepped solar feed in tariffs
Standardised representation time in EnergyPlanDetail schema and alignment of ENUM value for days across energy standards
The DSB would like to propose the following changes to address point 1 and 2 in the above comment:
The combination of change 2 and 3 should allow specifying controlled load plans where details of time of use are not known and contain description such as:
Below is the revised controlloedLoad schema:
{
"controlledLoad": [ //controlledLoad is an array
{
"displayName": "string",
"rateBlockUType": ["singleRate","timeOfUseRates"], //Attribute to specify rate type
"startDate": "DateString", //optional
"endDate": "DateString", //optional
"singleRate": {
"displayName": "string",
"description": "string",
"dailySupplyCharge": "AmountString",
"period": "string", //Formatted according to ISO 8601 Durations (excludes recurrence syntax)
"rates": [
{
"unitPrice": "AmountString",
"measureUnit": ["KWH", "KVA", "KVAR", "KVARH", "KW", "DAYS", "METER", "MONTH" ],
"volume": "number"
}
]
},
"timeOfUseRates": [ //Object to capture Time of Use based Controlled Load rates
{
"displayName": "string",
"description": "string",
"dailySupplyCharge": "AmountString",
"rates": [
{
"unitPrice": "AmountString",
"measureUnit": ["KWH", "KVA", "KVAR", "KVARH", "KW", "DAYS", "METER", "MONTH" ],
"volume": "number"
}
],
"timeOfUse": [
{
"days": ["SUNDAY", "MONDAY", "TUESDAY", "WEDNESDAY", "THURSDAY", "FRIDAY", "SATURDAY", "BUSINESS_DAYS"], //Make attribute optional. NOTE: The ENUM values will be aligned to agreed decision in CR 502
"startTime": "string", //Make attribute optional. NOTE: The type of this attribute will be aligned to agreed decision in CR 505
"endTime": "string", //Make attribute optional. NOTE: The type of this attribute will be aligned to agreed decision in CR 505
"additionalInfo" : "string", //Optional free text attribute to describe controlled load availability if specific day/time is not known
"additionalInfoUri" : "string" //Optional attribute to provide link to web page with additional information
}
],
"type": [ "PEAK | OFF_PEAK | SHOULDER | SOLAR_SPONGE" ] // added SOLAR_SPONGE
}
]
}
]
}
Feedback is welcome on the above recommended changes.
Following feedback received during Energy MI call on 14th June, the DSB discussed the structure internally and notes that the period
field in singleRate
structure does not apply anymore. The timeOfUseRates
structure in the proposed controlloedLoad
schema can be used to describe different rate periods. As a result, the DSB recommends removing it.
Below is the updated structure:
{
"controlledLoad": [ //controlledLoad is an array
{
"displayName": "string",
"rateBlockUType": ["singleRate","timeOfUseRates"], //Attribute to specify rate type
"startDate": "DateString", //optional
"endDate": "DateString", //optional
"singleRate": {
"displayName": "string",
"description": "string",
"dailySupplyCharge": "AmountString",
"rates": [
{
"unitPrice": "AmountString",
"measureUnit": ["KWH", "KVA", "KVAR", "KVARH", "KW", "DAYS", "METER", "MONTH" ],
"volume": "number"
}
]
},
"timeOfUseRates": [ //Object to capture Time of Use based Controlled Load rates
{
"displayName": "string",
"description": "string",
"dailySupplyCharge": "AmountString",
"rates": [
{
"unitPrice": "AmountString",
"measureUnit": ["KWH", "KVA", "KVAR", "KVARH", "KW", "DAYS", "METER", "MONTH" ],
"volume": "number"
}
],
"timeOfUse": [
{
"days": ["SUNDAY", "MONDAY", "TUESDAY", "WEDNESDAY", "THURSDAY", "FRIDAY", "SATURDAY", "BUSINESS_DAYS"], //Make attribute optional. NOTE: The ENUM values will be aligned to agreed decision in CR 502
"startTime": "string", //Make attribute optional. NOTE: The type of this attribute will be aligned to agreed decision in CR 505
"endTime": "string", //Make attribute optional. NOTE: The type of this attribute will be aligned to agreed decision in CR 505
"additionalInfo" : "string", //Optional free text attribute to describe controlled load availability if specific day/time is not known
"additionalInfoUri" : "string" //Optional attribute to provide link to web page with additional information
}
],
"type": [ "PEAK | OFF_PEAK | SHOULDER | SOLAR_SPONGE" ] // added SOLAR_SPONGE
}
]
}
]
}
If no further feedback is received, the DSB will recommend the above change to the Chair for approval.
Note: The ENUM values of days
field will be aligned to agreed decision in https://github.com/ConsumerDataStandardsAustralia/standards-maintenance/issues/502
and attribute type of startTime
and endTime
will be aligned to agreed decision in https://github.com/ConsumerDataStandardsAustralia/standards-maintenance/issues/505
This issue has been staged and can be reviewed here - https://github.com/ConsumerDataStandardsAustralia/standards-staging/compare/release/1.18.0...maintenance/472
This change was incorporated into release v1.18.0. Refer to Decision 249 for further details.
Description
Controlled Load rates can be fixed or based on time of use (e.g. different controlled load rates for peak vs off-peak). This change request is being created to modify the structure to cater for Controlled load Time of Use rates as current controlledLoad object does not cater it.
Area Affected
Following Energy APIs are affected
Change Proposed
The following change is proposed to the controlledLoad object to cater for TOU based controlled load rates: