Time format change request for Energy Plan Data

[eme-admin]

Description

Time within energy plan data e.g. Peak period start and end time were defined in a string format HHMM (24HR) (3pm will be defined as "1500"). This representation of time is different to how time was defined in other areas of the CDR standards for energy which was in the timestring format (3pm will be defined as 05:00:00+10:00, where +10 is the time offset to GMT). So Consumer Data Standards body made an update to the CDR standards in a maintenance iteration to align it with other areas of energy data but this introduced an issue

Area Affected

This representation of time introduced an issue when data recipients had to convert them back to the local time during the day light saving period for human readability. For example, 3 pm will be defined as 05:00:00+10:00 during the AEST to GMT conversion but when consumed by data recipient during the AEDT time its converted back to 2pm local time rather than 3pm. So there is no way in the CDS standard to represent the AEDT offset of 05:00:00+11:00 and AEST offset of 05:00:00+10:00 at the same time.

Change Proposed

AER and DEECA propose a CR highlighting this issue and request the time presentation to be reverted back to HHMM (24HR) which doesn't have the offset issue and will always represent the local time. If not CDS can propose a new format that would resolve this issue.

DSB Proposed Solution

The current DSB proposal for this issue is in this comment

[perlboy]

The "area affected" seems to misunderstand how RFC3339 works. Realistically the whole point of transmitting in these formats is so a recipient does not have to be aware of the holders local timezone. They can alter their presentation layer based on the users location or with other information. A simplistic example here might be a user comparing a plan, in Queensland (no DST), for a rental property in NSW (with DST), having this clearly articulated is more useful than not especially if things like usage charts are aligned with the users local time (which, they are, for a vast number of retailers).

It's unclear what the requirement being solved is but I think the idea is that a plan might have a time range of say 5-8pm local time, which means during DST it effectively "shifts" one hour on a UTC basis? That isn't so much a timezone issue as much as a way of describing "local time". I provided an answer to that which remained international standards aligned already without forcing the loss of precision here: https://github.com/ConsumerDataStandardsAustralia/standards-maintenance/issues/505#issuecomment-1278464803 .

I also identified inconsistent use of timezones (EST vs. LOCAL) in AER/EME data across plans/retailers which makes forcing it back to an unqualified time potentially also inaccurate, albeit in a different way. I guess repeating what I said in the comment "[...] leaving it as HHMM has the effect of making it impossible to ever have any precision despite the possibility it may be available". I also asked for a copy of the underlying document that defines the data specification EME receives to consider this information but was rejected on two separate occasions (here's the one that's public) essentially because it was commercially important to AER.

In summary I think permitting (although I don't think it's excluded now?) -00:00 to designate local time is internationally aligned and meets the objective while allowing for precision where it is known (which it probably is in the context of Energy Account Detail).

[HemangCDR]

I believe the issue is that daylight savings changes are not applicable to energy plan times. For e.g. a plan with rates applicable from 07:00 to 15:00 would be applicable for that specific time range regardless if the local time is in AEST or AEDT (@eme-admin please correct me if I've got this wrong).

As per my interpretation of RFC3339 (https://datatracker.ietf.org/doc/html/rfc3339#section-4.3), "-00:00" means the time is in UTC as the offset to local time zone is unknown. As a result using -00:00 won't work and can result in incorrect interpretation by the ADR. Using the above example, "07:00:00-00:00" would be interpreted as 5pm AEST or 6pm AEDT and "15:00:00-00:00" would be interpreted as 1am AEST or 2am AEDT.

An option to consider could be to update the description of startTime and endTime fields noting that timzone/UTC component is to be ignored. The time can then be provided in "HH:MM:SSZ" or some similar agreed format and the timzone/UTC component can be ignored . The EnergyPlanContractV2.timeZone​ field can be set as required to either LOCAL or AEST.

[jxeeno]

This issue seems to stem from a misinterpretation of RFC3339 4.3 (Unknown Local Offset Convention) back in #505 as being a means of representing local time (without a timezone). Reading 4.3 again, the first sentence makes clear that UTC time is known ("If the time in UTC is known, but the offset to local time is unknown...") and it's simply that the user's/client local timezone is unknown.

4.4 (Unqualified Local Time), on the other hand, is actually what we want to convey. However, RFC3339 is not equipped to represent local times (without a timezone). Instead, the RFC suggests means of either synchronising to UTC or requesting timezone from the end user which isn't useful in this context.

Despite RFC3339 being a close representation of ISO 8601, it is a subset with its own idiosyncrasies - one of which seems to be that the actual moment in time must be known and unqualified local time is unacceptable.

However, ISO 8601 actually has a way of representing local time! Getting a copy of the standard is tricky, however, there are credible references to this on Wikipedia and parser implementations such as Java DateTimeFormatter and Python time.fromisoformat. So in ISO 8601 world, you would have:

• 15:00:00 to mean 3pm local time, adhering to daylight saving timezone changes in the local timezone. If this was in Sydney, for example, this would mean 3pm AEST during standard time and 3pm AEDT during daylight saving time
• 15:00:00+10:00 would mean 3pm AEST during standard time and 4pm AEDT during daylight saving time
• 15:00:00Z would mean 5am AEST during standard time or 4am AEDT during daylight saving time.

So could the solution be to switch from RFC3339 to ISO 8601 for DateTimeString, DateString and TimeString? We already use ISO 8601 for durations.

Going back to the separate timezone field as suggested by @HemangCDR also solves the problem, however, I would lean towards an industry-standard way of representing date-time as initially envisaged in #505. Although, the suggested "HH:MM:SSZ" is probably not desirable as the "Z" suffix is used to mean "Zulu time" (i.e. UTC) in both RFC3339 and ISO8601.

[jxeeno]

At risk of opening another can of worms, there are also edge cases where the timezone changes can neither be represented as "local" time nor a UTC-offset fixed timezone across the whole year. I suspect the only way to be handle these are by explicit date range definitions of EnergyPlanTariffPeriod.

For example, Essential Energy (see last page) has its own "Summer Time" definition for a subset of customers which does not follow the "Summer Time" definition used by the NSW Standard Time Act:

• "Summer Time", as per Essential Energy's definition, is "...the period from the last Sunday in October at 2am to the last Sunday in March at 3am of the following year. Summer time adjustments will be made to ToU meters."

• "Summer Time", as per the NSW Standard Time Act, is "...a period starting at 2 am on the first Sunday in October in each year and ending at 2 am on the first Sunday in the following April"

Essential Energy customers in Essential Energy Far West (e.g. Broken Hill) also does not follow the local time as per the NSW Standard Time Act.

For Essential Energy's own definition of "Summer Time", you could feasibly represent the applicable periods with separate EnergyPlanTariffPeriods with startDate/endDate and explicit UTC offset timezones for each period. However, for accurate programatic calculations (e.g. using historical usage data to generate an estimate), you would probably want the date fields to include year values as the day of month would vary year-to-year. This is something that is currently omitted in the spec.

{ // EnergyPlanTariffPeriod
  "name": "Essential Energy - Summer Time",
  "startDate": "2023-10-29", // in this example, YYYY is provided as day of month changes year to year.  e.g. in 2024, the date is 2024-10-27
  "endDate": "2024-03-31", // as per above
  "timeZone": "AEDT", // explicit timezone to mean UTC+11, could be optionally omitted if ISO 8601 is adopted
  "timeOfUseRates": [
    {
      "timeOfUse": [{
        "startTime": "17:00:00+11:00", // if ISO 8601 is adopted. otherwise, "17:00:00" + offset specified in "timeZone" is used
        "endTime": "20:00:00+11:00", // as per above
        // ... fields omitted ...
      }],
      // ... fields omitted ...
    }
  ],
  // ... fields omitted ...
}

For a retailer providing a consumer's actual plan details where the metering equipment, tariff and customer location is known, this is likely the best solution for an accurate representation. However, I suspect it would be unlikely this fidelity of time variation is provided to AER by retailers for generic plan data, especially where the periods when timezones change depends on metering equipment, tariff and location (i.e. qualification step required).

[perlboy]

I believe the issue is that daylight savings changes are not applicable to energy plan times. For e.g. a plan with rates applicable from 07:00 to 15:00 would be applicable for that specific time range regardless if the local time is in AEST or AEDT (@eme-admin please correct me if I've got this wrong).

If there was a way of specifying a plan time with a date range this could be solved.

As per my interpretation of RFC3339 (https://datatracker.ietf.org/doc/html/rfc3339#section-4.3), "-00:00" means the time is in UTC as the offset to local time zone is unknown. As a result using -00:00 won't work and can result in incorrect interpretation by the ADR. Using the above example, "07:00:00-00:00" would be interpreted as 5pm AEST or 6pm AEDT and "15:00:00-00:00" would be interpreted as 1am AEST or 2am AEDT.

The issue here is that "local time" isn't a known offset because the sender and receiver may be at indeterminant locations at indeterminant times. If this simply moved to HH:MM all that is happening is one is declaring this is the time at the location of the meter, the ADR is still forced to identify where this is and whether it is currently under DST. The suggestion to use -00:00 was a way of signalling it isn't known what the offset should be while simultaneously preserving the ability to clearly communicate if it is and avoiding the need to redefine/introduce components into the spec.

The example provided is accurate because it may be 5pm-1am AEST or 6pm-2am AEDT. This would continue to be a problem if the format specified was HH:MM. It doesn't solve for the ADR guessing what it actually is.

An option to consider could be to update the description of startTime and endTime fields noting that timzone/UTC component is to be ignored. The time can then be provided in "HH:MM:SSZ" or some similar agreed format and the timzone/UTC component can be ignored . The EnergyPlanContractV2.timeZone​ field can be set as required to either LOCAL or AEST.

This actually seems like a reasonable suggestion that preserves alignment with an international time format but I note that designating "ignore Z" is really no different to using -00:00 and from an implementers perspective is actually worse (-00:00 will cause an exception in good timezone parsers allowing a solution to be developed, Z will be silently accepted). I hope we don't end up with more than the two though but as I understand it that isn't currently the case.

This issue seems to stem from a misinterpretation of RFC3339 4.3 (Unknown Local Offset Convention) back in https://github.com/ConsumerDataStandardsAustralia/standards-maintenance/issues/505#issuecomment-1278464803 as being a means of representing local time (without a timezone). Reading 4.3 again, the first sentence makes clear that UTC time is known ("If the time in UTC is known, but the offset to local time is unknown...") and it's simply that the user's/client local timezone is unknown.

"LOCAL" isn't a known offset and that's the source data from AER that's problematic. To be clear this is the server presenting a plan related to a location that may not be the same as the client (Recipient) or the user (Consumer). I'm certainly not pitching this as the perfect solution but I don't think creating a time format that doesn't solve for the problem and in fact loses accuracy when it may not need to is the solution.

The adoption of RFC3339 in the first place began I believe from the OAuth2 side that baselines on it. I'm pretty wary of confusing two different time formats in a single spec (one of which is ISO which means it's not freely distributable).

For a retailer providing a consumer's actual plan details where the metering equipment, tariff and customer location is known, this is likely the best solution for an accurate representation. However, I suspect it would be unlikely this fidelity of time variation is provided to AER by retailers for generic plan data, especially where the periods when timezones change depends on metering equipment, tariff and location (i.e. qualification step required).

I think this solution is probably the most sustainable long term especially if considered in an expansion beyond the NEM. Introducing a start/endDate side by side with a timezoned date is very likely the most consumable thing to do for Recipients. I didn't suggest it in the first place because the "AER format" was being used as the primary reason to not align in a variety of other cases (i.e. it didn't seem likely it'd get traction). From what we have seen most energy billing systems do precisely this so obtaining the data is actually easier.

While I accept AER may not be collecting this data I'd also note that such limitations put on the spec have cascading effects into the reused components (i.e. Energy Account Detail) and I question the ecosystem of one entity not being requested to uplift their own systems. Put another way whether AER has it or not is an implementation concern and the rest of the ecosystem is in many cases actually degrading their data set to achieve it.

[perlboy]

Final footnote here, introducing start/endDate would also introduce the ability for Retailers to advertise future price changes. This seems like it could be quite valuable and in fact a feature enhancement the CDR could offer over the current AER data for Recipients.

[perlboy]

Suggestion that would at least use RFC3339 as a foundation would be to describe this format as:

String aligned with the timespec-hour ABNF outlined in RFC3339 Appendix A

This would mean that existing RFC3339 libraries could be used to validate and parse albeit through some reflection of potentially private methods.

Alternatively from RFC3339 partial-time from Section 5.6.

[perlboy]

Following MI call, discussion seemed to be heading towards:

1. Introducing an ISO8601 time format (ISOTimeString)
2. Using a format definition (if it's aligned to a specific ISO ref) to HH:MM:[SSZ] or just sticking with ISO8601 and being ok with it

Will leave @HemangCDR to clarify further.

[AGL-CDR]

Setting expectations early here - we've got major milestones between now and May 2024 and a very full backlog. Our preference is that this change carries a FDO of Sept 2024.

[HemangCDR]

Thank you everyone for your input.

Having discussed the feedback, the DSB are proposing the following change to address the time representation issue:

• Update the relevant time fields in energy data to be formatted according to ISO 8601 Times specification. This would allow time to be provided with and without UTC offset (e.g. HH:MM and HH:MM:SS+UTC) in a compliant manner addressing both EME and Data holders requirements
• Specifically, the following fields will have their type updated to ExternalRef and description updated noting that time must be formatted according to ISO 8601 Time specification:
  • EnergyPlanControlledLoad.timeOfUseRates.timeOfUse.startTime
  • EnergyPlanControlledLoad.timeOfUseRates.timeOfUse.endTime
  • EnergyPlanSolarFeedInTariffV2.timeVaryingTariffs.timeVariations.startTime
  • EnergyPlanSolarFeedInTariffV2.timeVaryingTariffs.timeVariations.endTime
  • EnergyPlanTariffPeriod.timeOfUseRates.timeOfUse.startTime
  • EnergyPlanTariffPeriod.timeOfUseRates.timeOfUse.endTime
  • EnergyPlanTariffPeriod.demandCharges.startTime
  • EnergyPlanTariffPeriod.demandCharges.endTime
• The description will be updated to include the following text:
  • _Formatted according to ISO 8601 Times specification_

The above change would have no impacts to current DH and EME implementations. As a result, no FDO is being proposed.

Feedback on the above proposal is welcome.

[joshharris3]

AER would be accepting of this change

[jxeeno]

I think the only outstanding clarification would be how the optional timeZone field would be interpreted in conjunction with the TimeString as part of the transition. The current enums for timeZone are LOCAL or AEST, with AEST being the default if absent.

If the TimeString is explicitly defined with the timezone with the +UTC prefix, it's obvious to a consumer that it's as per ISO 8061. However, the default of AEST makes this transition tricky, as the existing interpretation would place all TimeStrings without timezone in +10:00 instead of local.

Could we introduce a third enum (e.g. ISO8061 or NONE) for timeZone to indicate the time strings provided by the DH conform with the changes specified in the CR? Then in a future version, the timeZone field can be removed and all time strings can be assumed by consumers to conform with this CR?

[DannyDuong]

@HemangCDR can you please ensure that this CR applies to all the startTime and endTime fields in the EnergyPlanResponseV2 schema; which is more than the 4 fields that you've nominated above

[HemangCDR]

@HemangCDR can you please ensure that this CR applies to all the startTime and endTime fields in the EnergyPlanResponseV2 schema; which is more than the 4 fields that you've nominated above

@DannyDuong thank you for picking this up. I've updated the proposal with more fields, hopefully including all of them.

[HemangCDR]

I think the only outstanding clarification would be how the optional timeZone field would be interpreted in conjunction with the TimeString as part of the transition. The current enums for timeZone are LOCAL or AEST, with AEST being the default if absent.

If the TimeString is explicitly defined with the timezone with the +UTC prefix, it's obvious to a consumer that it's as per ISO 8061. However, the default of AEST makes this transition tricky, as the existing interpretation would place all TimeStrings without timezone in +10:00 instead of local.

Could we introduce a third enum (e.g. ISO8061 or NONE) for timeZone to indicate the time strings provided by the DH conform with the changes specified in the CR? Then in a future version, the timeZone field can be removed and all time strings can be assumed by consumers to conform with this CR?

This is a good pickup. An option could be clarify in the description of the time fields that the value of timeZone is not applicable if time is provided with a UTC offset.

[jxeeno]

This is a good pickup. An option could be clarify in the description of the time fields that the value of timeZone is not applicable if time is provided with a UTC offset.

That would clarify for specific UTC offset in timestring, but we'll still need to clarify when timeZone is unspecified and the timestring is unqualified local time (e.g. 15:00). Would the following work?

Example TimeString value	timeZone value	ISO 8061	Interpretation
15:00+11:00	Any value	15:00+11:00	If the offset is provided in the TimeString, the timezone specified in the timestring must be interpreted as per ISO 8061.
15:00	Not defined, or AEST	15:00+10:00	If no timeZone is provided or if AEST is the specified timezone, times must be interpreted as AEST (UTC+10) for backwards compatability.
15:00	LOCAL	15:00	To specify unqualified local time, LOCAL enum value must be specified for the timeZone field.

[CDR-API-Stream]

We gave serious consideration to the introduction of an additional time field type for this energy specific change and have decided to move forward with ExternalRef. We will monitor the usage of this structure and future needs of new sectors, and will revisit the matter as necessary.

This issue has been staged accordingly and can be reviewed here - https://github.com/ConsumerDataStandardsAustralia/standards-staging/compare/release/1.29.0...maintenance/613