dataknut
commented
6 years ago yes, this is badly documented. The relevant code and explanations are scattered across R/gridSpy.R and some of the examples. I'll upgrade the documentation.
The brief answer is that some of the data was originally downloaded in NZ time (hence the TZ_orig = date NZ rows) and some in UTC. The data processing code does as good a job as it can in fixing the former to match up with the latter (r_dateTime is always UTC) but there may well be glitches especially as we think there may have been attempts to fix the DST switch hours in some way before we downloaded the data and we either do not understand what was done or it made things worse (or both).
The data processing duplicate check is done on linkID <-> r_dateTime (i.e. UTC) <-> circuit <-> powerW so would not pick up the repeated NZT values which are within the DST hour and may be a side effect of whatever fix was attempted.
I use readr::read_csv to load the .csv files as it auto-parses r_dateTime into NZ time (by auto-setting the tz from current location). You can then use lubridate to force/check tz if you wish. You can also use readr::read_csv to load as char and then do your own conversions. Always force dateTime_orig to load as char as it is a completely unreliable (and sometimes unparseable) date time.
My curent advice to all users is to avoid the DST hours/days entirely…!
I hate timezones but I hate DST even more...
@jkmair notes:
There are duplicated (or missing) observations if you use NZT in the DST break hours. e.g.:
The hours 02:00-02:59 NZ time are repeated at the end of DST in April. r_dateTime NZT 2015-04-04T13:59:00Z 5/04/15 2:59 2015-04-04T14:00:00Z 5/04/15 2:00 ... ... 2016-04-02T13:59:00Z 3/04/16 2:59 2016-04-02T14:00:00Z 3/04/16 2:00
There seems to consistently be a one hour period removed at 11:00-11:59UTC on the date of DST ending. However, this does not lineup with the NZ time of DST. Not sure if this is a gridSpy issue or data processing. r_dateTime NZT 2015-04-05T10:59:00Z 5/04/15 22:59 2015-04-05T12:00:00Z 6/04/15 0:00 ... ... 2016-04-03T10:59:00Z 3/04/16 22:59 2016-04-03T12:00:00Z 4/04/16 0:00
A one hour period corresponding to the start of DST, 14:00-14:59 UTC, is removed. This results in a 2 hour gap in NZ time. r_dateTime NZT 2015-09-26T13:59:00Z 27/09/15 1:59 2015-09-26T15:00:00Z 27/09/15 4:00 ... ... 2016-09-24T13:58:00Z 25/09/16 1:58 2016-09-24T15:00:00Z 25/09/16 4:00
-> For some reason the duplicate entries have a TZ_orig of "date NZ", while all of the non-duplicate entries are "date UTC".
How is (or should) DST be handled?