DST Switchover Behavior

[sherson]

I emailed Avail about this last week, and this morning I was informed they're looking into it. Issue created since this is coming up in 12 days (I'll update it with what Avail comes back with):

How do the Avail realtime feed and bus tracker deal with daylight savings between the switchover (DST ends at 2:00 AM on Sunday, Nov 2nd) and the end of bus service from the previous day?

Our day changes at ~4 AM, and we run Saturday evening service on several routes until after 2 AM Sunday (Routes 31, 38, 39, and B43, that I know of).

The realtime feed renders times in ISO 8601 format (2014-10-15T20:45:00), but it doesn't have time zone (2014-10-15T20:45:00-0400) and isn't UTC (2014-10-16T00:45:00Z).

Will the times in the feed change along with the DST right at 2:00? Or will they remain consistent with the previous day's times until the end of service, and then change when the next day's service begins? (I assume it's the latter)

[werebus]

Not to be pedantic, although it is my wont, but if it doesn't have the UTC offset or the "Z", then it isn't ISO 8601. It's just something you're (Avail is) making up.

[werebus]

Oh no! Oh no! If only I'd done just a smidge more reading before posting. I'm totally wrong, ignore the above comment. I'm I huge jerkface. :sheep:

[bcspragu]

:+1:

[sherson]

:v:

[sherson]

Got a reply from Avail... it's the assumed behavior:

You are correct, the times will remain consistent with the previous day's times until the end of service, and then change when the next day's service begins, so the scheduled times remain linear for that service day.

So buses will show up ahead of schedule by an hour (after 2 AM on Nov 2nd). @werebus suggested that we specify the timezone based on what it was when the service day started (perhaps with moment.js?).

[dfaulken]

Specifically, I think this might be of use to us:

moment('2014-03-09T01:59:00+04:00').isDSTShifted()//March 9th, 1:59 AM
false
moment('2014-03-09T02:00:00+04:00').isDSTShifted()//March 9th, 2:00 AM
true

I'll put together a pull request tomorrow.

[sherson]

Possibly closed by eaf6e78ef90ae1e824c0e9d804eaddc3333f5264, but leaving open until we see how it behaves at 2 AM on Sun (we couldn't think of an easy way to test this one).

[dfaulken]

Looked at 1:36 AM, times were already off by an hour (everything was shown at being in an hour or more).

[dfaulken]

At 1:00 AM after the change, times became correct. Hmm...

[dfaulken]

Correction: the 30 showed well past when it should have. Could be an unrelated glitch.

[sherson]

I looked at ~12:30 AM, and saw that 30 was correct, but 31 was off by an hour. Very strange.

I realized Friday night that work_day_start defaulting at zero would be an issue. @werebus concurred, but also noticed that it's even more broken. See #13.

[werebus]

Sooooo.... It turns out that Avail wasn't entirely correct above.

Even before the time change, their (JSON) API was returning UNIX timestamp-ish values that were already adjusted for the time change (sort of). Here's an example.

The 39 leaves Blanchard Hall for "Hampshire College" at 1:50am The estimated departure time in the feed returned "/Date(1414911000000-0500)/" which, ignoring the bogus timezone at the end, evaluates to: 2014-11-02T01:50:00-05:00 which is just wrong

The bus actually leaves at 2014-11-02T01:50:00-04:00, so the info board ends up saying that the bus is leaving the stop an hour later than it actually is ("3 hours" instead of "2 hours"). It's particularly noteworthy that this behavior was observed before the time change for a departure that was also before the time change.

The magic time appears to be 1:00am. Schedule times that are after 1:00 are interpreted on Avail's end as being in EST and then converted to a number of milliseconds after 1970-01-01T00:00 This is definitely an error in the API

It's worth mentioning that the XML feed did behave as they said it would. It returned 2014-11-02T01:50:00 which is at least sort of right. If the JSON feed had the same date format (given that said format is a valid date format in Javascript, while what is actually there isn't), there wouldn't be a problem; our code would have worked correctly.

[sherson]

Thanks. Nice work, as usual.

BTW, that explains what I saw last night this morning... the 30 trip was correct because it was estimated/scheduled before 1:00; the 31 trip was off because it was after 1:00.

Will warn OIT and PVTA, and mention it to Avail (and suggest they consider including timezone or switching to UTC).

[werebus]

Honestly, suggest that the JSON feed use the same date and time format as the XML feed. At least then it would be ambiguous but not wrong. The problem is that the JSON feed does include time zone information in a fashion, just the wrong time zone.

[sherson]

That'll be my fall back.

[sherson]

From Avail yesterday:

At this point, it has been logged into the engineering cases, but no work has been scheduled yet. The next step here is to schedule a review of the issue details with the necessary staff.

[sherson]

We talked about it again during today's conference call and the status of the issue hasn't changed. So at this point I don't see it getting fixed in time for the Spring forward switchover on 2015-03-08.

[dfaulken]

:100:

[sherson]

I brought this up at yesterday's Avail meeting and emailed Andrew about it.

[dfaulken]

Just to confirm here in the public repo, this is still an issue. No movement has been made on fixing the vendor endpoint.

[bcspragu]

This thread is thoroughly entertaining. :popcorn:

[sherson]

To address this, Avail has added additional LocalTime fields, which went live last Sunday. For example, from https://bustracker.pvta.com/InfoPoint/rest/StopDepartures/Get/63

          {
            "EDT": "/Date(1518637623000-0500)/",
            "SDT": "/Date(1518637623000-0500)/",
            "ADT": null,
            "ATA": null,
            "ETA": "/Date(1518637623000-0500)/",
            "STA": "/Date(1518637623000-0500)/",
            "EDTLocalTime": "2018-47-14T02:47:03",
            "SDTLocalTime": "2018-47-14T02:47:03",
            "ADTLocalTime": null,
            "ATALocalTime": null,
            "ETALocalTime": "2018-47-14T02:47:03",
            "STALocalTime": "2018-47-14T02:47:03",
            "Dev": "00:00:00",
            "Trip": {
              "TripId": 1430,
              "RunId": 331,
              "BlockFareboxId": 331,
              "TripRecordId": 42025,
              "RunRecordId": 0,
              "BlockRecordId": 80,
              "ServiceLevelRecordId": 81,
              "StopSequence": 1023,
              "PatternRecordId": 2676,
              "TripStartTime": "/Date(1136143800000-0500)/",
              "TripStartTimeLocalTime": "2006-30-01T02:30:00",
              "InternalSignDesc": "33 Puffers Pond",
              "InternetServiceDesc": "Puffers Pond",
              "IVRServiceDesc": "Puffers Pond",
              "TripDirection": "N"
            },
            "LastUpdatedLocalTime": "2018-18-14T02:18:52",
            "LastUpdated": "/Date(1518635932680-0500)/",
            "Bay": null
          }

However, they're currently 12 hour clock. We've asked that they use 24 hour clock.

[sherson]

It's going to be 24 hour clock:

Avail said that new build passed testing; we're now just waiting to hear back about when it will be deployed (it should be before the DST switchover on 3/11).

[sherson]

24 hour clock fix been deployed.

[dfaulken]

What will the behavior of these LocalTime fields be during a time change? Are we essentially saying that we should can use these new fields instead of the ones we currently do, and then we can remove our existing DST switchover logic?

[sherson]

^ didn't see your question here, but I replied in #76