Timezone issue causing "due_in_minutes" to return incorrect value

[StarScream159]

Not giving up on my issue #26 I think I have found the cause.

Sensor.py has a function "due_in_minutes" that seems to be using the wrong timezone in it's calculations which is making all trips "in the past" and therefore ignored. This is why I am getting the "not defined" message when everything points to it working correctly.

On my system dt_util.now().replace(tzinfo=None) returns the time in UTC (which I think it is suppose to) whereas my transit provider is seemingly returning local time. So when the calculation runs it's returning a negative value thereby skipping the trip. Modifying sensor.py slightly:

def due_in_minutes(timestamp):
    """Get the remaining minutes from now until a given datetime object."""
    diff = timestamp - dt_util.now().replace(tzinfo=None)
    log_debug(['stop timestamp:', timestamp, 'dt_util.now():', dt_util.now().replace(tzinfo=None), 'local now:',  datetime.now(), 'due_in_minutes', int(diff.total_seconds() / 60)], 3)
    return int(diff.total_seconds() / 60)

outputs:

DEBUG:sensor:      Stop: 2845 Stop Sequence: 0 Stop Time: 1726855661
DEBUG:sensor:         stop timestamp: 2024-09-20 14:07:41 dt_util.now(): 2024-09-20 18:03:56.486275 local now: 2024-09-20 14:03:56.486277 due_in_minutes -236

So as I'm writing this (at 2024-09-20 2:03 PM), this sample stop is at "1726855661" which is 2 minutes in the future. However the "now" (as returned from dt_util.now().replace(tzinfo=None)) is being returned as +4h. So the calculation is -236 and therefore thinks it's in the past.

I have my timezone set in Home Assistant and everything from the CLI returns what you'd expect (from within the Home Assistant Docker Container):

homeassistant:/config/custom_components/gtfs_rt# printenv | grep TZ
TZ=America/Toronto
homeassistant:/config/custom_components/gtfs_rt# python3 -c "import datetime; print(datetime.datetime.now())"
2024-09-20 11:01:36.998401
homeassistant:/config/custom_components/gtfs_rt# python3 -c "import homeassistant.util.dt as dt_util; print(dt_util.now())"
2024-09-20 15:03:20.890916+00:00
homeassistant:/config/custom_components/gtfs_rt# python3 -c "import homeassistant.util.dt as dt_util; print(dt_util.now().replace(tzinfo=None))"
2024-09-20 15:03:36.629754

Modifying the "due_in_minutes" function to use datetime.now() (which returns local time) instead of dt_util.now().replace(tzinfo=None) allows my test_translink.yaml to return correct values.

trip_update_url: 'https://opendata.hamilton.ca/GTFS-RT/GTFS_TripUpdates.pb'
vehicle_position_url: 'https://opendata.hamilton.ca/GTFS-RT/GTFS_VehiclePositions.pb'
route_delimiter: '-'
departures:
  - name: '21N Upper Kenilworth North Bound'
    route: '5230'
    stopid: '2845'
    directionid: '1'

returns:

homeassistant:/config/custom_components/gtfs_rt# python3 ./test.py -f test_translink.yaml -d INFO
INFO:root:Input file configuration is valid.
INFO:__main__:Adding Sensor: Name: 21N Upper Kenilworth North Bound, route id: 5230, direction id: 2845
INFO:sensor:trip_update_url: https://opendata.hamilton.ca/GTFS-RT/GTFS_TripUpdates.pb
INFO:sensor:vehicle_position_url: https://opendata.hamilton.ca/GTFS-RT/GTFS_VehiclePositions.pb
INFO:sensor:route_delimiter: -
INFO:sensor:header: None
INFO:sensor:Successfully updated vehicle positions: 200
INFO:sensor:Successfully updated trip data: 200
INFO:sensor:Sensor Update:
INFO:sensor:   Name: 21N Upper Kenilworth North Bound
INFO:sensor:   Route: 5230
INFO:sensor:   Stop ID: 2845
INFO:sensor:   Direction ID: 1
INFO:sensor:   Icon: mdi:bus
INFO:sensor:   Service Type: Service
INFO:sensor:   unit_of_measurement: min
INFO:sensor:   Due in: 12
INFO:sensor:   Due at: 14:20
INFO:sensor:   latitude: 43.1923942565918
INFO:sensor:   longitude: -79.81041717529297
INFO:sensor:   Next Service: not defined
homeassistant:/config/custom_components/gtfs_rt# 

Does anyone know what is going on here? From what I can tell the call to the home assistant datetime utility is returning the correct value. So I'm guessing my transit provider is returning local time when it should be returning UTC? If that's the case then the "fix" I applied is incorrect and in fact the proper solution would be to convert the stoptime to UTC and then compare that to the home assistant utility function value - effectively comparing UTC to UTC.

But blanket converting stoptime to UTC is likely incorrect as you don't want to convert values that are already UTC. So what would the proper fix look like here? Something universal.

[mark1foley]

Hi, Apologies for the delayed response - it's been a busy week. I am looking at this and will get back to you shortly. Mark

[mark1foley]

I have checked the realtime feed from the transport provider and the data is correct, i.e. the timestamps are correct. i.e. they are in UTC time. In fact, they are epoch timestamps so are UTC by definition. It looks like you are able to run python commands in your home assistant instance so it looks like you are not running a supervised instance (apologies in advance if I have that wrong). Nothing wrong with that but it does introduce the possibility of other configuration issues.
I had a similar issue reported over the last few days (#38) that turned out being a timezone configuration issues. Would you be able to check your configuration?

[StarScream159]

Thanks for the write-up. I read over the other issue and I am not sure what I could be missing. I think everything is set right.

I am running HAOS within a VM on a Proxmox host. Specifically I used the "Home Assistant OS VM" script from here: https://tteck.github.io/Proxmox/#home-assistant-os-vm

I believe this setup is using docker within the VM within Proxmox. When I check the docker config I see the following being passed:

                "SUPERVISOR=172.30.32.2",
                "HASSIO=172.30.32.2",
                "TZ=America/Toronto",      <-- Timezone set here
                "LANG=C.UTF-8",
                "S6_BEHAVIOUR_IF_STAGE2_FAILS=2",
                "S6_CMD_WAIT_FOR_SERVICES_MAXTIME=0",
                "S6_CMD_WAIT_FOR_SERVICES=1",
                "S6_SERVICES_READYTIME=50",
                "UV_EXTRA_INDEX_URL=https://wheels.home-assistant.io/musllinux-index/",
                "S6_SERVICES_GRACETIME=240000",
                "UV_SYSTEM_PYTHON=true"

Within the docker environment the timezone is set:

➜  ~ docker exec -it homeassistant /bin/bash
homeassistant:/config# printenv | grep TZ
TZ=America/Toronto
homeassistant:/config# exit
exit
➜  ~ 

And within the host VM the timezone is set:

 ~ printenv | grep TZ                     
TZ=America/Toronto
➜  ~ 

My time zone is also set within Home Assistant too:

homeassistant:
  external_url: "https://assistant.****"
  internal_url: "http://192.168.2.121:8123/"
  country: CA
  language: en
  time_zone: America/Toronto

The current time is correct on both the VM and within the docker container:

➜  ~ date                                   
Thu Oct 24 23:28:33 EDT 2024
➜  ~ docker exec -it homeassistant /bin/bash
homeassistant:/config# date
Thu Oct 24 23:28:41 EDT 2024
homeassistant:/config# exit
exit
➜  ~ 

I cannot think of anywhere else I would need to set this. Can you think of something that I can't?

Thanks again for you time on this.

[mark1foley]

Hi, I have no suggestions about your configuration sorry. It's something that I have no experience with.

I am first to admit that I am not a great python programmer. I especially struggle with how it handles timezones. It seems to have taken something relative simple and provided 27 different ways of doing the same thing for no good reason and created a complete mess in the process.

Back at the beginning of the thread you mentioned that using the HA python dt utility gave the wrong answer whereas the python datetime.now() function returned the correct one. All I can suggest is that you look at the source code for dt (https://github.com/home-assistant/core/blob/dev/homeassistant/util/dt.py), in particular how it handles timezones. It's above my level of skill to understand how it might be returning a different result in your environment but maybe you can work it out.

Lastly, I can't really do a lot of testing in HA as Hamilton Street Railway seem to be doing some kind of geo-blocking for the GTFS-RT data (i'm in Australia) so can't test it from my standard HA environment. I can setup a VPN in a non-HA test environment and it works but not the end-to-end test I was hoping to do.

Sorry I couldn't have been of more help.

[StarScream159]

My Python knowledge is limited too. I also hate timezone stuff lol.

When I said that the HA python dt utility was returning the wrong time that was an assumption. I would expect the HA helper utility to return local time for the HA instance (whichever timezone was configured in HA). But maybe it is suppose to return UTC. I am not sure.

I'll read over the dt utility class again and see if I can find anything different. I'll update thread this early next week.