[Bug]: cannot setup realtime integration for WMATA

[tylerhartwig]

Own Analysis

• [X] I cannot find my issue in the documentation

A clear and concise description of what the bug is.

I cannot setup realtime integration because more than 15 stops are found for my zone, even with a radius of 50.

How can I resolve this so I can get realtime train updates?

Provide details on the data source(s)

I just setup the integration with this endpoint

https://api.wmata.com/gtfs/rail-bus-gtfs-static.zip

Config in case of setup using start / stop station

No response

Config in case of setup using location based departures

Lat 38.880826296077004
Long -76.98508772924833

Info on your installed version

Latest hacs on haos

Relevant log output

No response

[vingerha]

Not hoped for but .. I was awaiting this at-some-point. The reason to limit is performance related and I just took 15 as 'a' number to limit it. What you can do is change config/custom_components/gtfs2/const.py DEFAULT_MAX_LOCAL_STOPS = 15 and restart HA. The only downside is that with each upgrade (or re-install) of gtfs2 you would need to redo this to setup new stops, I believe it will continue to run but not work for new configurations.

[vingerha]

The regular alternative is to introduce multiple local stops but I assume that in this case that would not work as they would all cover the same location

[tylerhartwig]

Got it, I will try this. In this context what is a "local stop"?

I assumed each entity is what it's referring to, but that particular service has less than 15 entities.

[vingerha]

https://github.com/vingerha/gtfs2/wiki/2c.-Acquire-local-stops-&-departures

If you define different lat/lon and with ranges small enough it may be able to cover part of the available stops, not sure if that will apply to your situation

[tylerhartwig]

Got it, ya I'm not too too worried about performance in this case, running on more than a pi :)

I tried to add real time trip updates to a local stop with this endpoint

https://api.wmata.com/gtfs/rail-gtfsrt-tripupdates.pb

Both with and without the protobuf header it results in the senor being unavailable with a red exclamation mark.

Any thoughts? Or I can provide whatever logs may be helpful

[vingerha]

Logs? And that endpoint is unavailable without subscription, so is your config ok?

[stboch]

@vingerha I have this reproduced for two of my providers, but without dox'ing myself can you send me an email so I can send you these privately. Stboch@gmail.com

[vingerha]

Please check logs for errors/warnings, I cannot test each individual provider / user / stop as you can understand. I tested wmata in Feb and it worked then

[stboch]

It's not all that valuable information for me to understand, but here are the only errors I get, basically. WMata works for Specified Stops, but the vicinity-based multi-stop breaks when RealTime is added; it works fine without real time.

2024-11-08 08:59:34.208 DEBUG (MainThread) [custom_components.gtfs2.coordinator] Finished fetching c422ff9d317b87332c8e45da3631fd4a data in 0.314 seconds (success: False) 2024-11-08 08:59:34.208 ERROR (MainThread) [homeassistant.helpers.entity_platform] gtfs2 raises exception ConfigEntryNotReady in forwarded platform sensor; Instead raise ConfigEntryNotReady before calling async_forward_entry_setups

[vingerha]

The config not ready is not a good sign, which version are you using?

[stboch]

0.5.4 on 2024.11.0

[vingerha]

ok and you use the stop with above lat/lon ?

[vingerha]

I am updating the static now and tested realtime so I have access but need your stop detail

[stboch]

latitude: 39.094131071199115 longitude: -76.85876412779196

[vingerha]

error replicated... not sure yet what it is but will try to fix over the w/e

[stboch]

Glad you were able to recreate it. I couldn't figure it out either was racking my brain, best I could think it something is falling out of an async process. Feel free to @ me if you have anything you want me to test.

[vingerha]

I have it working (again), some thing I overlooked when setting up the service call where I missed adding 'headers' Anyhow, the thing is...it is working without errors but it is not providing any realtime-results and to analyse this is too much for me as it is often a data-provider issue which takes a lot of (unpaid) time. My proposal is that you use the service/action to generate the realtime pb and refer to that file from the configured local stop realtime-url (http://...ip-address:8123/local/gtfs2/filename.pb) . When you setup the action, you can select debug_output to have a readable output, this may help identify if your stops are actually represented in the realtime data. Action needs steady refresh so you can also set this up as a automation and run it each minute ... but then the readable output updates as well each minute. ... it is a challenging task

https://github.com/vingerha/gtfs2/wiki/4.-Service-call

[vingerha]

In the meantime I discovered why it likely won't work as I am in a different timezone as the wmata agency TZ. Changing TZ is only possible via restarts so will see when I have more motivation. What I did notice is that a setup of 200m around your lat/lon gives 9 stops and when I add realtime, my instance seems to turn inaccessible over time, which I believe to put with the high amount of checks it has to perform.

[vingerha]

New update, it works (sort off) with setting the TZ to NY, but I need a bit more time as it does not align well (5h diff) but this may be related to the real TZ I am in (CET)...zooming in, likely publishing new release by tomorrow

[stboch]

You have a dev branch you are working from? I can test and provide more debug output.

[vingerha]

I will create a DEV release in the next 10 mins, is easier

[stboch]

By the way, the test of pointing to the scheduled RT also worked, but I'll let you know if the header issue is fixed in the release. Thanks for all the help on this.

[vingerha]

I still need to test if it works for regular trips with start and end station... it takes a lot of time but as long as I am fixing bugs it is ok

[stboch]

It's not WMATA but one of my other data sources. I'm not sure why, on one side, it has the key, but the other doesn't. I recreated these stops just to confirm. RTA-Home-Bus_localstop.rt contains the Error {"errorCode":401,"errorMessage":"Missing Authorization Header"}

This same feed works fine when manually triggering via service as well as via the specified stop.

2024-11-12 12:10:29.576 DEBUG (SyncWorker_60) [custom_components.gtfs2.gtfs_helper] self rt_data: {'url': 'https://api.goswift.ly/real-time/rtamaryland/gtfs-rt-trip-updates', 'api_key': None, 'api_key_name': 'Authorization', 'api_key_location': 'header', 'accept': None, 'file': 'RTA-Home-Bus_localstop'}, self headers: {'Authorization': '5622XXXXXcf', 'api_key_location': 'header', 'api_key_name': 'Authorization', 'Accept': 'application/x-protobuf'}, self data: {'schedule': <pygtfs.schedule.Schedule object at 0x7fe54ded4230>, 'include_tomorrow': True, 'gtfs_dir': 'gtfs2', 'name': 'RTA-Home-Bus', 'file': 'RTAMaryland', 'offset': 0, 'timerange': 30, 'radius': 1000, 'device_tracker_id': 'zone.home', 'extracting': False, 'gtfs_updated_at': '2024-11-12T17:10:29.568361+00:00'}
2024-11-12 12:10:29.576 DEBUG (SyncWorker_60) [custom_components.gtfs2.gtfs_rt_helper] Getting gtfs rt locally with data: {'url': 'https://api.goswift.ly/real-time/rtamaryland/gtfs-rt-trip-updates', 'api_key': None, 'api_key_name': 'Authorization', 'api_key_location': 'header', 'accept': None, 'file': 'RTA-Home-Bus_localstop'}
2024-11-12 12:10:29.576 DEBUG (SyncWorker_60) [custom_components.gtfs2.gtfs_rt_helper] Getting gtfs rt locally with headers: {'Authorization': None}

[vingerha]

Please send me a screenshot of the service call that works, hide the key of course

[stboch]

Attached both the service call and a snip it of the _converted.txt as part of the debug output, also had previously pointed at the RT at that those manual service calls, and it worked fine.

[stboch]

I see the issue... api_key has already been renamed in headers to the CONF_API_KEY_NAME

            CONF_API_KEY : self._headers.get(CONF_API_KEY,None),
            CONF_API_KEY_NAME : self._headers.get(CONF_API_KEY_NAME, None),

it should be, unless you change the upstream of _headers source

            CONF_API_KEY : self._headers.get(self._headers.get(CONF_API_KEY_NAME, None),None),
            CONF_API_KEY_NAME : self._headers.get(CONF_API_KEY_NAME, None),

[vingerha]

Did you install the new release? I have the impression that you did not as the logging changed

[vingerha]

And with me it shows fine

EDIT: nope, sorry... I missed something

[stboch]

Yeah, the easy way to verify is use any feed that doesn't use api_key as the key name. This is why WMATA works fine for me now but my RTA Transit doesn't because swifty uses Authorization instead.

[vingerha]

I have no idea right now, as I donot have the key it crashes on the file but I do see that the key_name is properly used at the beginning of the code... will test more

[vingerha]

tomorrow...day is at an end for me :)

[stboch]

Have a great night... here is my recommended fix.

‎custom_components/gtfs2/gtfs_helper.py :991

-            CONF_API_KEY : self._headers.get(CONF_API_KEY,None),
+           CONF_API_KEY : self._headers.get(self._headers.get(CONF_API_KEY_NAME, None),None),

2024-11-12 12:51:17.187 DEBUG (SyncWorker_42) [custom_components.gtfs2.gtfs_helper] self rt_data: {'url': 'https://api.goswift.ly/real-time/rtamaryland/gtfs-rt-trip-updates', 'api_key': '562XXXXXacf', 'api_key_name': 'Authorization', 'api_key_location': 'header', 'accept': None, 'file': 'RTA-Home-Bus_localstop'}, self headers: {'Authorization': '562xxxxxxeacf', 'api_key_location': 'header', 'api_key_name': 'Authorization', 'Accept': 'application/x-protobuf'}, self data: {'schedule': <pygtfs.schedule.Schedule object at 0x7f1444ddfb60>, 'include_tomorrow': True, 'gtfs_dir': 'gtfs2', 'name': 'RTA-Home-Bus', 'file': 'RTAMaryland', 'offset': 0, 'timerange': 30, 'radius': 1000, 'device_tracker_id': 'zone.home', 'extracting': False, 'gtfs_updated_at': '2024-11-12T17:51:17.171603+00:00'}
2024-11-12 12:51:17.187 DEBUG (SyncWorker_42) [custom_components.gtfs2.gtfs_rt_helper] Getting gtfs rt locally with data: {'url': 'https://api.goswift.ly/real-time/rtamaryland/gtfs-rt-trip-updates', 'api_key': '5622xxxxxxxxeacf', 'api_key_name': 'Authorization', 'api_key_location': 'header', 'accept': None, 'file': 'RTA-Home-Bus_localstop'}
2024-11-12 12:51:17.187 DEBUG (SyncWorker_42) [custom_components.gtfs2.gtfs_rt_helper] Getting gtfs rt locally with headers: {'Authorization': '5622XXXXXXacf'}

[vingerha]

the fix is in the gtfs_rt_helper... but there it looks fine and your log also shows Authorization as a key...hence. It just does not have the actual value (i.e. it shows none)...did you reload after adidng teh key-value?

[stboch]

Here is a breakdown of the issue.

api_key is not passed in the _headers as it is already written out as the value of api_key in my example api_key = Authorization In the data below I have bolded the areas that are the issue. If the values of API_KEY and API_KEY_NAME are stored elsewhere, then probably better than reading from the formatted self._headers

2024-11-12 12:10:29.576 DEBUG (SyncWorker_60) [custom_components.gtfs2.gtfs_helper] self rt_data: {'url': 'https://api.goswift.ly/real-time/rtamaryland/gtfs-rt-trip-updates', 'api_key': None, 'api_key_name': 'Authorization', 'api_key_location': 'header', 'accept': None, 'file': 'RTA-Home-Bus_localstop'}, self headers: {'Authorization': '5622XXXXXcf', 'api_key_location': 'header', 'api_key_name': 'Authorization', 'Accept': 'application/x-protobuf'}, self data: {'schedule': <pygtfs.schedule.Schedule object at 0x7fe54ded4230>, 'include_tomorrow': True, 'gtfs_dir': 'gtfs2', 'name': 'RTA-Home-Bus', 'file': 'RTAMaryland', 'offset': 0, 'timerange': 30, 'radius': 1000, 'device_tracker_id': 'zone.home', 'extracting': False, 'gtfs_updated_at': '2024-11-12T17:10:29.568361+00:00'}

[vingerha]

That is what I wrote too, I got a fix and am trying to improve error handling at the same time, next release likely tomorrow

[stboch]

@vingerha thanks, from one unpaid dev to another. Feel free to let me know if you have any updates and I'll test. Pretty good at QA work and breaking things ;)

[vingerha]

Will try to re-release later this afternoon, after work...and one cannot learn (much) without breaking things :)

[vingerha]

Tested it with wmata local stops and start<>end, did the same for my local provider so covered different TZ. I could not test goswift as not sure where to register for API (and ... too lazy too :) )

[stboch]

Working great for both my data feeds! Thanks for all your work on this project!

[vingerha]

Thx, closing this topic now. I will need to do some drastic debug-log clean up now :)