How to use Entity States as API-Key / Curling RT Data sucessfully, but Error when using the Integration

dafunkydan commented 2 months ago

Describe the bug I am facing two Problems integrating my Local Transportations' Provider Realtime-Data:

The API Key is dynamic, and needs to get refreshed regulary. I managed to generate the API Key with a Rest-Command and make it available as Sensor. How can i make the Integration use this Sensors' Value as API Key?
For Testing purpose, i pasted the momentary API Key into the field when setting up the Realtime Information. After reloading the Entry, the corresponding Entities render unavailable. I tried all combination of API-Key, X-API-Key, header, and query_string. when using 'curl' to query the Realtime-Data, I sucessfully get a JSON Response, which looks valid to me (see below).

Release used gtfs2: 0.4.4 HA: HAOS Core: 2024.4.3

Additional Path for static Zip-File: https://rnv-dds-prod-gtfs.azurewebsites.net/latest/gtfs.zip

Provided Realtime Query Information Sample from the Data Provider:

### aktueller tripupdates feed, json format
GET https://gtfs-rt-sandbox-dds.rnv-online.de/tripupdates/decoded
Authorization: Bearer {{getAccessTokenForClient.response.body.$.access_token}}

Querying the RT-Data with (Sample Key Expired):

curl -X GET \
     -H "Authorization: Bearer ****redacted****" \
     https://gtfs-rt-sandbox-dds.rnv-online.de/tripupdates/decoded

Results in (Example stripped):

{"header":{"gtfsRealtimeVersion":"2.0","incrementality":"FULL_DATASET","timestamp":"1712999280"},"entity":[{"id":"7-861-1007-39480","tripUpdate":{"trip":{"tripId":"7-861-1007-39480","scheduleRelationship":"CANCELED"}}}]}

Debug Log when Configuring the Integration with Option API-Key / Header:

2024-04-13 12:11:11.934 DEBUG (MainThread) [custom_components.gtfs2.config_flow] Checkstops option with data: {'local_stop_refresh_interval': 1, 'radius': 250, 'timerange': 30, 'real_time': True, 'file': 'GTFS-Static-RNV1', 'url': 'na', 'extract_from': 'zip', 'device_tracker_id': 'zone.haltestelle_rohrlachstrasse'}
2024-04-13 12:11:11.935 DEBUG (SyncWorker_3) [custom_components.gtfs2.gtfs_helper] Getting gtfs with data: {'local_stop_refresh_interval': 1, 'radius': 250, 'timerange': 30, 'real_time': True, 'file': 'GTFS-Static-RNV1', 'url': 'na', 'extract_from': 'zip', 'device_tracker_id': 'zone.haltestelle_rohrlachstrasse'}
2024-04-13 12:11:11.936 DEBUG (SyncWorker_3) [custom_components.gtfs2.gtfs_helper] Checking if extracting: GTFS-Static-RNV1
2024-04-13 12:11:11.962 DEBUG (SyncWorker_15) [custom_components.gtfs2.gtfs_helper] Getting local stops list with data: {'local_stop_refresh_interval': 1, 'radius': 250, 'timerange': 30, 'real_time': True, 'file': 'GTFS-Static-RNV1', 'url': 'na', 'extract_from': 'zip', 'device_tracker_id': 'zone.haltestelle_rohrlachstrasse'}
2024-04-13 12:11:11.967 DEBUG (SyncWorker_15) [custom_components.gtfs2.gtfs_helper] Local stops list output: 5
2024-04-13 12:11:11.968 DEBUG (MainThread) [custom_components.gtfs2.config_flow] UserInputs Options Init with realtime: {'local_stop_refresh_interval': 1, 'radius': 250, 'timerange': 30, 'real_time': True, 'file': 'GTFS-Static-RNV1', 'url': 'na', 'extract_from': 'zip', 'device_tracker_id': 'zone.haltestelle_rohrlachstrasse'}
2024-04-13 12:11:31.250 DEBUG (MainThread) [custom_components.gtfs2.config_flow] UserInput Realtime: {'local_stop_refresh_interval': 1, 'radius': 250, 'timerange': 30, 'real_time': True, 'file': 'GTFS-Static-RNV1', 'url': 'na', 'extract_from': 'zip', 'device_tracker_id': 'zone.haltestelle_rohrlachstrasse', 'trip_update_url': 'https://gtfs-rt-sandbox-dds.rnv-online.de/tripupdates/decoded', 'api_key': '****redacted****', 'api_key_location': 'header', 'x_api_key': ''}
2024-04-13 12:11:34.330 DEBUG (MainThread) [custom_components.gtfs2.coordinator] Previous data: None
2024-04-13 12:11:34.331 DEBUG (MainThread) [custom_components.gtfs2.gtfs_helper] Getting gtfs with data: {'file': 'GTFS-Static-RNV1', 'device_tracker_id': 'zone.haltestelle_rohrlachstrasse', 'name': 'Haltestelle Rohrlachstrasse', 'url': 'na', 'extract_from': 'zip'}
2024-04-13 12:11:34.335 DEBUG (MainThread) [custom_components.gtfs2.gtfs_helper] Checking if extracting: GTFS-Static-RNV1
2024-04-13 12:11:34.351 DEBUG (MainThread) [custom_components.gtfs2.gtfs_helper] Checking if extracting: GTFS-Static-RNV1
2024-04-13 12:11:34.354 DEBUG (SyncWorker_28) [custom_components.gtfs2.gtfs_helper] Get local stop departure with data: {'schedule': <pygtfs.schedule.Schedule object at 0x7f5ad882f0>, 'include_tomorrow': True, 'gtfs_dir': 'gtfs2', 'name': 'Haltestelle Rohrlachstrasse', 'file': 'GTFS-Static-RNV1', 'timerange': 30, 'radius': 250, 'device_tracker_id': 'zone.haltestelle_rohrlachstrasse', 'extracting': False, 'gtfs_updated_at': '2024-04-13T10:11:34.351620+00:00'}
2024-04-13 12:11:34.354 DEBUG (SyncWorker_28) [custom_components.gtfs2.gtfs_helper] Checking if extracting: GTFS-Static-RNV1
2024-04-13 12:11:34.355 DEBUG (SyncWorker_28) [custom_components.gtfs2.gtfs_helper] Includes Tomorrow
2024-04-13 12:11:34.383 DEBUG (SyncWorker_28) [custom_components.gtfs2.gtfs_rt_helper] Getting gtfs rt locally with data: {'url': 'https://gtfs-rt-sandbox-dds.rnv-online.de/tripupdates/decoded', 'headers': {'Authorization': '****redacted****'}, 'file': 'Haltestelle Rohrlachstrasse_localstop'}
2024-04-13 12:11:34.688 DEBUG (SyncWorker_28) [custom_components.gtfs2.gtfs_helper] Row from query: {'stop_id': '210001', 'stop_name': 'Rohrlachstraße', 'latitude': 49.4830861, 'longitude': 8.4298181, 'trip_id': '4-921-1004-41640', 'trip_headsign': 'Käfertaler Wald', 'direction_id': None, 'departure_time': '12:12:00', 'route_long_name': '(Bad Dürkheim Bf –) Oggersheim – LU Hauptbahnhof – Berliner Platz – Universität – MA Hauptbahnhof – Wasserturm – Paradeplatz – Abendakademie – Alte Feuerwache – Universitätsklinikum – Waldfriedhof', 'route_short_name': '4A', 'route_type': 0, 'today': 1, 'tomorrow': 0, 'start_date': '2024-04-12', 'end_date': '2024-04-26', 'calendar_date': '2024-04-13', 'today_cd': 0, 'route_id': '4-921-4A'}
2024-04-13 12:11:34.689 DEBUG (SyncWorker_28) [custom_components.gtfs2.gtfs_helper] Find rt for local stop route: 4-921-4A - direction: None - stop: 210001
2024-04-13 12:11:34.689 DEBUG (SyncWorker_28) [custom_components.gtfs2.gtfs_rt_helper] GTFS RT get_feed_entities for url: None , headers: {'Authorization': '****redacted****'}, label: vehicle positions
2024-04-13 12:11:34.691 ERROR (MainThread) [custom_components.gtfs2.coordinator] Unexpected error fetching 4a242706b96303e3e0cbada01b820e82 data: 'NoneType' object has no attribute 'startswith'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 315, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/gtfs2/coordinator.py", line 234, in _async_update_data
    self._data["local_stops_next_departures"] = await self.hass.async_add_executor_job(
                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/gtfs2/gtfs_helper.py", line 872, in get_local_stops_next_departures
    next_service = get_rt_route_trip_statuses(self)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/gtfs2/gtfs_rt_helper.py", line 154, in get_rt_route_trip_statuses
    vehicle_positions = get_rt_vehicle_positions(self)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/gtfs2/gtfs_rt_helper.py", line 232, in get_rt_vehicle_positions
    feed_entities = get_gtfs_feed_entities(
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/gtfs2/gtfs_rt_helper.py", line 66, in get_gtfs_feed_entities
    if url.startswith('file'):
       ^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'startswith'
2024-04-13 12:11:34.704 DEBUG (MainThread) [custom_components.gtfs2.coordinator] Finished fetching 4a242706b96303e3e0cbada01b820e82 data in 0.373 seconds (success: False)
2024-04-13 12:11:34.705 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up gtfs2 platform for sensor
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 315, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/gtfs2/coordinator.py", line 234, in _async_update_data
    self._data["local_stops_next_departures"] = await self.hass.async_add_executor_job(
                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/gtfs2/gtfs_helper.py", line 872, in get_local_stops_next_departures
    next_service = get_rt_route_trip_statuses(self)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/gtfs2/gtfs_rt_helper.py", line 154, in get_rt_route_trip_statuses
    vehicle_positions = get_rt_vehicle_positions(self)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/gtfs2/gtfs_rt_helper.py", line 232, in get_rt_vehicle_positions
    feed_entities = get_gtfs_feed_entities(
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/gtfs2/gtfs_rt_helper.py", line 66, in get_gtfs_feed_entities
    if url.startswith('file'):
       ^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'startswith'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 356, in _async_setup_platform
    await asyncio.shield(awaitable)
  File "/config/custom_components/gtfs2/sensor.py", line 75, in async_setup_entry
    await coordinator.async_config_entry_first_refresh()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 287, in async_config_entry_first_refresh
    raise ex
homeassistant.exceptions.ConfigEntryNotReady: 'NoneType' object has no attribute 'startswith'

vingerha commented 2 months ago

First, 0.4.4 has issues with the key's, try 0.4.4.4 (dev release) Then, the API key header that I am sending is {"Authorization": [API_KEY]} which does not contain Bearer, can you try it without Bearer? For automating, the only solution I can think of is that you use the service call and add a template into it, this will then download the .rt locally in www/gtfs2 and you could connect to that via http:/IPD/local/gtfs2/file.rt

vingerha commented 2 months ago

Any chance your realtime is covered with this one? https://gtfs.de/de/realtime/

vingerha commented 2 months ago

Just noticed that you can just add the word 'Bearer' as well into the field for the API key....maybe that could help too?

dafunkydan commented 2 months ago

First: Wow, you are Lightning Fast! 🚀 Second: Thank you so much for your support, highly appreciate it!! Now:

try 0.4.4.4 (dev release)

Done. What changed: The Entitites no longer render unavailable when turning on RT Data 👍

the API key header that I am sending is {"Authorization": [API_KEY]} which does not contain Bearer, can you try it without Bearer? Just noticed that you can just add the word 'Bearer' as well into the field for the API key....maybe that could help too?

I tried it in the Integration with X-API or API Key, with and without Prefix (no Quotes, just plain API Key, either with or without leading "Bearer "). Unfortunatly always same outcome:

- departure: '13:52:00'
  departure_realtime: '-'

I'm not sure how to interpret the Debuglog, in Case this might be helpful please see Attachment debuglog.txt

For automating, the only solution I can think of is that you use the service call and add a template into it, this will then download the .rt locally in www/gtfs2 and you could connect to that via http:/IPD/local/gtfs2/file.rt

That might be a doable Solution! 👍 But lets put that aside, i will check this after i got it to work using manually updated API Keys.

Any chance your realtime is covered with this one? https://gtfs.de/de/realtime/

Unfortunatly not 😢

vingerha commented 2 months ago

From tyhe logs I cannot see an error so it seems (?) to work. Can you try and use the service call and tick it to also receive readable output ? Then at least you can see if the key/logic works. Not receiving rt data can have multiple reasons...data missing (usually) or data format (less usual) or not the corrcet source.

vingerha commented 2 months ago

Can you send me the link to the rt source ? EDIT...where I can get the API key I mean...the link is in the log

dafunkydan commented 2 months ago

use the service call and tick it to also receive readable output ?

Sure! If i try it with Prefix Bearer:

service: gtfs2.update_gtfs_rt_local
data:
  url: https://gtfs-rt-sandbox-dds.rnv-online.de/tripupdates/decoded
  api_key_location: header
  accept: false
  debug_output: true
  file: GTFS-Static-RNV1
  api_key: >-
    Bearer
    ****redacted****

I get Failed to call service gtfs2.update_gtfs_rt_local. Unknown error Without that Prefix it runs through, but all files in \config\www\gtfs2\ are empty.

Can you send me the link to the rt source ?

Sure! Ill send you in a couple of minutes everything needed, or at least what i do have 😄

dafunkydan commented 2 months ago

As i couldnt find any Adress for you, i think i found you HAs' Forum - Check your PNs there! 😎

vingerha commented 2 months ago

Issue closed with 0.4.4.7

vingerha commented 2 months ago

This would be the service call based on your sensor colelcting the token...add it to automation or so

service: gtfs2.update_gtfs_rt_local
data:
  api_key: |
    Bearer {{ state_attr('sensor.rnv_gtfs_rt_api_key','access_token') }}
  url: https://gtfs-rt-sandbox-dds.rnv-online.de/tripupdates/
  file: rnv
  debug_output: true
  api_key_location: header
  accept: false

And from the gtfs2 integration to refer to the local file instead of the external sandbox

dafunkydan commented 2 months ago

This would be the service call based on your sensor colelcting the token...add it to automation or so

In Case somebody wants to get started with RNVs (Mannheim, Ludwigshafen, Heidelberg) Realtime Data: In Order to get an API Key (hourly changing), you need to register for API at: https://www.opendata-oepnv.de/ht/de/organisation/verkehrsunternehmen/rnv/openrnv/start

They will send you an Email containing Information like this:

clientID=****yourClientID****
tenantID=****yourTenantID****
clientSecret=-****yourClientSecret****
resource=****yourResource****
hostname=https://gtfs-rt-sandbox-dds.rnv-online.de

Now, in order to create an API Key, you can set up a Sensor like this:

- platform: rest
  name: RNV GTFS-RT API Key
  resource: "https://login.microsoftonline.com/****yourTenantID****/oauth2/token"
  method: POST
  scan_interval: 3300
  headers:
    Content-Type: "application/x-www-form-urlencoded"
  payload: "grant_type=client_credentials&client_id=****yourClientID****&client_secret=-****yourClientSecret****&resource=****yourResource****"
  value_template: "{{ value_json.expires_on | int | timestamp_custom('%Y-%m-%d %H:%M:%S') }}"
  json_attributes:
  - not_before
  - access_token
  - token_type

This will get you an API Key valid for an Hour. The Key will be in the Attributes, the State of the Entity contains the Timestamp til when it is valid. From there you can continue with the Automation / Information one Post above:

service: gtfs2.update_gtfs_rt_local

You can easily put this API Refreshment in one big Automation - check if the API-Key is still valid - if not refresh it, and then continue with calling service: gtfs2.update_gtfs_rt_local

vingerha / gtfs2

How to use Entity States as API-Key / Curling RT Data sucessfully, but Error when using the Integration #51