{BUG] Attributes for an Entity may exceed maximum allowed Size when using RT Data

[dafunkydan]

Describe the Bug Sometimes (i have the feeling especially at Peak Times) all Zones (Stops) using RT-Data, with all their Sensors fail and render unavailable. All .rt Files for all Stops/Connections going 1kb. The debug file _converted.txt still seem to work, Filesize (~800kb) and Date changes.

Zones (Stops) only using Static Data are unaffected, disabling RT-Data for that Zone and reloading makes them available again.

In Logs i can see e.g.: WARNING (Recorder) [homeassistant.components.recorder.db_schema] State attributes for sensor.241722_local_stop_zone_haltestelle_test_ma_hbf exceed maximum size of 16384 bytes. This can cause database performance issues; Attributes will not be stored

Steps/data to reproduce the behavior: Download the static GTFS: https://rnv-dds-prod-gtfs.azurewebsites.net/latest/gtfs.zip Register for the API Key for RT-Data: https://github.com/vingerha/gtfs2/issues/51#issuecomment-2057470068 and use them Set up a Zone with some Stops Enable RT for that Stop with *.rt as Source Reload, see the next Departures Wait for a crowded Time 😝 See all Sensors going unavailable

Release used gtfs2: 0.4.4.7 HA: HAOS Core: 2024.4.3

Additional See Attachment example log for rt data causing attributes overflow.txt

[vingerha]

1. Is this only (!) with sensors having RT data? Can you possibly setup a similar set without RT and compare?
2. Does the JSON continue to update even if the .rt is empty?
3. Can you possibly switch on constant debug log and see if something is thrown for this, you'd need to do this via the configuration.yaml as via the GUI it would auto-deactivate debug loggingafter some time
4. For as far as I know, the size warning is only that, it does not mena much, I have 3 others in other integrations with considerable attibutes at times but they pose no further issues other than truncating the attributes and throwing this warning (note that it is not an error-message).

[dafunkydan]

1. Is this only (!) with sensors having RT data?

   Zones (Stops) only using Static Data are unaffected, disabling RT-Data for that Zone and reloading makes them available again

Yes, it is only for my Zones (=multiple Sensors) with RT Data. I can easily turn that on/off. So yes, i turned a Zone beeing static, and that was still (again) working.

2. Does the JSON continue to update even if the .rt is empty?

   The debug file *_converted.txt still seem to work, Filesize (~800kb) and Date changes.

With JSON you mean the debug_file, right? Yes, that keeps updating.

3. Can you possibly switch on constant debug log and see if something is thrown for this

Sure! As this happens only with RT Data, i could simply trigger the Problem and log it. Maybe i focused on the wrong Line - maybe this from the attached log is somewhat more useful? If needed yes i can set up the constant debug log and go for a hunt!

2024-04-16 07:50:45.799 ERROR (MainThread) [custom_components.gtfs2.coordinator] Unexpected error fetching d08c657b55db76ad8e5a76043fe45c02 data: 'NoneType' object is not iterable
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 315, in _async_refresh
    self.data = await self._async_update_data()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/gtfs2/coordinator.py", line 244, in _async_update_data
    self._data["local_stops_next_departures"] = await self.hass.async_add_executor_job(
                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/gtfs2/gtfs_helper.py", line 873, in get_local_stops_next_departures
    next_service = get_rt_route_trip_statuses(self)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/gtfs2/gtfs_rt_helper.py", line 168, in get_rt_route_trip_statuses
    for entity in feed_entities:
TypeError: 'NoneType' object is not iterable

[vingerha]

And your rt url is in the sensor or ... via the service call and then use a reference to the rt-file?

[vingerha]

I guess you use the service call while it would allow you to embed the changing api-key. Further guessing is that 'sometmes' you cannot get the data from the external url for whatever reason and thus the thing crashes. The json will not update with the same pace as it needs the rt first. When you have a 1kb rt, can you please open it and examine the content ... as it is 1kb I assume it contains an error from the rt-providing side, posisbly apikey mismatch or timeout. I need to know a bit more on how to A. prevent the error B. maybe build in a re-try

[dafunkydan]

via the service call and then use a reference to the rt-file?

Yes, like that.

Further guessing is that 'sometmes' you cannot get the data from the external url for whatever reason and thus the thing crashes. The json will not update with the same pace as it needs the rt first.

But why would it continue to update the debug_file? Isnt the _converted.txt build from the .rt, should't that stay empty or not get created at all?

16.04.2024  08:00                63 GTFS-Static-RNV1.rt
16.04.2024  08:00           801.666 GTFS-Static-RNV1_converted.txt
16.04.2024  08:01                63 Haltestelle Rohrlachstrasse_localstop.rt

When you have a 1kb rt, can you please open it and examine the content

Oh my, why am i so stupid? I once looked at a correct *.rt file, didnt understand anything, and then just forgot to recheck 😵‍ {"message":"Cannot read properties of null (reading 'header')"}

Yet, i don't understand how the debug_file can get created, if it depends on the *.rt File...

[vingerha]

Yet, i don't understand how the debug_file can get created, if it depends on the *.rt File...

It cannot, I am near to 100% sure that you look at the result of a previous run.

From the file-content above, I cannot resolve this untill ou can demonstrate it is somewhere in the process :) Closing this ticket