search

vingerha / gtfs2

Support GTFS in Home Assistant GUI-only
https://github.com/vingerha/gtfs2
MIT License
65 stars 4 forks source link

Issue with SNCF TER Real Time #90

Closed tchup closed 1 week ago

tchup commented 2 weeks ago

Hi,

Describe the bug I did not succeed to add the source for SNCF trains Real time data (France). The .zip file and the .sqlite file are created, but nothing happens.

Capture d'écran 2024-06-13 222035

Steps/data to reproduce the behavior:

Release used

Additional Sorry, I don't know how to get the logs in HAOS.

Thanks for your help!

vingerha commented 2 weeks ago

The static data is not correct,, it should be many Mb in size. Remove those files and start again...please provide the source

vingerha commented 1 week ago

Examined this again, seems you used the realtime source to setup the static routes, that will never work.

  1. setup static data with the regular gtfs source
  2. add realtime info, e.g. using above link Closing this topic with above instruction
tchup commented 1 week ago

Sorry about that, I misunderstood the user guide and thought I had to add the realtime source directly.

Thanks for your work and the advice!

tchup commented 1 week ago

Unfortunately it still doesn't work. Realtime source seems to be added correctly, but no realtime update is gathered.

For exemple, trip id OCESN871471F3308845:2024-06-19T00:38:12Z is present and delayed in realtime data (see TER GTFS_converted.txt). But no realtime information is displayed by the sensor.

RT

Sources are

vingerha commented 1 week ago

As you may understand I cannot analyse this for each case/person so will need to ask for more input as qhat you provide is insufficient for me to even judge where it could be You need to compare the stop_id in the entity with the stop_id in the RT file for the same trip_id, not just look at trip_id as sometimes (god knows why this is sometimes) TER leaves out stops in their trips (I have plenty of examples here in the 06)

tchup commented 1 week ago

No worry, I wasn't expecting a deep analysis but rather some guidance on where to look ! I will check further the stop IDs within the files.

Merci encore !

tchup commented 1 week ago

The "stop_id" is correct, but the "direction_id" is 1 in the static data, 0 in the realtime data. Can it be the cause of the issue ? If yes, is there an easy fix or should I contact the GTFS generator?

vingerha commented 1 week ago

So what you are saying is that the static stop shows the same trip_id as the on in teh RT and ths stop_id match too, that should (!) be sufficient. I am not able to look at the code before end of the week but from what I recall it is a deviation from what I then saw as 'logic'...use-case seems to be good so probably I need to update the code

It should not need the direction_id as this should (?) be part of the trip itself, i.e. a trip_id from A>B should be a different id when returning B>A but...well... I have been surprised too often.

tchup commented 1 week ago

Yes, trip_id and stop_id are consistent from what I saw. In the example I checked, the trip was this one in the static data :

FR:Line::714D1126-C14B-45DD-8752-90285B45A41F:,002225,OCESN854047F2479441:2024-06-19T00:38:12Z,854047,1,26236,

and the stop this one :

StopPoint:OCETrain TER-87334508,La Poterie,,48.0923450,-1.6307730,,,0,StopArea:OCE87334508

The realtime data contained this :

{ "id": "OCESN854047F2479441:2024-06-19T00:38:12Z", "trip_update": { "trip": { "trip_id": "OCESN854047F2479441:2024-06-19T00:38:12Z", "start_time": "14:15:00", "start_date": "20240619", "route_id": "", "direction_id": "0" }, "stop_time_update": [ { "stop_sequence": 0, "stop_id": "StopPoint:OCETrain TER-87481648", "arrival": { "delay": 0, "time": 0 }, "departure": { "delay": 1500, "time": 1718800800 } }, { "stop_sequence": 0, "stop_id": "StopPoint:OCETrain TER-87471367", "arrival": { "delay": 1800, "time": 1718802000 }, "departure": { "delay": 1800, "time": 1718802060 } }, { "stop_sequence": 0, "stop_id": "StopPoint:OCETrain TER-87471359", "arrival": { "delay": 1800, "time": 1718802780 }, "departure": { "delay": 1800, "time": 1718802840 } }, { "stop_sequence": 0, "stop_id": "StopPoint:OCETrain TER-87471342", "arrival": { "delay": 1800, "time": 1718803020 }, "departure": { "delay": 1800, "time": 1718803080 } }, { "stop_sequence": 0, "stop_id": "StopPoint:OCETrain TER-87471334", "arrival": { "delay": 1800, "time": 1718803500 }, "departure": { "delay": 1800, "time": 1718803560 } }, { "stop_sequence": 0, "stop_id": "StopPoint:OCETrain TER-87471326", "arrival": { "delay": 1800, "time": 1718803920 }, "departure": { "delay": 1800, "time": 1718803980 } }, { "stop_sequence": 0, "stop_id": "StopPoint:OCETrain TER-87471318", "arrival": { "delay": 1800, "time": 1718804220 }, "departure": { "delay": 1800, "time": 1718804280 } }, { "stop_sequence": 0, "stop_id": "StopPoint:OCETrain TER-87471029", "arrival": { "delay": 1800, "time": 1718804520 }, "departure": { "delay": 1800, "time": 1718804580 } }, { "stop_sequence": 0, "stop_id": "StopPoint:OCETrain TER-87334508", "arrival": { "delay": 1800, "time": 1718804880 }, "departure": { "delay": 1800, "time": 1718804940 } }, { "stop_sequence": 0, "stop_id": "StopPoint:OCETrain TER-87471003", "arrival": { "delay": 1800, "time": 1718805300 }, "departure": { "delay": 0, "time": 0 } } ] } },

Thanks for looking at it, and obviously take your time, there is no hurry!

vingerha commented 1 week ago

Will look at it later, probably tomorrow or w/e. Do you know how overwrite the files in config/custom_components/gtfs2 ? I will provide something so that testing is faster, without a need to create releases (this is confusing for the others)

vingerha commented 1 week ago

I found the culprit. For route-based setup I am trying to extract multiple real-time values, if available. This only works when I use the route which can have mulitple trip_id which I am ignoring to get to 'all' rt-values. Now, your RT data does not have a router (yes...again a new situation) so it will not work as-is. Hereby a fix, could you please test this? gtfs_rt_helper.zip

tchup commented 1 week ago

I tried replacing gtfs_rt_helper.py in the config/custom_components/gtfs2 folder, but when I restarted home assistant I got the following errors. Did I do something wrong?

Detected blocking call to import_module with args ('custom_components.gtfs2',) in /usr/src/homeassistant/homeassistant/loader.py, line 1052: ComponentProtocol, importlib.import_module(self.pkg_path) inside the event loop Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/usr/src/homeassistant/homeassistant/main.py", line 223, in sys.exit(main()) File "/usr/src/homeassistant/homeassistant/main.py", line 209, in main exit_code = runner.run(runtime_conf) File "/usr/src/homeassistant/homeassistant/runner.py", line 190, in run return loop.run_until_complete(setup_and_run_hass(runtime_config)) File "/usr/local/lib/python3.12/asyncio/base_events.py", line 672, in run_until_complete self.run_forever() File "/usr/local/lib/python3.12/asyncio/base_events.py", line 639, in run_forever self._run_once() File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1988, in _run_once handle._run() File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/usr/src/homeassistant/homeassistant/setup.py", line 165, in async_setup_component result = await _async_setup_component(hass, domain, config) File "/usr/src/homeassistant/homeassistant/setup.py", line 320, in _async_setup_component component = await integration.async_get_component() File "/usr/src/homeassistant/homeassistant/loader.py", line 1004, in async_get_component comp = self._get_component() File "/usr/src/homeassistant/homeassistant/loader.py", line 1052, in _get_component ComponentProtocol, importlib.import_module(self.pkg_path)

Unexpected exception importing component custom_components.gtfs2 Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/loader.py", line 1052, in _get_component ComponentProtocol, importlib.import_module(self.pkg_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/util/loop.py", line 131, in protected_loop_func return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/importlib/init.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 1387, in _gcd_import File "", line 1360, in _find_and_load File "", line 1331, in _find_and_load_unlocked File "", line 935, in _load_unlocked File "", line 995, in exec_module File "", line 488, in _call_with_frames_removed File "/config/custom_components/gtfs2/init.py", line 12, in from .coordinator import GTFSUpdateCoordinator, GTFSLocalStopUpdateCoordinator File "/config/custom_components/gtfs2/coordinator.py", line 29, in from .gtfs_helper import get_gtfs, get_next_departure, check_datasource_index, create_trip_geojson, check_extracting, get_local_stops_next_departures File "/config/custom_components/gtfs2/gtfs_helper.py", line 35, in from .gtfs_rt_helper import get_rt_route_trip_statuses, get_gtfs_rt File "/config/custom_components/gtfs2/gtfs_rt_helper.py", line 197 if route_id = "": ^^^^^^^^^^^^^ SyntaxError: invalid syntax. Maybe you meant '==' or ':=' instead of '='?

Setup failed for custom integration 'gtfs2': Unable to import component: Exception importing custom_components.gtfs2 Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/loader.py", line 1052, in _get_component ComponentProtocol, importlib.import_module(self.pkg_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/util/loop.py", line 131, in protected_loop_func return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/importlib/init.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 1387, in _gcd_import File "", line 1360, in _find_and_load File "", line 1331, in _find_and_load_unlocked File "", line 935, in _load_unlocked File "", line 995, in exec_module File "", line 488, in _call_with_frames_removed File "/config/custom_components/gtfs2/init.py", line 12, in from .coordinator import GTFSUpdateCoordinator, GTFSLocalStopUpdateCoordinator File "/config/custom_components/gtfs2/coordinator.py", line 29, in from .gtfs_helper import get_gtfs, get_next_departure, check_datasource_index, create_trip_geojson, check_extracting, get_local_stops_next_departures File "/config/custom_components/gtfs2/gtfs_helper.py", line 35, in from .gtfs_rt_helper import get_rt_route_trip_statuses, get_gtfs_rt File "/config/custom_components/gtfs2/gtfs_rt_helper.py", line 197 if route_id = "": ^^^^^^^^^^^^^ SyntaxError: invalid syntax. Maybe you meant '==' or ':=' instead of '='?

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/loader.py", line 992, in async_get_component comp = await self.hass.async_add_import_executor_job( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/loader.py", line 1063, in _get_component raise ImportError(f"Exception importing {self.pkg_path}") from err ImportError: Exception importing custom_components.gtfs2

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/loader.py", line 1052, in _get_component ComponentProtocol, importlib.import_module(self.pkg_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/util/loop.py", line 131, in protected_loop_func return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/importlib/init.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "", line 1387, in _gcd_import File "", line 1360, in _find_and_load File "", line 1331, in _find_and_load_unlocked File "", line 935, in _load_unlocked File "", line 995, in exec_module File "", line 488, in _call_with_frames_removed File "/config/custom_components/gtfs2/init.py", line 12, in from .coordinator import GTFSUpdateCoordinator, GTFSLocalStopUpdateCoordinator File "/config/custom_components/gtfs2/coordinator.py", line 29, in from .gtfs_helper import get_gtfs, get_next_departure, check_datasource_index, create_trip_geojson, check_extracting, get_local_stops_next_departures File "/config/custom_components/gtfs2/gtfs_helper.py", line 35, in from .gtfs_rt_helper import get_rt_route_trip_statuses, get_gtfs_rt File "/config/custom_components/gtfs2/gtfs_rt_helper.py", line 197 if route_id = "": ^^^^^^^^^^^^^ SyntaxError: invalid syntax. Maybe you meant '==' or ':=' instead of '='?

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/setup.py", line 320, in _async_setup_component component = await integration.async_get_component() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/loader.py", line 1012, in async_get_component self._component_future.result() File "/usr/src/homeassistant/homeassistant/loader.py", line 1004, in async_get_component comp = self._get_component() ^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/loader.py", line 1063, in _get_component raise ImportError(f"Exception importing {self.pkg_path}") from err ImportError: Exception importing custom_components.gtfs2

tchup commented 1 week ago

I corrected line 197 to if route_id == "": and it launched without error. I'll run some testing and keep you updated.

vingerha commented 1 week ago

Sorry about that...I would love to say that I donot make this error often but sadly... I do :)

tchup commented 1 week ago

Still not working. :(

Trip :

FR:Line::6349C213-7B19-4078-BF2B-43CBECDA9545:,000065,OCESN886837F3331184:2024-06-20T00:38:09Z,886837,0,39662,

Stops:

OCESN886837F3331184:2024-06-20T00:38:09Z,15:20:00,15:20:00,StopPoint:OCETrain TER-87726000,0,,0,1, OCESN886837F3331184:2024-06-20T00:38:09Z,15:29:00,15:30:00,StopPoint:OCETrain TER-87726307,1,,0,0, OCESN886837F3331184:2024-06-20T00:38:09Z,15:36:00,15:37:00,StopPoint:OCETrain TER-87726331,2,,0,0, OCESN886837F3331184:2024-06-20T00:38:09Z,15:47:00,15:48:00,StopPoint:OCETrain TER-87722405,3,,0,0, OCESN886837F3331184:2024-06-20T00:38:09Z,16:06:00,16:12:00,StopPoint:OCETrain TER-87723197,4,,0,0, OCESN886837F3331184:2024-06-20T00:38:09Z,16:17:00,16:18:00,StopPoint:OCETrain TER-87723502,5,,0,0, OCESN886837F3331184:2024-06-20T00:38:09Z,16:22:00,16:23:00,StopPoint:OCETrain TER-87723528,6,,0,0, OCESN886837F3331184:2024-06-20T00:38:09Z,16:25:00,16:26:00,StopPoint:OCETrain TER-87723536,7,,0,0, OCESN886837F3331184:2024-06-20T00:38:09Z,16:29:00,16:30:00,StopPoint:OCETrain TER-87723544,8,,0,0, OCESN886837F3331184:2024-06-20T00:38:09Z,16:33:00,16:34:00,StopPoint:OCETrain TER-87723569,9,,0,0, OCESN886837F3331184:2024-06-20T00:38:09Z,16:38:00,16:39:00,StopPoint:OCETrain TER-87723577,10,,0,0, OCESN886837F3331184:2024-06-20T00:38:09Z,16:45:00,16:46:00,StopPoint:OCETrain TER-87723585,11,,0,0, OCESN886837F3331184:2024-06-20T00:38:09Z,16:55:00,16:55:00,StopPoint:OCETrain TER-87743716,12,,1,0,

Realtime :

{
    "id": "OCESN886837F3331184:2024-06-20T00:38:09Z",
    "trip_update": {
        "trip": {
            "trip_id": "OCESN886837F3331184:2024-06-20T00:38:09Z",
            "start_time": "15:20:00",
            "start_date": "20240620",
            "route_id": "",
            "direction_id": "0"
        },
        "stop_time_update": [
            {
                "stop_sequence": 0,
                "stop_id": "StopPoint:OCETrain TER-87726000",
                "arrival": {
                    "delay": 0,
                    "time": 0
                },
                "departure": {
                    "delay": 600,
                    "time": 1718890200
                }
            },
            {
                "stop_sequence": 0,
                "stop_id": "StopPoint:OCETrain TER-87726307",
                "arrival": {
                    "delay": 300,
                    "time": 1718890440
                },
                "departure": {
                    "delay": 300,
                    "time": 1718890500
                }
            },
            {
                "stop_sequence": 0,
                "stop_id": "StopPoint:OCETrain TER-87726331",
                "arrival": {
                    "delay": 300,
                    "time": 1718890860
                },
                "departure": {
                    "delay": 300,
                    "time": 1718890920
                }
            },
            {
                "stop_sequence": 0,
                "stop_id": "StopPoint:OCETrain TER-87722405",
                "arrival": {
                    "delay": 300,
                    "time": 1718891520
                },
                "departure": {
                    "delay": 300,
                    "time": 1718891580
                }
            },
            {
                "stop_sequence": 0,
                "stop_id": "StopPoint:OCETrain TER-87723197",
                "arrival": {
                    "delay": 300,
                    "time": 1718892660
                },
                "departure": {
                    "delay": 300,
                    "time": 1718893020
                }
            },
            {
                "stop_sequence": 0,
                "stop_id": "StopPoint:OCETrain TER-87723502",
                "arrival": {
                    "delay": 600,
                    "time": 1718893620
                },
                "departure": {
                    "delay": 600,
                    "time": 1718893680
                }
            },
            {
                "stop_sequence": 0,
                "stop_id": "StopPoint:OCETrain TER-87723528",
                "arrival": {
                    "delay": 600,
                    "time": 1718893920
                },
                "departure": {
                    "delay": 600,
                    "time": 1718893980
                }
            },
            {
                "stop_sequence": 0,
                "stop_id": "StopPoint:OCETrain TER-87723536",
                "arrival": {
                    "delay": 600,
                    "time": 1718894100
                },
                "departure": {
                    "delay": 600,
                    "time": 1718894160
                }
            },
            {
                "stop_sequence": 0,
                "stop_id": "StopPoint:OCETrain TER-87723544",
                "arrival": {
                    "delay": 600,
                    "time": 1718894340
                },
                "departure": {
                    "delay": 600,
                    "time": 1718894400
                }
            },
            {
                "stop_sequence": 0,
                "stop_id": "StopPoint:OCETrain TER-87723569",
                "arrival": {
                    "delay": 600,
                    "time": 1718894580
                },
                "departure": {
                    "delay": 600,
                    "time": 1718894640
                }
            },
            {
                "stop_sequence": 0,
                "stop_id": "StopPoint:OCETrain TER-87723577",
                "arrival": {
                    "delay": 600,
                    "time": 1718894880
                },
                "departure": {
                    "delay": 600,
                    "time": 1718894940
                }
            },
            {
                "stop_sequence": 0,
                "stop_id": "StopPoint:OCETrain TER-87723585",
                "arrival": {
                    "delay": 600,
                    "time": 1718895300
                },
                "departure": {
                    "delay": 600,
                    "time": 1718895360
                }
            },
            {
                "stop_sequence": 0,
                "stop_id": "StopPoint:OCETrain TER-87743716",
                "arrival": {
                    "delay": 600,
                    "time": 1718895900
                },
                "departure": {
                    "delay": 0,
                    "time": 0
                }
            }
        ]
    }
},
tchup commented 1 week ago

No, each time I change for a trip which is late,, because I don"t know if I will see something if it's on time. This time it was Meximieux - Pérouges to Ambérieu-en-Bugey

vingerha commented 1 week ago

No clue what went wrong but I found a completely different route base on above... you showed La poterie i.e. copy/paste: StopPoint:OCETrain TER-87334508,La Poterie,,48.0923450,-1.6307730,,,0,StopArea:OCE87334508 and now it is something else?

vingerha commented 1 week ago

Almost there ... just a bit of context.... you may not be that interested but still :) For regular bus/tram-routes I take a hard/unique stop_id and that allows to properly search the realtime data. For trains however, there is a different challenge as there can be multiple trains between two cities: short-stop, intercity, tgv, etc. and the main challenge is that they can depart/arrive on multiple platforms both at origin and destination. The platforms are the unique stop_id. If I would use the same method as for bus/tram then each individual stop_id (platform) needs to be specified and you would have to know the exact destination stop_id too. For a larger station with (say) 4 platforms and a destination with (say) 3 platforms, this leads to 12 combinations...which is unmanageable. Hence, for trains I use the 'city' and it searches free-text. This I based on a dataset having both buses, trams and trains Instead of a unique stop_id, it shows the city name and this cannot be found in the realtime data.

What I did no anticipate was a datasource TER which is only trains and still allows both city-city and stop-stop setup....I clearly missed this Working on a solution but this requires a bit of redesign

tchup commented 1 week ago

Sorry for the confiusion about the settings! I'm changing constantly to have a delayed train set when I do the testing.

Thanks for looking into it! I don't have the competency and time to modify myself the code, but I think these details can be useful if someone encounters some similar issues in the future with other sources.

vingerha commented 1 week ago

Try these 3 updates please upd.zip

tchup commented 1 week ago

Yeah, it's working!!! Thank you! image

vingerha commented 1 week ago

Thanks for the feedback, closing (hopefully now for good)

tchup commented 1 day ago

Just for information, SNCF TER alerts are using a truncated trip ID, for example

informed_entity {
    trip {
      trip_id: "OCESN864765F"
    }

instead of OCESN864765F3273412:2024-06-27T00:35:30Z

No need to reopen, because this is not the normal behavior (explained here: https://transport.data.gouv.fr/datasets/horaires-des-lignes-ter-sncf#dataset-discussions)

vingerha commented 1 day ago

The use of a timstamp in the RT trips was already a big surprise. I mean, how non-unique is this OCESN864765F3273412 ??? Why would you need a trip id related to a date/time in static data as well. But then truncating it even further ....sigh There is way that I can relatively build a simple, global applicable solution here. BTW... the alerts that I am using is based on route so this would not work by default. I have only analysed 2 sources for alerts and neither of them had trip_id as identifier, so this one is a surprise too (gtfs is FULL of surprise)

THANKS for sharing this, good to have more knowledge than before :)