search

vingerha / gtfs2

Support GTFS in Home Assistant GUI-only
https://github.com/vingerha/gtfs2
MIT License
81 stars 7 forks source link

Issue adding Data Base #68

Closed liammaclean closed 6 months ago

liammaclean commented 6 months ago

Describe the bug I've tried to add the MBTA Data base about 10 times over the last week and still no luck, never finishes extracting. (https://cdn.mbta.com/MBTA_GTFS.zip)

Release used This is on Home Assistant on my Raspberry Pi 4

vingerha commented 6 months ago

Can you please add:

liammaclean commented 6 months ago

Error occurred loading flow for integration gtfs2: No module named 'google.transit'

vingerha commented 6 months ago

Please provide a more complete set of logs, from the moment you submitted the request of a new datasource till and including the error. Start afresh, i.e. remove the MBTA files beforehand

liammaclean commented 6 months ago

I've attached both a screen shot of the file and the logs that I downloaded.Screenshot_20240509-011633.png

error_log.txt

vingerha commented 6 months ago

There are no errors on gtfs except a statement: Cannot use this datasource as still unpacking: MBTA, this is because the journal file is stuck, I miss lots of log entries related to the unpacking itself ...please put the gtfs2 to debug and add a new datasource with a slightly different name....then send me the logs

liammaclean commented 6 months ago

how do I put it to debug?

vingerha commented 6 months ago

If visible already (probably not as it is your first install) image

Alternatively

configuration.yaml and restart HA 


logger:
  default: warning
  logs:
    custom_components.gtfs2: debug
liammaclean commented 6 months ago

Got it, ty. Here is what i have so far: 2024-05-09 01:56:36.275 DEBUG (SyncWorker_6) [custom_components.gtfs2.gtfs_helper] Getting gtfs with data: {'extract_from': 'url', 'file': 'MBTA2', 'url': 'https://cdn.mbta.com/MBTA_GTFS.zip'} 2024-05-09 01:56:36.277 DEBUG (SyncWorker_6) [custom_components.gtfs2.gtfs_helper] Checking if extracting: MBTA2 2024-05-09 01:56:43.722 INFO (SyncWorker_6) [custom_components.gtfs2.gtfs_helper] Exiting main after start subprocess for unpacking: MBTA2.zip 2024-05-09 01:56:43.731 DEBUG (MainThread) [custom_components.gtfs2.config_flow] Checkdata pygtfs: extracting with data: {'extract_from': 'url', 'file': 'MBTA2', 'url': 'https://cdn.mbta.com/MBTA_GTFS.zip'}

Will update if anything more is logged

vingerha commented 6 months ago

it should already show black-colored text with the entities that pygtfs is trying to extract

liammaclean commented 6 months ago

Nothing of that sort, this is the full of what I have: 2024-05-09 01:55:39.938 WARNING (SyncWorker_3) [homeassistant.loader] We found a custom integration hacs which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant 2024-05-09 01:55:39.939 WARNING (SyncWorker_3) [homeassistant.loader] We found a custom integration gtfs2 which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant 2024-05-09 01:55:47.854 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'hacs' accesses hass.components.frontend. This is deprecated and will stop working in Home Assistant 2024.9, it should be updated to import functions used from frontend directly at custom_components/hacs/frontend.py, line 68: hass.components.frontend.async_register_built_in_panel(, please create a bug report at https://github.com/hacs/integration/issues 2024-05-09 01:55:50.865 ERROR (MainThread) [homeassistant.components.mqtt.client] Failed to connect to MQTT server due to exception: [Errno -5] Name has no usable address 2024-05-09 01:56:04.431 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'hacs' accesses hass.helpers.event. This is deprecated and will stop working in Home Assistant 2024.11, it should be updated to import functions used from event directly at custom_components/hacs/base.py, line 642: self.hass.helpers.event.async_track_time_interval(, please create a bug report at https://github.com/hacs/integration/issues 2024-05-09 01:56:36.275 DEBUG (SyncWorker_6) [custom_components.gtfs2.gtfs_helper] Getting gtfs with data: {'extract_from': 'url', 'file': 'MBTA2', 'url': 'https://cdn.mbta.com/MBTA_GTFS.zip'} 2024-05-09 01:56:36.277 DEBUG (SyncWorker_6) [custom_components.gtfs2.gtfs_helper] Checking if extracting: MBTA2 2024-05-09 01:56:43.722 INFO (SyncWorker_6) [custom_components.gtfs2.gtfs_helper] Exiting main after start subprocess for unpacking: MBTA2.zip 2024-05-09 01:56:43.731 DEBUG (MainThread) [custom_components.gtfs2.config_flow] Checkdata pygtfs: extracting with data: {'extract_from': 'url', 'file': 'MBTA2', 'url': 'https://cdn.mbta.com/MBTA_GTFS.zip'}

vingerha commented 6 months ago

meaning you have nothing more than this? does it stop reporting at 01:56?

liammaclean commented 6 months ago

Yup, nothing beyond 1:56

vingerha commented 6 months ago

????? ...ok , I will try myself now...

vingerha commented 6 months ago

Hmm...I now see what you mean..the HA log is not showing all log entries.. I always use the container log and never look at HA..the container log shows this


2024-05-09 08:04:48.164 DEBUG (SyncWorker_2) [custom_components.gtfs2.gtfs_helper] Getting gtfs with data: {'extract_from': 'url', 'file': 'mbta2', 'url': 'https://cdn.mbta.com/MBTA_GTFS.zip'}
2024-05-09 08:04:48.168 DEBUG (SyncWorker_2) [custom_components.gtfs2.gtfs_helper] Checking if extracting: mbta2
2024-05-09 08:04:56.297 INFO (SyncWorker_2) [custom_components.gtfs2.gtfs_helper] Exiting main after start subprocess for unpacking: mbta2.zip
2024-05-09 08:04:56.310 DEBUG (MainThread) [custom_components.gtfs2.config_flow] Checkdata pygtfs: extracting with data: {'extract_from': 'url', 'file': 'mbta2', 'url': 'https://cdn.mbta.com/MBTA_GTFS.zip'}
Loading GTFS data for <class 'pygtfs.gtfs_entities.Agency'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Stop'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Transfer'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Route'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Fare'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.FareRule'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.ShapePoint'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Service'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.ServiceException'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Trip'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Frequency'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.StopTime'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.FeedInfo'>:
Loading GTFS data for <class 'pygtfs.gtfs_entities.Translation'>:
2 records read for <class 'pygtfs.gtfs_entities.Agency'>.
Process Process-1:
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/config/custom_components/gtfs2/gtfs_helper.py", line 429, in extract_from_zip
    pygtfs.append_feed(gtfs, os.path.join(gtfs_dir, file))
  File "/usr/local/lib/python3.12/site-packages/pygtfs/loader.py", line 87, in append_feed
    instance = gtfs_class(feed_id=feed_id, **record._asdict())
.Failure while writing Stops(stop_id='node-123-platform', stop_code=None, stop_name='Andrew', stop_desc='Andrew - Bottom of left outbound exit-only escalator', platform_code=None, stop_lat=None, stop_lon=None, zone_id=None, stop_url=None, location_type='3', parent_station='place-andrw', wheelchair_boarding='1')
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 4, in __init__
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/orm/state.py", line 564, in _initialize_instance
    with util.safe_reraise():
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/orm/state.py", line 562, in _initialize_instance
    manager.original_init(*mixed[1:], **kwargs)
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/orm/decl_base.py", line 2142, in _declarative_constructor
    setattr(self, k, kwargs[k])
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/orm/attributes.py", line 537, in __set__
    self.impl.set(
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/orm/attributes.py", line 1277, in set
    value = self.fire_replace_event(
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/orm/attributes.py", line 1292, in fire_replace_event
    value = fn(
            ^^^
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/orm/events.py", line 2573, in wrap
    return fn(target, *arg)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/sqlalchemy/orm/util.py", line 326, in set_
    return validator(state.obj(), key, value)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pygtfs/gtfs_entities.py", line 73, in in_range
    float_value = float(value)
                  ^^^^^^^^^^^^
TypeError: float() argument must be a string or a real number, not 'NoneType'
liammaclean commented 6 months ago

Where would I find container logs

vingerha commented 6 months ago

And this is the line that is frustrating, stop-code cannot be None So as (sadly) often...the source data is incorrect.... check for possible other sources ...e.g. on gtfs.pro

.Failure while writing Stops(stop_id='node-123-platform', stop_code=None, stop_name='Andrew', stop_desc='Andrew - Bottom of left outbound exit-only escalator', platform_code=None, stop_lat=None, stop_lon=None, zone_id=None, stop_url=None, location_type='3', parent_station='place-andrw', wheelchair_boarding='1')

vingerha commented 6 months ago

Where would I find container logs

In the main container HA is running, but if yo uare on HAOS...that will not work easily....I am on docker/container version and using portainer

liammaclean commented 6 months ago

understood. Will you let me know if https://gtfs.pro/en/usa/Massachusetts-Bay-Transportation-Authority/boston-mbta works better for you? I'm trying it now, but if i can't see the logs i have no idea how I'll know if it is working or just stalling like before

vingerha commented 6 months ago

Am working on improving the logging....you can verify if the .sqlite grows...should become Mb

liammaclean commented 6 months ago

Yup, looks like it's growing now. Hope it works thanks so much for the help