[BUG] Detected blocking call to listdir

[noah2121]

Installed plugin using HACS and added transit agency with zip file link. Plugin took unreasonably long to load data from transit agency. Viewed logs, which contained instructions to make a bug report. Please let me know if I can offer any more information to help reproduce the bug.

Transit agency is MBTA - data here

Release used

• gtfs2 0.4.9
• HAOS VM, core 2024.7.1, supervisor 2024.06.2, HAOS 12.4, frontend 20240705.0

Logs

HA Core Logs:

2024-07-07 20:41:38.071 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to listdir with args ('/config/gtfs2',) inside the event loop by custom integration 'gtfs2' at custom_components/gtfs2/gtfs_helper.py, line 596: files = os.listdir(gtfs_dir) (offender: /config/custom_components/gtfs2/gtfs_helper.py, line 596: files = os.listdir(gtfs_dir)), please create a bug report at https://github.com/vingerha/gtfs2/issues
For developers, please see https://developers.home-assistant.io/docs/asyncio_blocking_operations/#listdir
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 223, in <module>
    sys.exit(main())
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 209, in main
    exit_code = runner.run(runtime_conf)
  File "/usr/src/homeassistant/homeassistant/runner.py", line 190, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1990, in _run_once
    handle._run()
  File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_app.py", line 543, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_middlewares.py", line 114, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 92, in security_filter_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 83, in forwarded_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 26, in request_context_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 85, in ban_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 242, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/headers.py", line 32, in headers_middleware
    response = await handler(request)
  File "/usr/src/homeassistant/homeassistant/helpers/http.py", line 73, in handle
    result = await handler(request, **request.match_info)
  File "/usr/src/homeassistant/homeassistant/components/http/decorators.py", line 81, in with_admin
    return await func(self, request, *args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 222, in post
    return await super().post(request, flow_id)
  File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 74, in wrapper
    return await method(view, request, data, *args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 122, in post
    result = await self._flow_mgr.async_configure(flow_id, data)
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 368, in async_configure
    result = await self._async_configure(flow_id, user_input)
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 411, in _async_configure
    result = await self._async_handle_step(
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 518, in _async_handle_step
    result: _FlowResultT = await getattr(flow, method)(user_input)
  File "/config/custom_components/gtfs2/config_flow.py", line 116, in async_step_local_stops
    datasources = get_datasources(self.hass, DEFAULT_PATH)
  File "/config/custom_components/gtfs2/gtfs_helper.py", line 596, in get_datasources
    files = os.listdir(gtfs_dir)

2024-07-07 20:45:11.855 WARNING (SyncWorker_27) [custom_components.gtfs2.gtfs_helper] Cannot use this datasource as still unpacking: MBTA
2024-07-07 20:55:07.820 WARNING (SyncWorker_34) [custom_components.gtfs2.gtfs_helper] Cannot use this datasource as still unpacking: MBTA
2024-07-07 21:03:41.498 WARNING (SyncWorker_38) [custom_components.gtfs2.gtfs_helper] Cannot use this datasource as still unpacking: MBTA
2024-07-07 21:15:44.332 WARNING (SyncWorker_18) [custom_components.gtfs2.gtfs_helper] Cannot use this datasource as still unpacking: MBTA
2024-07-07 21:32:41.001 WARNING (SyncWorker_29) [custom_components.gtfs2.gtfs_helper] Cannot use this datasource as still unpacking: MBTA

[vingerha]

Thanks for the feedback, had not seen this before ...but that is also because HA issues this warning since recently. Will fix it over time The real problem is that your datasource is not in line with the conditions as per the pygtfs library that gtfs2 is using. You can see that when setting up a new datasource and looking at the log. You have to find another source as I cannot change the library's behavior. try gtfs.pro

---

I think (not 100% sure) it crashes on stop_lat and stop_lon ...these should not have 'None' i.e. not be empty

.Failure while writing Stops(stop_id='node-123-platform', stop_code=None, stop_name='Andrew', stop_desc='Andrew - Bottom of left outbound exit-only escalator', platform_code=None, stop_lat=None, stop_lon=None, zone_id=None, stop_url=None, location_type='3', parent_station='place-andrw', wheelchair_boarding='1')
  File "/usr/local/lib/python3.12/site-packages/pygtfs/loader.py", line 87, in append_feed
    instance = gtfs_class(feed_id=feed_id, **record._asdict())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[vingerha]

Hi, I looked at it but weirdly enough, when I apply the suggested settings it does (!) throw that error with me. I also noted that there were some inconsistencies on when the error was raised and they fixed things. Can you please verify if you still have these errors as I have not (yet) seen these with me locally

[vingerha]

Forget my request...just got someone else that reports this issue is there... not sure why I donot see it, odd