[Bug]: config.yaml set up and GUI setup not working for EKM Mittelsachsen (ics)

riegera2412 commented 1 month ago

I Have A Problem With:

A specific source, The integration in general

What's Your Problem

After updating wcs to 2.30 my manual configuration in the config.yaml doesn't work. I tried downgrading to 2.2.0 and 2.1.0 but neither does the trick. I used a downloaded ics from my waste collection company that worked flawlessly for years. The config is shown in the config section.

I get three log messages:

ics source not available. I tried different paths to the ics file, but neither of them worked. Before it was in config/www/file.ics

Logger: custom_components.waste_collection_schedule.waste_collection_schedule.source_shell
Quelle: custom_components/waste_collection_schedule/waste_collection_schedule/source_shell.py:221
Integration: Waste Collection Schedule ([Dokumentation](https://github.com/mampfes/hacs_waste_collection_schedule#readme))
Erstmals aufgetreten: 12:25:50 (1 Vorkommnisse)
Zuletzt protokolliert: 12:25:50

source not found: ics

setup of wcs is taking more than 10s warning

Logger: homeassistant.setup
Quelle: runner.py:189
Erstmals aufgetreten: 12:25:59 (1 Vorkommnisse)
Zuletzt protokolliert: 12:25:59

Setup of waste_collection_schedule is taking over 10 seconds.

Error while setting up waste_collection_schedule platform for sensor

Logger: homeassistant.components.sensor
Quelle: helpers/entity_platform.py:361
Integration: Sensor (Dokumentation, Probleme)
Erstmals aufgetreten: 12:26:17 (5 Vorkommnisse)
Zuletzt protokolliert: 12:26:17

Error while setting up waste_collection_schedule platform for sensor
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 361, in _async_setup_platform
    await asyncio.shield(awaitable)
  File "/config/custom_components/waste_collection_schedule/sensor.py", line 147, in async_setup_platform
    raise ValueError(
ValueError: source_index 0 out of range (0--1) please check your sensor configuration

Next, I commented out the configuration and tried the setup via the GUI, but when choosing Germany and then my source (EKM Mittelsachsen (ics)), I get an unknown error message.

In the logs the following warning appears:

Logger: homeassistant.util.loop
Quelle: util/loop.py:136
Erstmals aufgetreten: 12:29:13 (1 Vorkommnisse)
Zuletzt protokolliert: 12:29:13

Detected blocking call to open with args (PosixPath('/config/custom_components/waste_collection_schedule/sources.json'),) inside the event loop by custom integration 'waste_collection_schedule' at custom_components/waste_collection_schedule/config_flow.py, line 331: with p.open(encoding="utf-8") as json_file: (offender: /config/custom_components/waste_collection_schedule/config_flow.py, line 331: with p.open(encoding="utf-8") as json_file:), please report it to the author of the 'waste_collection_schedule' custom integration For developers, please see https://developers.home-assistant.io/docs/asyncio_blocking_operations/#open Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/usr/src/homeassistant/homeassistant/__main__.py", line 223, in <module> sys.exit(main()) File "/usr/src/homeassistant/homeassistant/__main__.py", line 209, in main exit_code = runner.run(runtime_conf) File "/usr/src/homeassistant/homeassistant/runner.py", line 189, in run return loop.run_until_complete(setup_and_run_hass(runtime_config)) File "/usr/local/lib/python3.12/asyncio/base_events.py", line 674, in run_until_complete self.run_forever() File "/usr/local/lib/python3.12/asyncio/base_events.py", line 641, in run_forever self._run_once() File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1990, in _run_once handle._run() File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 556, in start task = asyncio.Task(coro, loop=loop, eager_start=True) File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 477, in _handle_request resp = await request_handler(request) File "/usr/local/lib/python3.12/site-packages/aiohttp/web_app.py", line 559, in _handle return await handler(request) File "/usr/local/lib/python3.12/site-packages/aiohttp/web_middlewares.py", line 117, in impl return await handler(request) File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 92, in security_filter_middleware return await handler(request) File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 210, in forwarded_middleware return await handler(request) File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 26, in request_context_middleware return await handler(request) File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 85, in ban_middleware return await handler(request) File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 242, in auth_middleware return await handler(request) File "/usr/src/homeassistant/homeassistant/components/http/headers.py", line 32, in headers_middleware response = await handler(request) File "/usr/src/homeassistant/homeassistant/helpers/http.py", line 73, in handle result = await handler(request, **request.match_info) File "/usr/src/homeassistant/homeassistant/components/http/decorators.py", line 81, in with_admin return await func(self, request, *args, **kwargs) File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 74, in wrapper return await method(view, request, data, *args, **kwargs) File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 172, in post return await self._post_impl(request, data) File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 179, in _post_impl return await super()._post_impl(request, data) File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 84, in _post_impl result = await self._flow_mgr.async_init( File "/usr/src/homeassistant/homeassistant/config_entries.py", line 1287, in async_init flow, result = await self._async_init(flow_id, handler, context, data) File "/usr/src/homeassistant/homeassistant/config_entries.py", line 1312, in _async_init flow = await self.async_create_flow(handler, context=context, data=data) File "/usr/src/homeassistant/homeassistant/config_entries.py", line 1505, in async_create_flow flow = handler() File "/config/custom_components/waste_collection_schedule/config_flow.py", line 306, in __init__ self._sources = self._get_source_list() File "/config/custom_components/waste_collection_schedule/config_flow.py", line 331, in _get_source_list with p.open(encoding="utf-8") as json_file:

There is also an HA error message at the moment of setting up wcs.

Logger: aiohttp.server
Quelle: /usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py:448
Erstmals aufgetreten: 12:51:24 (1 Vorkommnisse)
Zuletzt protokolliert: 12:51:24

Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 477, in _handle_request
    resp = await request_handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_app.py", line 559, in _handle
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 92, in security_filter_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 210, in forwarded_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 26, in request_context_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 85, in ban_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 242, in auth_middleware
    return await handler(request)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/headers.py", line 32, in headers_middleware
    response = await handler(request)
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/http.py", line 73, in handle
    result = await handler(request, **request.match_info)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/decorators.py", line 81, in with_admin
    return await func(self, request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 222, in post
    return await super().post(request, flow_id)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 74, in wrapper
    return await method(view, request, data, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 122, in post
    result = await self._flow_mgr.async_configure(flow_id, data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 370, in async_configure
    result = await self._async_configure(flow_id, user_input)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 417, in _async_configure
    result = await self._async_handle_step(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 520, in _async_handle_step
    result: _FlowResultT = await getattr(flow, method)(user_input)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/waste_collection_schedule/config_flow.py", line 366, in async_step_source
    return await self.async_step_args()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/waste_collection_schedule/config_flow.py", line 548, in async_step_args
    schema, module = await self.__get_arg_schema(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/waste_collection_schedule/config_flow.py", line 389, in __get_arg_schema
    module = await self.hass.async_add_executor_job(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/loop.py", line 200, in protected_loop_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/source/ics.py", line 9, in <module>
    from waste_collection_schedule.service.ICS import ICS
  File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/service/ICS.py", line 7, in <module>
    from icalevents import icalevents
  File "/usr/local/lib/python3.12/site-packages/icalevents/icalevents.py", line 3, in <module>
    from .icalparser import parse_events, Event
  File "/usr/local/lib/python3.12/site-packages/icalevents/icalparser.py", line 18, in <module>
    from icalendar.windows_to_olson import WINDOWS_TO_OLSON
ModuleNotFoundError: No module named 'icalendar.windows_to_olson'

Any help and/or advice is greatly appreciated.

A.

Source (if relevant)

No response

Logs

see above

Relevant Configuration

configuration.yaml:
...
waste_collection_schedule: !include waste.yaml
...

waste.yaml:

sources:
  - name: ics
    args:
      url: https://www.ekm-mittelsachsen.de/service-dienstleistungen/entsorgungstermine-abfallkalender?tx_ekmabfallkalender_abfallkalender%5Baction%5D=getIcal&tx_ekmabfallkalender_abfallkalender%5Bcity_id%5D=11400&tx_ekmabfallkalender_abfallkalender%5Bstreet_id%5D=52187&tx_ekmabfallkalender_abfallkalender%5Byear%5D={%Y}}&cHash=83925ca887a5af7bdb638a5df97c696e&tx_ekmabfallkalender_abfallkalender[types][]=r&tx_ekmabfallkalender_abfallkalender[types][]=p&tx_ekmabfallkalender_abfallkalender[types][]=l&tx_ekmabfallkalender_abfallkalender[types][]=b&tx_ekmabfallkalender_abfallkalender[trigger_days]=0
      #file: "homeassistant/www/abfallkalenderFreiberg2022_2023_2024.ics"
    customize:
      - type: Bio
        alias: biomuell
        icon: mdi:flower-outline
      - type: Leichtstoffe
        alias: leichtstoffe
        icon: mdi:recycle
      - type: Papier
        alias: papier
        icon: mdi:trash-can-outline
      - type: Restmuell
        alias: restmuell
        icon: mdi:trash-can          
fetch_time: "04:00"
day_switch_time: "10:00"

Checklist Source Error

[X] Use the example parameters for your source (often available in the documentation) (don't forget to restart Home Assistant after changing the configuration)
[X] Checked that the website of your service provider is still working
[X] Tested my attributes on the service provider website (if possible)
[X] I have tested with the latest version of the integration (master) (for HACS in the 3 dot menu of the integration click on "Redownload" and choose master as version)

Checklist Sensor Error

[ ] Checked in the Home Assistant Calendar tab if the event names match the types names (if types argument is used)

Required

[X] I have searched past (closed AND opened) issues to see if this bug has already been reported, and it hasn't been.
[X] I understand that people give their precious time for free, and thus I've done my very best to make this problem as easy as possible to investigate.

5ila5 commented 1 month ago

ModuleNotFoundError: No module named 'icalendar.windows_to_olson'

Sounds like icalendar >= 6.0.0 is installed, but icalevents (Libraary we use to parse ICS files) currently only supports icalender < 6.0.0 (see https://github.com/jazzband/icalevents/pull/145).

I'm not quite sure how this happens, as icalevents dependency should be respected when HA installs components dependencies

ReneNulschDE commented 1 month ago

Just guessing: the manifest.json has no version identifier for icalendar. Could be that the user has a fresh install and the highest version was installed... Looks like the ics_v1 service is using icalendar.

5ila5 commented 1 month ago

Not exactly sure how HA resolves dependencies but as icalevents requires icalendar==5.0.13 so HA (I think the se uv now) should not install icalendar 6....

I doubt ics_v1 is the problem here, as it does not use the changed functions/modules of icalendar 6 (the stack trace also shows File "/config/custom_components/waste_collection_schedule/waste_collection_schedule/service/ICS.py", line 7, in <module> from icalevents import icalevents )

It seems to be issue with multiple instances as there a few source not found issues reported here.

The error message is pretty bad as every import error inside a source will trigger this error message. I fixed this in #2908 (will test and merge it later)

ReneNulschDE commented 1 month ago

... When I install in a clean venv environment with pip: pip install icalendar icalevents!=0.1.28 icalendar gets installed with version icalendar-5.0.13 icalevents-0.1.29

when I use uv with: uv pip install icalendar icalevents!=0.1.28 the result is:

+ icalendar==6.0.1
+ icalevents==0.1.25

see that icalevents get installed with 0.1.25

I'll check it in a fresh HA environement now.

Update: No this is not the reason:

after a fresh install ha install and adding WCS via UI the result is:

icalendar                          5.0.13
icalevents                         0.1.29

ReneNulschDE commented 1 month ago

@riegera2412 : Could you share the list of other custom_components that you have installed? And please share some context about your environment. This would help us to create a solution description as this is not a code more an environment problem.

ReneNulschDE commented 1 month ago

@5ila5 : Besides my comments above...

It could happen that icalendar >5 gets installed when you try to install/use a custom_component with a hard dependency to icalendar>=6.0.0.

Flow:

Install HA
copy WCS and for example the cutom component "ics_calendar" into the custom_components folder
restart and check that no icalendar file is installed (pip list)
add integration "wcs" in the ha frontend --> icalendar version 5.0.13 is installed
add integration "ics_calendar" in the ha frontend --> icalendar version 6.0.1is installed and WCS is failing

ReneNulschDE commented 1 month ago

Potential solution

riegera2412 commented 1 month ago

@riegera2412 : Could you share the list of other custom_components that you have installed? And please share some context about your environment. This would help us to create a solution description as this is not a code more an environment problem.

I am running HassOS 2024.10.3 on a HP T620 Thin Client.

Also using the RaspberryMatic Addon to integrate Hometic IP hardware. For Ikea hardware I use a igbee 3 USB Dongle.

Not sure what other information you might need. If you require any specific information please ask.

EDIT: I have ics_calendar installed. Just checked the manifest.json and it specifies the following requirements:

"requirements": ["icalendar>=6.0.0","python-dateutil>=2.9.0.post0","pytz>=2024.1","recurring_ical_events>=3.3.2","ics>=0.7.2","arrow"],

Guess that could be the problem?

ReneNulschDE commented 1 month ago

thx, please check the solution that I added above.

riegera2412 commented 1 month ago

Potential solution

Worked like a charm! Thanks a lot!

5ila5 commented 1 month ago

icalevents 0.1.25 did not define a specific version of icalendar as dependency, this is why it installs it like this

We could set icalevents>=0.1.26,!=0.1.28 as dependency this would also force icalendar<6 as long as https://github.com/jazzband/icalevents/pull/145 is not resolved and a new version of icalevents is released.

But this approach would lead to dependency conflicts with integrations like ics_calendar.

But this is probably still the way to go, so users need to decide and check what they want.

I opened a PR for this (#2911)

@ReneNulschDE what do you think?

ReneNulschDE commented 1 month ago

This will not have an impact based on my tests. Whenever the user are upgrading to ics_calendar 5.0.4, the icalendar >=6 will be installed. I checked all HACS-published integrations and only ics_calendar has this dependency.

My opinion: Let it as it is, say do not upgrade ics_calendar higher then 5.0.3, in case you have it done, use the process that I published - and wait for a new version of icalevents. And, you should pin the icalendar version in the manifest.json.

Maybe you could ask the maintainer of ics_calendar to revert to 5.0.3 or go back to the old icalendar lib? your component has 10x more users than ics_calendar. (could be offending... :-()

btw the command that I used to download the hacs-repos (found on discord some month before, can't remember who gets the credit) : for repo in $(curl -sSL https://data-v2.hacs.xyz/integration/repositories.json | jq -r '.[]'); do if [ ! -d "${repo}" ]; then echo mkdir "${repo%%/*}" && git clone "git@github.com:${repo}" "${repo}"; else git -C "${repo}" pull origin; fi; done

5ila5 commented 4 weeks ago

Pinning icalevents>=0.1.26 should have the same effect as pinning icalendar, as icalevents>=0.1.26 defines a fixed version of icalevents as dependency. But if icalevents publishes a new version with support for icalendar=6 the dependency conflict should vanish without the need for a release of WCS

mampfes / hacs_waste_collection_schedule