search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
72.62k stars 30.38k forks source link

Calendar missing on CalDav integration #71048

Closed Aohzan closed 2 years ago

Aohzan commented 2 years ago

The problem

Since 2022.5 beta, I only have 2 of my 5 caldav calendars (Travail XX and Vacances from my yaml config)

What version of Home Assistant Core has the issue?

2022.5.0b2

What was the last working version of Home Assistant Core?

2022.4.0

What type of installation are you running?

Home Assistant Supervised

Integration causing the issue

caldav

Link to integration documentation on our website

https://www.home-assistant.io/integrations/caldav/

Diagnostics information

No response

Example YAML snippet

calendar:
  - platform: caldav
    username: user
    password: !secret caldav
    url: https://mysogocaldavserver
    custom_calendars:
      - name: XX
        calendar: Agenda personnel
        search: ".*"
      - name: YY
        calendar: YY
        search: ".*"
      - name: Travail XX
        calendar: Agenda personnel
        search: "^(Bureau|Travail)$"
      - name: Vacances
        calendar: Agenda personnel
        search: "^(Vacances|Absent)$"
      - name: Travail YY
        calendar: YY
        search: "^(Travail)$"

Anything in the logs that might be useful for us?

2022-04-29 10:26:06 ERROR (MainThread) [homeassistant.components.calendar] caldav: Error on device update!
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 431, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 739, in async_device_update
    await task
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/caldav/calendar.py", line 146, in update
    event = copy.deepcopy(self.data.event)
  File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.9/copy.py", line 264, in _reconstruct
    y = func(*args)
  File "/usr/local/lib/python3.9/copy.py", line 263, in <genexpr>
    args = (deepcopy(arg, memo) for arg in args)
  File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.9/copy.py", line 205, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.9/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle '_thread.lock' object

Additional information

No response

probot-home-assistant[bot] commented 2 years ago

caldav documentation caldav source (message by IssueLinks)

tobixen commented 2 years ago

Is it easily reproducible, and does the same error appear in the log all the time?

Aohzan commented 2 years ago

error appears at each HA start

PhilRW commented 2 years ago

I see the same error, although mine starts with Update for calendar.home fails. Same TypeError about cannot pickle '_thread.lock' object.

Aohzan commented 2 years ago

I found this error, 3 seconds before the log on the first post:

2022-05-05 13:35:49 ERROR (SyncWorker_8) [caldav] Deviation from expectations found.  Please raise an issue at https://github.com/python-caldav/caldav/issues or reach out to t-caldav@tobixen.no, include this error and the traceback and te
ll what server you are using
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/caldav/lib/error.py", line 25, in assert_
    assert(condition)
AssertionError
tobixen commented 2 years ago

That one is "mine" (from the python caldav library), I'll have a look

tobixen commented 2 years ago

Uff, that traceback is unfortunately quite useless. I need to read up on how to log a proper traceback ...

There is an environmental variable PYTHON_CALDAV_DEBUGMODE that can be adjusted, if setting it to DEBUG there may be that there will be more information in the logs, but I don't know how to set environmental variables prior to starting HomeAssistant (it depends on how you're running it).

I have no idea about _thread.lock, that's probably not coming from "my" library.

Aohzan commented 2 years ago

I edtited directly the lib/error.py file:

debugmode = "DEBUG"

and now I don't have any calendar :grin: 

2022-05-05 18:13:45 DEBUG (SyncWorker_7) [caldav] url: https://path/personal/
2022-05-05 18:13:45 DEBUG (SyncWorker_7) [caldav] self.url: https://path/mail/Calendar/personal/
2022-05-05 18:13:46 ERROR (MainThread) [homeassistant.components.calendar] Error while setting up caldav platform for calendar
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 249, in _async_setup_platform
    await asyncio.shield(task)
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/caldav/calendar.py", line 87, in setup_platform
    calendars = client.principal().calendars()
  File "/usr/local/lib/python3.9/site-packages/caldav/davclient.py", line 350, in principal
    self._principal = Principal(client=self, *largs, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/caldav/objects.py", line 399, in __init__
    cup = self.get_property(dav.CurrentUserPrincipal())
  File "/usr/local/lib/python3.9/site-packages/caldav/objects.py", line 174, in get_property
    foo = self.get_properties([prop], **passthrough)
  File "/usr/local/lib/python3.9/site-packages/caldav/objects.py", line 197, in get_properties
    response = self._query_properties(props, depth)
  File "/usr/local/lib/python3.9/site-packages/caldav/objects.py", line 141, in _query_properties
    return self._query(root, depth)
  File "/usr/local/lib/python3.9/site-packages/caldav/objects.py", line 159, in _query
    ret = getattr(self.client, query_method)(
  File "/usr/local/lib/python3.9/site-packages/caldav/davclient.py", line 398, in propfind
    return self.request(url or self.url, "PROPFIND", props,
  File "/usr/local/lib/python3.9/site-packages/caldav/davclient.py", line 577, in request
    self.verify_login()
  File "/usr/local/lib/python3.9/site-packages/caldav/davclient.py", line 543, in verify_login
    error.assert_(auth_type in ('Basic', 'Digest'))
  File "/usr/local/lib/python3.9/site-packages/caldav/lib/error.py", line 25, in assert_
    assert(condition)
AssertionError
tobixen commented 2 years ago

Aha. If you can manage to upgrade to caldav 0.9, this problem is fixed.

tobixen commented 2 years ago

The error message above is the same as #65804 and several others. Regression in the 0.8.2-version of the python caldav library.

tobixen commented 2 years ago

(Protocol error on the server side, it uses 'basic' or 'digest' rather than 'Basic' or 'Digest')

Aohzan commented 2 years ago

I upgraded to caldav 0.9, I don't have the Assertion error anymore, but still the original:

2022-05-06 11:48:06 ERROR (MainThread) [homeassistant.components.calendar] caldav: Error on device update!
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 431, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 736, in async_device_update
    await task
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/caldav/calendar.py", line 146, in update
    event = copy.deepcopy(self.data.event)
  File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.9/copy.py", line 264, in _reconstruct
    y = func(*args)
  File "/usr/local/lib/python3.9/copy.py", line 263, in <genexpr>
    args = (deepcopy(arg, memo) for arg in args)
  File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.9/copy.py", line 205, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.9/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.9/copy.py", line 270, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.9/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.9/copy.py", line 230, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.9/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle '_thread.lock' object
Aohzan commented 2 years ago

on the line event = copy.deepcopy(self.data.event) where the error occurs, the content of self.data.event: image

when the event is None it doesn't raise an error

tobixen commented 2 years ago

This is probably outside what I can help with currently, haven't had time to look into the HomeAssistant caldav integration yet. But from the screen shot, I notice that the tzinfo class name looks a bit weird. If you're able to debug it interactively, try to replace it with zoneinfo.ZoneInfo(key='Europe/Paris') and see if that solves the problem?

tobixen commented 2 years ago

(the zoneinfo module is new in python 3.9 ... so there is a lot of mess with timezones in python at the moment, as pytz is considered "legacy")

Aohzan commented 2 years ago

I try to set to the value but if put tzicalvtz back image

szimszon commented 2 years ago

I upgraded to caldav 0.9, I don't have the Assertion error anymore, but still the original:

Same here :(

szimszon commented 2 years ago

With 2022.5.5 I have still the same issue... :(

tobixen commented 2 years ago

I'm sorry if it seems like I've taken any kind of responsibility of this ticket.

There seems to be two different problems here, one is the error message Deviation from expectations found reported by @Aohzan which is fixed in the caldav python library version 0.9.0, and which will be fixed in the next Home Assistant release.

Then there is another error TypeError: cannot pickle '_thread.lock' object, which does not seem to be related to "my" python caldav library, but probably a bug in the Home Assistant caldav integration. I have a suspicion that the unpickable object is a timezone object. Someone with knowledge of the Home Assistant caldav code should probably look more into this.

szimszon commented 2 years ago

For clarity my problem is this: TypeError: cannot pickle '_thread.lock' object

eliahru commented 2 years ago

Same here :(

szimszon commented 2 years ago

Somebody?

eliahru commented 2 years ago

Since it looks like nobody of developers of caldav code of HASS is looking into this issue yet, I found a workaround to have voice notifications for events from my caldav calendars via Node-RED plug-in to HASS. It required some minor programming for proper parsing of early notifications like "!!-5" specified in event subjects, but in general it looks to be even more flexible solution.

szimszon commented 2 years ago

The problem that it is not an option for me ;(

szimszon commented 2 years ago

Problem is still present in 2022.6.0 :(

szimszon commented 2 years ago

I don't get it. The code referred in traceback are old:

File "/srv/homeassistant-blue/lib/python3.9/site-packages/homeassistant/helpers/entity_platform.py", line 431, in _async_add_entity await entity.async_device_update(warning=False)

4 years ago

File "/srv/homeassistant-blue/lib/python3.9/site-packages/homeassistant/helpers/entity.py", line 736, in async_device_update await task

2 years ago

...

File "/srv/homeassistant-blue/lib/python3.9/site-packages/homeassistant/components/caldav/calendar.py", line 146, in update

3 years ago

Also I didn't change python.

szimszon commented 2 years ago

This is the last commit: https://github.com/home-assistant/core/commit/f99b6004ea2dd0725483db7246672d468fe32088 as far as I can tell. It's huge: Showing 9 changed files with 342 additions and 182 deletions.

szimszon commented 2 years ago

Workaround: copy back the old calendar.py from here to homeassistant/components/caldav/ and restart homeassistant. Seems to work.

szimszon commented 2 years ago

Maybe if I may ask @allenporter or @MartinHjelmare please could you check the last commit? Thank you!

allenporter commented 2 years ago

I'll have a look.

szimszon commented 2 years ago

Confirmed this version fixed the issue for me.

allenporter commented 2 years ago

Thanks folks. This is tagged for the next patch release 2022.6.1