search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
73.52k stars 30.71k forks source link

Caldav: no events appear for calendars #81624

Closed rewquey closed 1 year ago

rewquey commented 2 years ago

The problem

In 2022.11.1, my calendars appear empty in the home assistant integration due to an Exception occurring when reading/ updating calendar entries. After a downgrade to 2022.10.2, calendar entries show up as expected.

What version of Home Assistant Core has the issue?

2022.11.1

What was the last working version of Home Assistant Core?

2022.10.2

What type of installation are you running?

Home Assistant OS

Integration causing the issue

caldav

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

calendar:
  - platform: caldav
    username: !secret caldav_user
    password: !secret caldav_pwd
    url: !secret caldav_url
    custom_calendars:
      - name: "Home Office"
        calendar: "home automation"
        search: "work"
      - name: "Travel"
        calendar: "home automation"
        search: "away"

Anything in the logs that might be useful for us?

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 60, in security_filter_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 100, in forwarded_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 82, in ban_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 236, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 136, in handle
    result = await result
  File "/usr/src/homeassistant/homeassistant/components/calendar/__init__.py", line 273, in get
    [
  File "/usr/src/homeassistant/homeassistant/components/calendar/__init__.py", line 274, in <listcomp>
    dataclasses.asdict(event, dict_factory=_api_event_dict_factory)
  File "/usr/local/lib/python3.10/dataclasses.py", line 1239, in asdict
    return _asdict_inner(obj, dict_factory)
  File "/usr/local/lib/python3.10/dataclasses.py", line 1246, in _asdict_inner
    value = _asdict_inner(getattr(obj, f.name), dict_factory)
  File "/usr/local/lib/python3.10/dataclasses.py", line 1280, in _asdict_inner
    return copy.deepcopy(obj)
  File "/usr/local/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.10/copy.py", line 265, in _reconstruct
    y = func(*args)
  File "/usr/local/lib/python3.10/copy.py", line 264, in <genexpr>
    args = (deepcopy(arg, memo) for arg in args)
  File "/usr/local/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.10/copy.py", line 206, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/usr/local/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.10/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle '_thread.lock' object

Additional information

The error message seems very similar, but still different than in #71048.

home-assistant[bot] commented 2 years ago

caldav documentation caldav source

Andre-Schuiki commented 2 years ago

The problem

In 2022.11.1, my calendars appear empty in the home assistant integration due to an Exception occurring when reading/ updating calendar entries. After a downgrade to 2022.10.2, calendar entries show up as expected.

i have the same issue - i use SoGo for caldav.

jamieb60 commented 1 year ago

Same issue starting with 2022.11.1. Using radicale.

cottiAC commented 1 year ago

Same here with radicale starting with 2022.11.1. Still an issue with 2022.11.2

mklooss commented 1 year ago

you should take a look at this...

https://github.com/python-caldav/caldav/issues/226

OverZealous commented 1 year ago

Still happening with 2022.11.3 for me. I'm using a Zimbra server, but it's happening locally, not in getting the events. If you look at the calendar entity, you can see the next event in the attributes, but if you try to pull down events with that calendar, it throws the same error.

I'm not sure if there's a way to get more information than this. I've tried pulling down the data for specific dates or ranges, but it always renders the same error.

Logger: aiohttp.server
Source: components/calendar/__init__.py:274
First occurred: 8:44:08 PM (5 occurrences)
Last logged: 8:45:27 PM

Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 60, in security_filter_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 222, in forwarded_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 236, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 136, in handle
    result = await result
  File "/usr/src/homeassistant/homeassistant/components/calendar/__init__.py", line 273, in get
    [
  File "/usr/src/homeassistant/homeassistant/components/calendar/__init__.py", line 274, in <listcomp>
    dataclasses.asdict(event, dict_factory=_api_event_dict_factory)
  File "/usr/local/lib/python3.10/dataclasses.py", line 1239, in asdict
    return _asdict_inner(obj, dict_factory)
  File "/usr/local/lib/python3.10/dataclasses.py", line 1246, in _asdict_inner
    value = _asdict_inner(getattr(obj, f.name), dict_factory)
  File "/usr/local/lib/python3.10/dataclasses.py", line 1280, in _asdict_inner
    return copy.deepcopy(obj)
  File "/usr/local/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.10/copy.py", line 265, in _reconstruct
    y = func(*args)
  File "/usr/local/lib/python3.10/copy.py", line 264, in <genexpr>
    args = (deepcopy(arg, memo) for arg in args)
  File "/usr/local/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.10/copy.py", line 206, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/usr/local/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.10/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle '_thread.lock' object
mlg9000 commented 1 year ago

I have the same problem with Zimbra. Oddly I have 3 calendars there, two calendars work and one doesn't. The calendar that doesn't work is unchecked when I looked at the calendar on the sidebar. If I check it it shows no events and it doesn't retain the fact I've selected it the next time I view the calendar in HA. The next event for that calendar does show up in the attributes for the calendar entity, however. Nothing shows up on the calendar card for the calendar or atomic-calendar-revive cards.

palhaland commented 1 year ago

I have the same issue at hand as well. Currently I'm receiving the same error with "TypeError: cannot pickle '_thread.lock' object". I only get this error when I have an calendar event spanning multiple days, and it is not a full day event. When I change it to a full day event, then I do no longer get this error. (Although the event is only shown on the first day, unless seen as a single day)

calvinbui commented 1 year ago

Still happening on 2022.11.5

Logger: aiohttp.server
Source: components/calendar/__init__.py:274
First occurred: 5:17:33 PM (9 occurrences)
Last logged: 5:30:54 PM

Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.10/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 60, in security_filter_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 222, in forwarded_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 82, in ban_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 236, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 136, in handle
    result = await result
  File "/usr/src/homeassistant/homeassistant/components/calendar/__init__.py", line 273, in get
    [
  File "/usr/src/homeassistant/homeassistant/components/calendar/__init__.py", line 274, in <listcomp>
    dataclasses.asdict(event, dict_factory=_api_event_dict_factory)
  File "/usr/local/lib/python3.10/dataclasses.py", line 1239, in asdict
    return _asdict_inner(obj, dict_factory)
  File "/usr/local/lib/python3.10/dataclasses.py", line 1246, in _asdict_inner
    value = _asdict_inner(getattr(obj, f.name), dict_factory)
  File "/usr/local/lib/python3.10/dataclasses.py", line 1280, in _asdict_inner
    return copy.deepcopy(obj)
  File "/usr/local/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.10/copy.py", line 265, in _reconstruct
    y = func(*args)
  File "/usr/local/lib/python3.10/copy.py", line 264, in <genexpr>
    args = (deepcopy(arg, memo) for arg in args)
  File "/usr/local/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.10/copy.py", line 206, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/usr/local/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/usr/local/lib/python3.10/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle '_thread.lock' object
MichaKersloot commented 1 year ago

Importing the same Zimbra calendar with https://github.com/franc6/ics_calendar as an ics seems to work without problems.

OverZealous commented 1 year ago

Thank you, @MichaKersloot! That worked for me, as well, with a Zimbra calendar. I'm guessing something in the CalDAV integration is breaking how multi-day events are being stored.

itchannel commented 1 year ago

From what I can tell the issue lies with the CalendarEvent object that is being sent to the home assistant calendar class. The caldav integration includes a tzinfo object as part of the start and end time/date for the event, it also uses the tzicalvtz object which is where I believe the issue exists. 

[CalendarEvent(start=datetime.datetime(2022, 12, 7, 12, 30, tzinfo=<tzicalvtz 'Australia/Brisbane'>), end=datetime.datetime(2022, 12, 7, 13, 30, tzinfo=<tzicalvtz 'Australia/Brisbane'>), summary='TEST123', description=None, location=None), CalendarEvent(start=datetime.datetime(2022, 12, 9, 10, 0, tzinfo=<tzicalvtz 'Australia/Brisbane'>), end=datetime.datetime(2022, 12, 9, 11, 0, tzinfo=<tzicalvtz 'Australia/Brisbane'>), summary='Rusty', description='The following information couldn’t be added to Exchange:\nurl:', location=None)]

As shown above it includes the time object as 

"start=datetime.datetime(2022, 12, 9, 10, 0, tzinfo=<tzicalvtz 'Australia/Brisbane'>)"

If I edit the following lines ( 190, 191)to the caldav "calendar.py" file which removes the tzinfo object completely it then works fine and I can see all my calendar entries in HA. 

start=vevent.dtstart.value.replace(tzinfo=None),
end=self.get_end_date(vevent).replace(tzinfo=None),

This results in the following CalendarEvent being sent instead which is parsed fine by HA. 

[CalendarEvent(start=datetime.datetime(2022, 12, 7, 12, 30), end=datetime.datetime(2022, 12, 7, 13, 30), summary='TEST123', description=None, location=None), CalendarEvent(start=datetime.datetime(2022, 12, 9, 10, 0), end=datetime.datetime(2022, 12, 9, 11, 0), summary='Rusty', description='The following information couldn’t be added to Exchange:\nurl:', location=None)]

In the long run it's probably better to have the timezone included so more research would need to be done to find out why having tzicalvtz causes the thread lock error but for now this workaround works for me and is the same as how the google calendar integration works.

calvinbui commented 1 year ago

Still broken in 2022.12.1, but switching to https://github.com/franc6/ics_calendar works!

tjl10 commented 1 year ago

Still broken in 2022.12.8.

Logger: aiohttp.server
Source: components/calendar/__init__.py:341
...
TypeError: cannot pickle '_thread.lock' object

Is the CalDAV integration still being maintained?

What can I do to help diagnose or fix this?

cmsimike commented 1 year ago

Weirdly everything was working fine in 2022.11 for me. I only started noticing when I upgraded to 2022.12.

tjl10 commented 1 year ago

Looks like it was broken by some of @allenporter's work for the new Local Calendar feature. Specifically pull #79598 (Streamline calendar dataclass API/attribute conversions)

Removing the dataclasses.asdict(...) call and reverting it to how it was before (building an object manually out of the four properties) fixes it for me.

I'm talking about this block: https://github.com/home-assistant/core/pull/79598/files#diff-a1473c45f850b58510c6b9371eb1126416908d5a3228fdf6abe677fd15dc6cb2L255

I don't know what I'm doing so I'm sure something else will be broken now but at least the CalDAV events show up!

allenporter commented 1 year ago

Yes was discussing here https://github.com/home-assistant/core/issues/40127#issuecomment-1325637346 though didn't realize this has not been addressed yet. I had the impression it was going to be resolved. I can take a look and get this accelerated to simplify caldav.

allenporter commented 1 year ago

We've also seen similar issues back in #71048 from May which I also broke and helped resolve, though last time I just avoided the unnecessary copying. This time i think we need to change the behavior.

allenporter commented 1 year ago

To answer your question @tjl10 no caldav has no maintainer at the moment.

allenporter commented 1 year ago

Do you have a way to know what the ics files coming back from the server look like for this? So far I have been unable to reproduce this with caldav via Nextcloud and I suspect it has to do with some timezone setting on the calendar. In the mean time, i'm looking at ways to get caldav python to trigger this timezone code.

tjl10 commented 1 year ago

@allenporter thanks for fast response.

Yes was discussing here #40127 (comment)

Looks like you and @daniele-athome are on to something wrt tzid. I tried applying the changes in the patch from that thread, but it didn't make a difference to this problem (I guess because that thread is more about recurring events).

To answer your question @tjl10 no caldav has no maintainer at the moment.

I feel like I should offer to help, but I'm not a python programmer... This may not be a good project to learn it 🤔

Do you have a way to know what the ics files coming back from the server look like

I'm using radicale and am in UK (Europe/London) timezone. Here's my VCALENDAR with timezone stuff in it followed by a VEVENT:

BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//PYVOBJECT//NONSGML Version 1//EN
X-WR-CALNAME;VALUE=TEXT:RadiCalendar
BEGIN:VTIMEZONE
TZID:Europe/London
LAST-MODIFIED:20220816T024022Z
BEGIN:STANDARD
DTSTART:19961027T020000
RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU
TZNAME:GMT
TZOFFSETFROM:+0100
TZOFFSETTO:+0000
END:STANDARD
BEGIN:DAYLIGHT
DTSTART:19810329T010000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU
TZNAME:BST
TZOFFSETFROM:+0000
TZOFFSETTO:+0100
END:DAYLIGHT
END:VTIMEZONE
...
BEGIN:VEVENT
UID:850e3388-410b-42f3-9abd-1afbfeced739
DTSTART;TZID=Europe/London:20221228T080000
DTEND;TZID=Europe/London:20221229T140000
COLOR:seagreen
CREATED:20221222T194418Z
DTSTAMP:20221222T194436Z
LAST-MODIFIED:20221222T194436Z
STATUS:CONFIRMED
SUMMARY:No work
TRANSP:TRANSPARENT
END:VEVENT

Maybe putting that into the test data in core/tests/components/caldav/test_calendar.py might be a way to repro?

allenporter commented 1 year ago

Yeah, exactly what I was thinking about testing, thanks will give this a try.

allenporter commented 1 year ago

Thanks. I was able to reproduce this with your event. The part I was missing was the VTIMEZONE being present causes the python library to use a custom timezone implementation (with the thread lock)

tjl10 commented 1 year ago

Glad you've managed to repro! It's confusing me trying to see how it works. I assume there is some property on the event object (other than start, end, summary, description, and location) that uses the custom tz code. Is the best solution to exclude that property or fix the pickling like @daniele-athome has tried to do in his PR?

allenporter commented 1 year ago

Yes, rfc5545 calendars encode all timezone information so there is no ambiguity. It is doing the right thing by creating a python timezone object based on the timezone in the calendar. However, yeah the tzinfo object seems quite heavy weight if it requires a lock and probably should be simplified: Being able to copy a datetime seems very straight forward to me, so the right fix is in dateutil.

I am not optimistic that we'll see that happen soon as I mentioned before, so I think core needs a workaround either in core or in caldav python library.

tjl10 commented 1 year ago

Awesome, thank you!