search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
70.14k stars 29.18k forks source link

CalDAV does not create calendar entities for Kopano servers after 2023.11.0 #105680

Open t0mcat1337 opened 7 months ago

t0mcat1337 commented 7 months ago

The problem

I used caldav component for a while now which was working fine till nowadays. I just realized that all configured calendars are gone from the calendar view in the GUI and some of my automations using them stopped working, too.

I found out that the last working Core version was 2023.10.5. After upgrading to 2023.11.0 calendars disappear from the GUI. When calendars should be synced I just see a few of these lines in homeassistant.log:

2023-12-13 19:45:17.249 WARNING (SyncWorker_20) [urllib3.connectionpool] Connection pool is full, discarding connection: kopano.server.url . Connection pool size: 10

Could this PR be responsible? https://github.com/home-assistant/core/pull/102089 (Update caldav to use an DataUpdateCoordinator for fetching data)

This could be reproduced no matter what type of installation (HomeAssistant OS, Container...)

What version of Home Assistant Core has the issue?

core-2023.11.0 and following

What was the last working version of Home Assistant Core?

core-2023.10.5

What type of installation are you running?

Home Assistant OS

Integration causing the issue

caldav

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

calendar:
  - platform: caldav
    username: <user>
    password: <pass>
    url: http://kopano.server.url:8080/caldav/user/calendar

Anything in the logs that might be useful for us?

2023-12-13 19:45:17.249 WARNING (SyncWorker_20) [urllib3.connectionpool] Connection pool is full, discarding connection: kopano.server.url . Connection pool size: 10

Additional information

Calendar Server used is a Kopano Server running in the same LAN as HomeAssistant does.

home-assistant[bot] commented 7 months ago

caldav documentation caldav source

onkelbeh commented 7 months ago

same here, HA 2023.12.3, caldav-1.3.8, nextcloud-27.1.4. Looks like this happens immediately at startup, later the calendar resumes to work. Have not seen this message again since 13 hours.

t0mcat1337 commented 7 months ago

What @onkelbeh describes does not happen for me... HA is running for many days without restart now and calendars never reappeared.

onkelbeh commented 6 months ago

Just wnted to tell, the error still persists with 2024.1.2, and and my case only at startup, calender resumes work after a couple of minutes.

t0mcat1337 commented 6 months ago

For me, the error persists exactly as described until the current version.

I tracked down the problem with the help of Kopanos ical gateway debug functionalities. In those logs, the requests/responses from Homeassistant can be followed in detail and I was able to compare the "chats" between Homeassistant Version 2023.10.5 (the last working version) with the most recent stable version, where caldav doesn't work any more in this case. These are the results:

First a few PROPFIND requests to the configured base calendar URL are made for determining the available calendars and details for the user. Here is the last one where the real calendars and their URLs are reported:

Request

< PROPFIND /caldav/myusername HTTP/1.1
Request body:
<?xml version='1.0' encoding='utf-8'?>
<D:propfind xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"><D:prop><D:displayname/><D:resourcetype/></D:prop></D:propfind>

Response

> HTTP/1.1 207 Multi-Status
Response body:
<?xml version="1.0" encoding="UTF-8"?>
<C:multistatus xmlns:D="urn:ietf:params:xml:ns:caldav" xmlns:C="DAV:">
 <C:response>
  <C:href>/caldav/myusername/</C:href>
  <C:propstat>
   <C:prop>
    <C:displayname>My Name</C:displayname>
    <C:resourcetype>
     <C:collection/>
    </C:resourcetype>
   </C:prop>
   <C:status>HTTP/1.1 200 OK</C:status>
  </C:propstat>
 </C:response>
 <C:response>
  <C:href>/caldav/myusername/FLDPRFX_4FB1F03DBBAC4751977DC1CBCB1F6209/</C:href>
  <C:propstat>
   <C:prop>
    <C:displayname>CalendarArchive</C:displayname>
    <C:resourcetype>
     <C:collection/>
     <D:calendar/>
    </C:resourcetype>
   </C:prop>
   <C:status>HTTP/1.1 200 OK</C:status>
  </C:propstat>
 </C:response>
 <C:response>
  <C:href>/caldav/myusername/FLDPRFX_021D495F22714572A266238372F5C8BE/</C:href>
[...]

Until this point, the two Homassistant versions behave exactly identical. After that the real data of the reported calendars should be getted. Here lays the difference between the working version and the currently not working one:

Current version 2024.1.3: Here, a PROPFIND request is made to the real calendar URL:

< PROPFIND /caldav/myusername/FLDPRFX_4FB1F03DBBAC4751977DC1CBCB1F6209/ HTTP/1.1
Request body:
<?xml version='1.0' encoding='utf-8'?>
<D:propfind xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"><D:prop><C:supported-calendar-component-set/></D:prop></D:propfind>

This is answered like this by Kopano:

> HTTP/1.1 207 Multi-Status
Response body:
<?xml version="1.0" encoding="UTF-8"?>
<C:multistatus xmlns:D="urn:ietf:params:xml:ns:caldav" xmlns:C="DAV:">
 <C:response>
  <C:href>/caldav/myusername/FLDPRFX_4FB1F03DBBAC4751977DC1CBCB1F6209/</C:href>
  <C:propstat>
   <C:prop>
    <D:supported-calendar-component-set>
     <D:comp name="VTIMEZONE"/>
    </D:supported-calendar-component-set>
   </C:prop>
   <C:status>HTTP/1.1 200 OK</C:status>
  </C:propstat>
 </C:response>
</C:multistatus>

After that, no request is made any more from Homeassistant.

Working version 2023.10.5 This behaves slightly different... The above request to the calendar URL for getting data is NOT a PROPFIND, BUT a REPORT one and - as a plus - with different request - body:

< REPORT /caldav/myusername/FLDPRFX_4FB1F03DBBAC4751977DC1CBCB1F6209/ HTTP/1.1
Request body:
<?xml version='1.0' encoding='utf-8'?>
<C:calendar-query xmlns:D="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav"><D:prop><C:calendar-data><C:expand start="20240115T230000Z" end="20240116T230000Z"/></C:calendar-data></D:prop><C:filter><C:comp-filter name="VCALENDAR"><C:comp-filter name="VEVENT"><C:time-range start="20240115T230000Z" end="20240116T230000Z"/></C:comp-filter></C:comp-filter></C:filter></C:calendar-query>

This of course is answered different / correctly with calendar data from Kopano:

> HTTP/1.1 207 Multi-Status
Response body:
<?xml version="1.0" encoding="UTF-8"?>
<C:multistatus xmlns:D="urn:ietf:params:xml:ns:caldav" xmlns:C="DAV:">
 <C:response>
  <C:href>/caldav/myusername/FLDPRFX_4FB1F03DBBAC4751977DC1CBCB1F6209/7418df20-c8f1-11da-ae80-910833c1dace.ics</C:href>
  <C:propstat>
   <C:prop>
    <D:calendar-data>BEGIN:VCALENDAR&#13;
VERSION:2.0&#13;
PRODID:-//Kopano//8.7.25//EN&#13;
CALSCALE:GREGORIAN&#13;
METHOD:PUBLISH&#13;
BEGIN:VTIMEZONE&#13;
TZID:Europe/Berlin&#13;
[...]

I have no idea, why the current version obviously does a PROPFIND and not a REPORT request to the real calendar URL (which seems to be correct), as the former version did...

t0mcat1337 commented 6 months ago

OK, digged a bit deeper and found the root cause for my special problem:

Starting with version 2023.11.0 the caldav component filters out caldav calendars which supposedly do not support "VEVENTS". This is the first occurance in the caldav component code (in calendar.py):

image

entered with this PR: https://github.com/home-assistant/core/pull/102059

But in my case Kopano answers "get_supported_components()" just with "VTIMEZONE", missing "VEVENT", despite calendars of course supporting events (I think, this is a bug in Kopano). This also can be seen in the "PROPFIND" request of my last post.

In the current version, code hase changed further, so the relevant snippet is this:

image

When I "dirty" change this var to "VTIMEZONE" and restart homeassistant, calendars reappear and working as expected.

Perhaps it's worth thinking about a HomeAssistant option, if those (buggy?) calendars should really been filtered out...

allenporter commented 6 months ago

The motivation for this change is to not return random objects (e.g. todo calendars) as event calendars. have you filed a bug upstream with the Kopano Server folks? Would be nice to make it correctly return that it supports VEVENT if it supports VEVENT

allenporter commented 6 months ago

From https://icalendar.org/CalDAV-Access-RFC-4791/5-2-3-caldav-supported-calendar-component-set-property.html:

"Any attempt by the client to store calendar object resources with component types not listed in this property, if it exists, MUST result in an error, with the CALDAV:supported-calendar-component precondition (Section 5.3.2.1) being violated"

so i think what this is doing is correct on the home assistant side unless i misunderstand

t0mcat1337 commented 6 months ago

The motivation for this change is to not return random objects (e.g. todo calendars) as event calendars. have you filed a bug upstream with the Kopano Server folks? Would be nice to make it correctly return that it supports VEVENT if it supports VEVENT

No, I didn't... when following the news regarding Kopano, it seems they end the on-premise installation option and just offer cloud - based solutions, which for us will result in migrating to sth. completely different.

t0mcat1337 commented 6 months ago

From https://icalendar.org/CalDAV-Access-RFC-4791/5-2-3-caldav-supported-calendar-component-set-property.html:

"Any attempt by the client to store calendar object resources with component types not listed in this property, if it exists, MUST result in an error, with the CALDAV:supported-calendar-component precondition (Section 5.3.2.1) being violated"

so i think what this is doing is correct on the home assistant side unless i misunderstand

Sure, concerning these specs, HA is behaving correctly... but as we all know, not each and every other software might respect those specs completely for whatever reason (not knowing them in detail? Bugs?) and so I wondered if such an option could help to (read - only) consume such buggy calendar sources...

allenporter commented 6 months ago

I'm asking to explore fixing the buggy server so that all clients of that buggy server benefit. I am not aware that such outreach has happened?

allenporter commented 5 months ago

Reading further in the spec there could be some ambiguity in how this is interpreted:

The empty-element tag <C:comp name="VTIMEZONE"/> MUST
 only be specified if support for calendar object resources that
 only contain VTIMEZONE components is provided or desired.

Enough so that I can see it being reasonable to add a hack for this.

issue-triage-workflows[bot] commented 2 months ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

gitandriu commented 2 months ago

Still does not work with latest versions