search

python-caldav / caldav

Apache License 2.0
319 stars 94 forks source link

CalDAV-Error due to a location #151

Open Friedi1970 opened 3 years ago

Friedi1970 commented 3 years ago

The problem

Hi all,

Actually with the integration everything is working well but after restarting HA I realized the following problem with CalDAV. The thing is that suddenly no calendar data were loaded. Therefore I went to logs and got the following:

Logger: homeassistant.components.calendar
Source: components/caldav/calendar.py:201
Integration: Kalender (documentation, issues)
First occurred: 10:58:50 (6 occurrences)
Last logged: 10:58:53

caldav: Error on device update!
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 431, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 646, in async_device_update
    await task
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/caldav/calendar.py", line 135, in update
    self.data.update()
  File "/usr/src/homeassistant/homeassistant/util/__init__.py", line 277, in wrapper
    result = method(*args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/caldav/calendar.py", line 201, in update
    vevent = event.instance.vevent
  File "/usr/local/lib/python3.9/site-packages/caldav/objects.py", line 1058, in _get_vobject_instance
    self._set_vobject_instance(vobject.readOne(to_unicode(self._get_data())))
  File "/usr/local/lib/python3.9/site-packages/vobject/base.py", line 1155, in readOne
    return next(readComponents(stream, validate, transform, ignoreUnreadable,
  File "/usr/local/lib/python3.9/site-packages/vobject/base.py", line 1101, in readComponents
    vline = textLineToContentLine(line, n)
  File "/usr/local/lib/python3.9/site-packages/vobject/base.py", line 925, in textLineToContentLine
    return ContentLine(*parseLine(text, n), **{'encoded': True,
  File "/usr/local/lib/python3.9/site-packages/vobject/base.py", line 813, in parseLine
    raise ParseError("Failed to parse line: {0!s}".format(line), lineNumber)
vobject.base.ParseError: At line 13: Failed to parse line: X-APPLE-STRUCTURED-LOCATION;VALUE=URI;X-APPLE-MAPKIT-HANDLE=CAESoAIIrk0Q5cuty9Xtjc0ZGhIJvCAiNe3ySEARwCK/fojNJUAiZQoLRGV1dHNjaGxhbmQSAkRFGgZCYXllcm4qB0JhbWJlcmcyB0JhbWJlcmc6BTk2MDUwUhRLdW5pZ3VuZGVucnVoc3RyYcOfZVoCMzBiF0t1bmlndW5kZW5ydWhzdHJhw59lIDMwKhBUYXZlcm5hIEdlb3JnaW9zMhdLdW5pZ3VuZGVucnVoc3RyYcOfZSAzMDINOTYwNTAgQmFtYmVyZzILRGV1dHNjaGxhbmQ4L1pNCiQI5cuty9Xtjc0ZEhIJvCAiNe3ySEARwCK/fojNJUAYrk2QAwGiHyQI5cuty9Xtjc0ZGhgKEFRhdmVybmEgR2Vvcmdpb3MQACoCZGU=;X-APPLE-RADIUS=141.1750744752817;X-APPLE-REFERENCEFRAME=0;X-TITLE="Taverna Georgios

In the last line there is the parse error. What is really starnge for me is the fact that as soon as I delete the location from the corresponding event everything works fine again. Of course I have a lot of events with locations and only in some cases this error occur. In this case the address is as follows:

Taverna Georgios Kunigundenruhstraße 30, 96050 Bamberg, Deutschland

It seems the integration has a problem with locations.

The other strange thing is, that this event does not match the the search condition for the custom calendar.

Here's the YAML:

calendar:
  - platform: caldav
    username: 'Marco'
    password: 'areallysuperpassword'
    url: https://my.syno.installation.de/caldav.php/Marco
    custom_calendars:
      - name: urlaub_marco
        calendar: "Gemeinsam"
        search: "Marco: Urlaub"

What is version of Home Assistant Core has the issue?

core-2021.9.3

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

CalDAV

Link to integration documentation on our website

https://www.home-assistant.io/int…

Example YAML snippet

calendar:
  - platform: caldav
    username: 'Marco'
    password: 'areallysuperpassword'
    url: https://my.syno.installation.de/caldav.php/Marco
    custom_calendars:
      - name: urlaub_marco
        calendar: "Gemeinsam"
        search: "Marco: Urlaub"

Anything in the logs that might be useful for us?

Logger: homeassistant.components.calendar
Source: components/caldav/calendar.py:201
Integration: Kalender (documentation, issues)
First occurred: 10:58:50 (6 occurrences)
Last logged: 10:58:53

caldav: Error on device update!
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 431, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 646, in async_device_update
    await task
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/caldav/calendar.py", line 135, in update
    self.data.update()
  File "/usr/src/homeassistant/homeassistant/util/__init__.py", line 277, in wrapper
    result = method(*args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/caldav/calendar.py", line 201, in update
    vevent = event.instance.vevent
  File "/usr/local/lib/python3.9/site-packages/caldav/objects.py", line 1058, in _get_vobject_instance
    self._set_vobject_instance(vobject.readOne(to_unicode(self._get_data())))
  File "/usr/local/lib/python3.9/site-packages/vobject/base.py", line 1155, in readOne
    return next(readComponents(stream, validate, transform, ignoreUnreadable,
  File "/usr/local/lib/python3.9/site-packages/vobject/base.py", line 1101, in readComponents
    vline = textLineToContentLine(line, n)
  File "/usr/local/lib/python3.9/site-packages/vobject/base.py", line 925, in textLineToContentLine
    return ContentLine(*parseLine(text, n), **{'encoded': True,
  File "/usr/local/lib/python3.9/site-packages/vobject/base.py", line 813, in parseLine
    raise ParseError("Failed to parse line: {0!s}".format(line), lineNumber)
vobject.base.ParseError: At line 13: Failed to parse line: X-APPLE-STRUCTURED-LOCATION;VALUE=URI;X-APPLE-MAPKIT-HANDLE=CAESoAIIrk0Q5cuty9Xtjc0ZGhIJvCAiNe3ySEARwCK/fojNJUAiZQoLRGV1dHNjaGxhbmQSAkRFGgZCYXllcm4qB0JhbWJlcmcyB0JhbWJlcmc6BTk2MDUwUhRLdW5pZ3VuZGVucnVoc3RyYcOfZVoCMzBiF0t1bmlndW5kZW5ydWhzdHJhw59lIDMwKhBUYXZlcm5hIEdlb3JnaW9zMhdLdW5pZ3VuZGVucnVoc3RyYcOfZSAzMDINOTYwNTAgQmFtYmVyZzILRGV1dHNjaGxhbmQ4L1pNCiQI5cuty9Xtjc0ZEhIJvCAiNe3ySEARwCK/fojNJUAYrk2QAwGiHyQI5cuty9Xtjc0ZGhgKEFRhdmVybmEgR2Vvcmdpb3MQACoCZGU=;X-APPLE-RADIUS=141.1750744752817;X-APPLE-REFERENCEFRAME=0;X-TITLE="Taverna Georgios

Additional information

There is already An issue in core (https://github.com/home-assistant/core/issues/55778). I already tested the suggested solution with caldav 0.8.0 but unfortunately the error still occurs.

tobixen commented 3 years ago

So ...

This looks like a non-standard location. I do believe the ical standard supports location, but "X-APPLE-STRUCTURED-LOCATION" is apparently an Apple extension of the ical standard. Anyway, it starts with "X-", hence it should pass according to the standard.

The vobject library is having problems with this line though. That means one out of three, either that the caldav library does something funky before passing the data to the vobject library (we did have some issues with line breaks handled wrongly in some earlier version, but it should be fixed in 0.8.0 if I'm not mistaken), that there is a bug in vobject, or that the line is not valid icalendar data. I can see two things, the data you've pasted ends with X-TITLE="Taverna Georgios, and the address you've pasted contains a line break. I guess the line break is breaking things.

Can you send the full icalendar data? (I can help you a bit later on turning on debugging logging in the caldav library if you can't figure out of it).

The second issue should probably be split out in a separate issue. I have to run now ...

Friedi1970 commented 3 years ago

Can you send the full icalendar data? (I can help you a bit later on turning on debugging logging in the caldav library if you can't figure out of it).

Then I need help. No idea how to do.

tobixen commented 3 years ago

I've looked a bit into it. The traceback you've posted is from version 0.7.1. In version 0.8.0, the ical should be logged if vobject won't accept it. Please try version 0.8.0 and post the traceback and any extra logging from 0.8.0.

Friedi1970 commented 3 years ago

Here you are 

Logger: homeassistant.components.calendar
Source: custom_components/caldav/calendar.py:201 
Integration: Kalender (documentation, issues) 
First occurred: 21:03:11 (6 occurrences) 
Last logged: 21:03:15

caldav: Error on device update!
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 431, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 633, in async_device_update
    await task
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/caldav/calendar.py", line 135, in update
    self.data.update()
  File "/usr/src/homeassistant/homeassistant/util/__init__.py", line 277, in wrapper
    result = method(*args, **kwargs)
  File "/config/custom_components/caldav/calendar.py", line 201, in update
    vevent = event.instance.vevent
  File "/usr/local/lib/python3.9/site-packages/caldav/objects.py", line 1516, in _get_vobject_instance
    self._set_vobject_instance(vobject.readOne(to_unicode(self._get_data())))
  File "/usr/local/lib/python3.9/site-packages/vobject/base.py", line 1155, in readOne
    return next(readComponents(stream, validate, transform, ignoreUnreadable,
  File "/usr/local/lib/python3.9/site-packages/vobject/base.py", line 1101, in readComponents
    vline = textLineToContentLine(line, n)
  File "/usr/local/lib/python3.9/site-packages/vobject/base.py", line 925, in textLineToContentLine
    return ContentLine(*parseLine(text, n), **{'encoded': True,
  File "/usr/local/lib/python3.9/site-packages/vobject/base.py", line 813, in parseLine
    raise ParseError("Failed to parse line: {0!s}".format(line), lineNumber)
vobject.base.ParseError: At line 28: Failed to parse line: X-APPLE-STRUCTURED-LOCATION;VALUE=URI;X-APPLE-REFERENCEFRAME=0;X-TITLE=Georgios Restaurant Taverna\
tobixen commented 3 years ago

Hm, the critical-logging is missing - it probably was "eaten up" by homeassistant. I suggest hacking up /usr/local/lib/python3.9/site-packages/caldav/objects.py around line 1516. It should look like this:

    def _get_vobject_instance(self):
        if not self._vobject_instance:
            try:
                self._set_vobject_instance(vobject.readOne(to_unicode(self._get_data())))
            except:
                log.critical("Something went wrong while loading icalendar data into the vobject class.  ical url: " + str(self.url))
                raise
        return self._vobject_instance

Change the logic in the except-block into this:

raise(Exception("Something went wrong while loading icalendar data into the vobject class.  ical url: " + str(self.url))

... and see if you can get out the ical data this way.

Friedi1970 commented 3 years ago

At the moment I am in Greece (vacation) but as soon as possible I will try.

Let me understand. At the moment it is just logging and after the change is it. 🤔 yeah, what?

tobixen commented 3 years ago

I'm also relatively far from home at the moment :-)

There are two kind of logging here, it's the regular logging and the traceback. The traceback is visible for you, but we don't know where the logging is going in your case (maybe home assistant puts it into a log file or simply mutes it), so we make a workaround by throwing the log data into the traceback.

Tracebacks occurs whenever the application or a function in the application encounters some problem it cannot recover from. Tracebacks can often come without the programmer intending it, but one can also explicitly jump out of the program with a traceback by raising an exception using the "raise" statement. Exceptions can also be caught using the "except"-statement.

In this case the vobject library throws an exception because it deems the icalendar data as invalid. Caldav 0.7 would not do anything special with this, so there would just be some traceback from vobject there. Due to complaints about caldav breaking with odd error messages, I decided to catch this exception in 0.8 just to log some explanation and the ical data. "raise" without anything more will cause the original exception to be re-raised. Raise with some exception object will raise a new exception.

Friedi1970 commented 3 years ago

I'm also relatively far from home at the moment :-)

Hm, but my problem is that I really have no idea how to edit the file you‘re talking about. This is nothing within the HA environment, right?

tobixen commented 3 years ago

Hm, but my problem is that I really have no idea how to edit the file you‘re talking about. This is nothing within the HA environment, right?

I don't know anything about HA.

How did you upgrade the library? I just assumed that if you can upgrade the library, you should also be able to edit the file :-)

Friedi1970 commented 3 years ago

Hm, but my problem is that I really have no idea how to edit the file you‘re talking about. This is nothing within the HA environment, right?

I don't know anything about HA.

How did you upgrade the library? I just assumed that if you can upgrade the library, you should also be able to edit the file :-)

😀well, in Home Assistant the library is in a custom folder. The file you are talking about is part of the library?

tobixen commented 3 years ago

The traceback points to /usr/local/lib/python3.9/site-packages/caldav/objects.py. This is a part of the caldav library, and I can read from the line number that's the 0.8-version of caldav. If Home Assistant runs in a chroot environment or a container, then it needs to be fixed inside Home Assistant. Otherwise /usr/local/lib/python3.9/site-packages/caldav/objects.py should be hacked up directly at the computer or virtual machine running Home Assistant.

What I want to do is to check that the ical data from the server is valid. This can probably also be done by using some third party tool, or maybe even google for problems with X-APPLE-STRUCTURED-LOCATION and vobject. I can probably do the latter.

tobixen commented 3 years ago

I don't think it's needed to bother with fishing out the ical. I found an old issue, #37, someone having the same problem, but with the ical given. I will re-open issue #37 and close this as a duplicate.

tobixen commented 3 years ago

With #37, the problem was apparently with some trailing whitespace. I've added a potential workaround, stripping away trailing whitespace before sending the data to the vobject class, that ought to fix the problem. The fix is in the master branch of this git repository. I would like some feedback on weather it works or not before I release it ... but it may be a bit catch-22 as I don't know if you will be able to upgrade the library to an unreleased version?

Of course it could be that there is some different problem with your data, in that case we will need to reopen this issue.

Friedi1970 commented 2 years ago

I asked @mib1185 for help because I really have no idea how to help on this issue :-/

tobixen commented 2 years ago

Perhaps this works for you:

sudo -H pip install -U git+https://github.com/python-caldav/caldav.git
tobixen commented 2 years ago

I just released 0.8.1. I would be happy if you can manage to test that version of the library and see if it will fix your issues.

Friedi1970 commented 2 years ago

Unfortunately sometimes the problem still occurs. Here is one which occured today: `Logger: homeassistant.components.calendar Source: components/caldav/calendar.py:210 Integration: Kalender (documentation, issues) First occurred: 20:19:16 (7 occurrences) Last logged: 20:19:25

caldav: Error on device update! Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 431, in _async_add_entity await entity.async_device_update(warning=False) File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 736, in async_device_update await task File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, self.kwargs) File "/usr/src/homeassistant/homeassistant/components/caldav/calendar.py", line 144, in update self.data.update() File "/usr/src/homeassistant/homeassistant/util/init.py", line 192, in wrapper result = method(*args, *kwargs) File "/usr/src/homeassistant/homeassistant/components/caldav/calendar.py", line 210, in update if not hasattr(event.instance, "vevent"): File "/usr/local/lib/python3.9/site-packages/caldav/objects.py", line 1585, in _get_vobject_instance self._set_vobject_instance(vobject.readOne(to_unicode(self._get_data()))) File "/usr/local/lib/python3.9/site-packages/vobject/base.py", line 1155, in readOne return next(readComponents(stream, validate, transform, ignoreUnreadable, File "/usr/local/lib/python3.9/site-packages/vobject/base.py", line 1101, in readComponents vline = textLineToContentLine(line, n) File "/usr/local/lib/python3.9/site-packages/vobject/base.py", line 925, in textLineToContentLine return ContentLine(parseLine(text, n), {'encoded': True, File "/usr/local/lib/python3.9/site-packages/vobject/base.py", line 813, in parseLine raise ParseError("Failed to parse line: {0!s}".format(line), lineNumber) vobject.base.ParseError: At line 13: Failed to parse line: X-APPLE-STRUCTURED-LOCATION;VALUE=URI;X-TITLE=Greifenklau Privater Brauereigasthof\ `

tobixen commented 2 years ago

Is there really a line break between Greifenklau and Privater Brauereigasthof in the error message?

Friedi1970 commented 2 years ago

What do you mean? The error above is copied and pasted. Can’t tell more 🤷‍♂️

Friedi1970 commented 2 years ago

Is there really a line break between Greifenklau and Privater Brauereigasthof in the error message?

Btw. There is no line break …

Friedi1970 commented 1 year ago

It was a nice time when everything worked fine for a year or so ;) Since the last updates of IOS/MacOS something is going wrong again :( @tobixen could you please have a look? I got in Home Assistant (caldav integration) the error below. I got these errors with new created calendar entries after the updates . After deletion of the new created calendar entries everything was fine again.

Logger: homeassistant.components.calendar
Source: components/caldav/calendar.py:212
Integration: Kalender (documentation, issues)
First occurred: 05:38:08 (2 occurrences)
Last logged: 05:38:11
caldav: Error on device update!

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 507, in _async_add_entity
    await entity.async_device_update(warning=False)
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 732, in async_device_update
    await task
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/caldav/calendar.py", line 147, in update
    self.data.update()
  File "/usr/src/homeassistant/homeassistant/util/__init__.py", line 190, in wrapper
    result = method(*args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/caldav/calendar.py", line 212, in update
    results = self.calendar.search(
  File "/usr/local/lib/python3.10/site-packages/caldav/objects.py", line 1067, in search
    component = o.icalendar_component
  File "/usr/local/lib/python3.10/site-packages/caldav/objects.py", line 1818, in _get_icalendar_component
    for x in self.icalendar_instance.subcomponents
  File "/usr/local/lib/python3.10/site-packages/caldav/objects.py", line 2283, in _get_icalendar_instance
    self.icalendar_instance = icalendar.Calendar.from_ical(
  File "/usr/local/lib/python3.10/site-packages/icalendar/cal.py", line 328, in from_ical
    name, params, vals = line.parts()
  File "/usr/local/lib/python3.10/site-packages/icalendar/parser.py", line 348, in parts
    values = unescape_string(st[value_split + 1:])
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
tobixen commented 1 year ago

The error above seems to come from the icalendar library and not the caldav library. It would be nice to get this problem isolated and reproduced though. Do you have the ical data for the events?

Friedi1970 commented 1 year ago

The error above seems to come from the icalendar library and not the caldav library.

I can tell you, it is caldav. But it was (again) one isolated appointment and it seems it has something to do with the location.

I deleted that one (unfortunately) but I try to reproduce.

tobixen commented 1 year ago

The traceback shows that it's homeassistant calling on caldav which again calls on icalendar to parse the ical. When the icalendar library is parsing the ical, something goes wrong. Hence I will insist that what we are seeing here is either a bug in the icalendar library or a problem with broken ical data. :-)

This code (in the caldav library) has been changed a couple of times during the last year, earlier the library would just pass on the icalendar data without doing any parsing, then someone wrote code to do client-side expansion of recurring events, but used the vobject library for parsing the ical. Later I rewrote it to use the icalendar library. Hence the problem may have been triggered by upgrading home assistant.

tobixen commented 1 year ago

(given the history of this problem, I'd say the problem is likely to be with the software producing the ical code - the resolution to the original problem was to throw in some workarounds in the caldav code fixing broken icalendar code)

Friedi1970 commented 1 year ago

Hmm. As far as I remember there was some Apple spec. stuff and I think it is the same problem again. I tried to reproduce the error but I can't. Anyway, it seems it has something to do with the location field. It always happened when I took over the suggested value. It never happened when I manually put in the location.

Unfortunately I can't tell you whether it is on MacOS or on IOS or on both of them.