search

pimutils / vdirsyncer

📇 Synchronize calendars and contacts.
https://vdirsyncer.pimutils.org/
Other
1.57k stars 163 forks source link

error: Unknown error occurred for calendars #832

Open rkintzi opened 4 years ago

rkintzi commented 4 years ago

Hi,

I receive following error since today. Let me know if there is any more information I can provide.

  • Your vdirsyncer version 
    [~]$ vdirsyncer --version
vdirsyncer, version 0.16.9.dev0+gb5dd092.d20200612
  • If applicable, which server software (and which version) you're using Fastmail
  • Your Python version 
    [~]$ python --version
Python 3.8.3
  • Your operating system

Arch Linux

  • Your config file 
    
[~]$ cat .config/vdirsyncer/config
[general]
status_path = "~/.vdirsyncer/status"

[pair contacts] a = "contacts_local" b = "contacts_remote" collections = ["from a", "from b"]

[pair calendars] a = "calendars_local" b = "calendars_remote" collections = ["from a", "from b"]

[storage contacts_local] type = "filesystem" path = "~/.contacts" fileext = ".vcf"

[storage contacts_remote] type = "carddav" url = "https://carddav.fastmail.com/" username = "..." password.fetch = ["command", "/home/radek/bin/getpw.py", "...", "..."]

[storage calendars_local] type = "filesystem" path = "~/.calendars" fileext = ".ics"

[storage calendars_remote] type = "caldav" url = "https://caldav.fastmail.com/" username = "..." password.fetch = ["command", "/home/radek/bin/getpw.py", "...", "..."]

> * Use `vdirsyncer -vdebug` for debug output. The output is sensitive, but
  please attach at least the last few lines before the error (if applicable),
  censored as necessary. This is almost always the most useful 
information.

error: Unknown error occurred for calendars/...: /dav/calendars/user/.../.../... error: Use -vdebug to see the full traceback. debug: File "/usr/lib/python3.8/site-packages/vdirsyncer/cli/tasks.py", line 62, in sync_collection debug: sync.sync( debug: File "/usr/lib/python3.8/site-packages/vdirsyncer/sync/init.py", line 135, in sync debug: b_nonempty = b_info.prepare_new_status() debug: File "/usr/lib/python3.8/site-packages/vdirsyncer/sync/init.py", line 57, in prepare_new_status debug: for href, item, etag in (self.storage.get_multi(prefetch) debug: File "/usr/lib/python3.8/site-packages/vdirsyncer/storage/dav.py", line 503, in get_multi debug: raise exceptions.NotFoundError(href) error: 1 out of 13 tasks failed.

stweller commented 4 years ago

I see the same issue. Unfortunately the calendars in question are not synchronized at all, after starting from scratch with empty status and empty local storage, nothing is synced anymore.

stweller commented 4 years ago

Oh, and by the way I'm also syncing with fastmail, so it is possible that this is an error on their side. @WhyNotHugo, is there a simple way to check the consistency of the caldav data? At least one of the calendars where this error happens does not have very personal data in it, so I would also be willing to share it.

WhyNotHugo commented 4 years ago

Do you have event entries and todo entries in the same local calendar?

WhyNotHugo commented 4 years ago

(Your config looks fine, BTW, the fastmail-specific part is almost identical to mine).

stweller commented 4 years ago

Do you have event entries and todo entries in the same local calendar?

I don't, and the error also happens when I remove the status directory and start from scratch.

bmarcote commented 4 years ago

I have the issue with vdirsyncer 0.16.7 when synchronizing an iCloud account. Five calendar, four synchronize properly but in one I get the same issue.

It seems like it cannot find one of the .ics files that (before) got from the given calendar? (follows the error part from -vdebug; the rest of the sync seems to go smooth).

debug: Already normalized: '/268437010/calendars/E71734DA-CC16-4DE5-AA5D-4B7BC03A66CC/F9905C5E-7612-4FA9-9D24-804232303ACB.ics'
debug: Already normalized: '/268437010/calendars/E71734DA-CC16-4DE5-AA5D-4B7BC03A66CC/6445043D-A199-490E-9317-5AD03DFFFCD2.ics'
debug:   File "/usr/lib/python3/dist-packages/vdirsyncer/cli/tasks.py", line 64, in sync_collection
debug:     sync.sync(
debug:   File "/usr/lib/python3/dist-packages/vdirsyncer/sync/__init__.py", line 136, in sync
debug:     b_nonempty = b_info.prepare_new_status()
debug:   File "/usr/lib/python3/dist-packages/vdirsyncer/sync/__init__.py", line 58, in prepare_new_status
debug:     for href, item, etag in (self.storage.get_multi(prefetch)
debug:   File "/usr/lib/python3/dist-packages/vdirsyncer/storage/dav.py", line 505, in get_multi
debug:     raise exceptions.NotFoundError(href)
Syncing calendars/EEF9E72D-C708-4687-A29D-1885F7016143
debug: PROPFIND https://p24-caldav.icloud.com:443/268437010/calendars/EEF9E72D-C708-4687-A29D-1885F7016143/
debug: {'User-Agent': 'vdirsyncer/0.16.7', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '1'}
debug: <?xml version="1.0" encoding="utf-8" ?>
debug:             <D:propfind xmlns:D="DAV:">
debug:                 <D:prop>
debug:                     <D:resourcetype/>
debug:                     <D:getcontenttype/>
debug:                     <D:getetag/>
debug:                 </D:prop>
debug:             </D:propfind>
debug:             
debug: Sending request...
debug: 207

Given that the local configuration seem to be only calendar related, it is not obvious how to solve this issue. Maybe an exception catcher and convert a failing .ics reading into a warning, so the rest of the calendar can be retrieved? As currently no data is obtained for the whole calendar.

LeSnake04 commented 4 years ago

may try moving ~/.calenders and ~./.vdirsyncer away from home (so you can restore them later if needed) and regenerate the directories with: vdirsyncer discover vdirsyncer metasync vdirsyncer sync

It helped me out when I got a weird errors.

stweller commented 4 years ago

Thanks, @LeSnake04 , but that's what I meant above by "starting from scratch with empty status and empty local storage". It doesn't help. Either there is some bug in vdirsyncer or in the fastmail server implementation, I cannot really tell which is the case.

edigiacomo commented 4 years ago

I have the same issue with vdirsyncer 0.16.7

debug: Normalized URL from '/caldav/Y2FsOi8vMC8zMQ/20120830T181649Z-1346422920000%40fe80%3A0%3A0%3A0%3A215%3A17ff%3Afedc%3A562c%15.ics' to '/caldav/Y2FsOi8vMC8zMQ/20120830T181649Z-1346422920000@fe80:0:0:0:215:17ff:fedc:562c%15.ics'
debug: Skipping '/caldav/Y2FsOi8vMC8zMQ/20120830T181649Z-1346422920000@fe80:0:0:0:215:17ff:fedc:562c%15.ics', properties are missing.
error: Unknown error occured for mailboxdotorg_calendars: /caldav/Y2FsOi8vMC8zMQ/20130125T144935Z-1359137220000@fe80:0:0:0:215:17ff:fedc:5224%15.ics
error: Use `-vdebug` to see the full traceback.
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/cli/tasks.py", line 64, in sync_collection
debug:     sync.sync(
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/sync/__init__.py", line 136, in sync
debug:     b_nonempty = b_info.prepare_new_status()
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/sync/__init__.py", line 58, in prepare_new_status
debug:     for href, item, etag in (self.storage.get_multi(prefetch)
debug:   File "/usr/lib/python3.8/site-packages/vdirsyncer/storage/dav.py", line 505, in get_multi
debug:     raise exceptions.NotFoundError(href)

I have tried to start from scratch but the problem still persists.

acheam0 commented 3 years ago

Not a pretty solution, but just commenting out the lines that raise the exception worked for me.

dennis-hamann commented 1 year ago

I got the same error as @edigiacomo . I have multiple google calendars but only one of them yields this error. @ArmaanB Which lines did you edit? May you provide more details about your solution?

acheam0 commented 1 year ago

@ArmaanB Which lines did you edit? May you provide more details about your solution?

Sorry, it’s been a few years, I don’t recall what I did

stweller commented 1 year ago

@dennis-hamann I believe the fix is just to go to line 505 in dav.py (line may be different in your version, just whatever the error message says) and replace the raise exceptions.NotFoundError(href) with something like dav_logger.warning("Not found: {}".format(href)). That being said, I also had this issue in the past and I think it was fixed in some more recent version. So maybe updating is all you need...

WhyNotHugo commented 1 year ago

@edigiacomo The URL for your calendar entry here have some pretty unusual character combinations, and I suspect that it's not being escaped properly somehow.

Can you use davcli and check its output? In particular davcli caldav ... list-calendar-components and davcli caldav ... get.

davcli is a small tool to inspect caldav remote servers, and I want to compare its output to what vdirsyncer is logging.

ghost commented 1 year ago

I have a similar issue and I notice that only happens with 

start_date = "datetime.now() - timedelta(days=60)"                                                                                                                                                                                           
end_date = "datetime.now() + timedelta(days=60)"

I change from 150 to 60 and same issue, after remove it completely the issue gone, I guess is related with recurrent events that are too old so the next occurrence event Not Found the parent.

Also with start_date end_date the warning warning: Skipping identical href: appears

edigiacomo commented 1 year ago

@edigiacomo The URL for your calendar entry here have some pretty unusual character combinations, and I suspect that it's not being escaped properly somehow.

Can you use davcli and check its output? In particular davcli caldav ... list-calendar-components and davcli caldav ... get.

davcli is a small tool to inspect caldav remote servers, and I want to compare its output to what vdirsyncer is logging.

In list-calendar-components.txt you can find the output of list-calendar-components:

$ DAVCLI_PASSWORD="$PASSWORD" target/release/davcli caldav --server-url "$SERVER_URL" --username "$USERNAME" list-calendar-components /caldav/Y2FsOi8vMC8zMQ/ > list-calendar-components.txt

I have found the resource href of the calendar entry and it seems that it's escaped in the same way:

$ grep 20120830T181649Z list-calendar-components.txt 
/caldav/Y2FsOi8vMC8zMQ/20120830T181649Z-1346422920000@fe80:0:0:0:215:17ff:fedc:562c%15.ics

But I can't download the entry 

$ DAVCLI_PASSWORD="$PASSWORD" target/release/davcli caldav --server-url "$SERVER_URL" --username "$USERNAME" get "/caldav/Y2FsOi8vMC8zMQ/20120830T181649Z-1346422920000@fe80:0:0:0:215:17ff:fedc:562c%15.ics"
Error: http request returned 400 Bad Request

And I get the same error with every href in the entries list: maybe I'm using davcli wrong?

I don't know if it can be useful, but I managed to download the entry escaping the % (%25) in the string /caldav/Y2FsOi8vMC8zMQ/20120830T181649Z-1346422920000@fe80:0:0:0:215:17ff:fedc:562c%15.ics (I have redacted some text to ***):


$ curl -u "$USERNAME:$PASSWORD" "${SERVER_URL}/caldav/Y2FsOi8vMC8zMQ/20120830T181649Z-1346422920000@fe80:0:0:0:215:17ff:fedc:562c%2515.ics"
BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Open-Xchange//7.10.6-Rev49//EN
BEGIN:VTIMEZONE
TZID:Europe/Rome
TZURL:http://tzurl.org/zoneinfo-outlook/Europe/Rome
X-LIC-LOCATION:Europe/Rome
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:19700329T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:19701025T030000
RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:19700101T000000Z
ATTENDEE;PARTSTAT=NEEDS-ACTION;CUTYPE=INDIVIDUAL;EMAIL=****@***.com:mailto:***@****.com
ATTENDEE;CN=****;PARTSTAT=ACCEPTED;CUTYPE=INDIVIDUAL;EMAIL=***@***.org:mailto:***@***.org
CLASS:PUBLIC
CREATED:20171106T161958Z
DESCRIPTION:****
DTEND;TZID=Europe/Rome:20120831T193500
DTSTART;TZID=Europe/Rome:20120831T162200
LAST-MODIFIED:20171106T161958Z
LOCATION:Bologna Centrale
ORGANIZER;EMAIL=webmaster@trenitalia.it:mailto:webmaster@trenitalia.it
SEQUENCE:0
SUMMARY:*** 
TRANSP:OPAQUE
UID:20120830T181649Z-1346422920000@fe80:0:0:0:215:17ff:fedc:562c%15
END:VEVENT
END:VCALENDAR
WhyNotHugo commented 1 year ago

Thanks for testing this out with davcli. This is some useful information! :)

For the record, the output of davcli is un-escaped. I.e.: the following is the un-escaped path:

/caldav/Y2FsOi8vMC8zMQ/20120830T181649Z-1346422920000@fe80:0:0:0:215:17ff:fedc:562c%15.ics

You shouldn't need to escape the value that you provide for the get command -- there's some code that explicitly handles escaping it. Apparently you've found a bug.

What's the output of git rev-parse HEAD? I want to make sure that you're not trying an old version before I start further debugging this.

edigiacomo commented 1 year ago

What's the output of git rev-parse HEAD? I want to make sure that you're not trying an old version before I start further debugging this.

$ git rev-parse HEAD
0ef2830c23ff61b074f4ce0de8f04810e1dc4d1c
WhyNotHugo commented 1 year ago

@edigiacomo It turns out the bug was in davcli (the command line interface itself), not in libdav (the actual caldav implementation).

Indeed, URLs with % work fine with the write. If you want to re-do the above tests with the latest version davcli, they should work.

edigiacomo commented 1 year ago

Thanks @WhyNotHugo, I confim that the bug is fixed and I was able to run the tests:

$ git rev-parse HEAD
5b416e756b1a950727c754d7be7ba4de1d9047e9

$ target/release/davcli --version
davcli libdav-0.1.1-261-g5b416e7

$ DAVCLI_PASSWORD="$PASSWORD" target/release/davcli caldav --server-url "$SERVER_URL" --username "$USERNAME" list-calendar-components /caldav/Y2FsOi8vMC8zMQ/ > list-calendar-components.txt

$ grep 20120830T181649Z list-calendar-components.txt 
/caldav/Y2FsOi8vMC8zMQ/20120830T181649Z-1346422920000@fe80:0:0:0:215:17ff:fedc:562c%15.ics

# Note: some info are redacted to ***
$ DAVCLI_PASSWORD="$PASSWORD" target/release/davcli caldav --server-url "$SERVER_URL" --username "$USERNAME" get '/caldav/Y2FsOi8vMC8zMQ/20120830T181649Z-1346422920000@fe80:0:0:0:215:17ff:fedc:562c%15.ics'
BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Open-Xchange//7.10.6-Rev52//EN
BEGIN:VTIMEZONE
TZID:Europe/Rome
TZURL:http://tzurl.org/zoneinfo-outlook/Europe/Rome
X-LIC-LOCATION:Europe/Rome
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:19700329T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=-1SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:19701025T030000
RRULE:FREQ=YEARLY;BYMONTH=10;BYDAY=-1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:19700101T000000Z
ATTENDEE;PARTSTAT=NEEDS-ACTION;CUTYPE=INDIVIDUAL;EMAIL=***@***:mailto:***@***
ATTENDEE;CN=Emanuele Di Giacomo;PARTSTAT=ACCEPTED;CUTYPE=INDIVIDUAL;EMAIL=***@***:mailto:***@***
CLASS:PUBLIC
CREATED:20171106T161958Z
DESCRIPTION:***
DTEND;TZID=Europe/Rome:20120831T193500
DTSTART;TZID=Europe/Rome:20120831T162200
LAST-MODIFIED:20171106T161958Z
LOCATION:Bologna Centrale
ORGANIZER;EMAIL=webmaster@trenitalia.it:mailto:webmaster@trenitalia.it
SEQUENCE:0
SUMMARY:***
TRANSP:OPAQUE
UID:20120830T181649Z-1346422920000@fe80:0:0:0:215:17ff:fedc:562c%15
END:VEVENT
END:VCALENDAR
jealcalat commented 1 year ago

@dennis-hamann I believe the fix is just to go to line 505 in dav.py (line may be different in your version, just whatever the error message says) and replace the raise exceptions.NotFoundError(href) with something like dav_logger.warning("Not found: {}".format(href)). That being said, I also had this issue in the past and I think it was fixed in some more recent version. So maybe updating is all you need...

This solved the same problem with google calendar