search

pimutils / vdirsyncer

📇 Synchronize calendars and contacts.
https://vdirsyncer.pimutils.org/
Other
1.53k stars 160 forks source link

sync against davmail results in raise exceptions.NotFoundError(href) #214

Closed divansantana closed 9 years ago

divansantana commented 9 years ago

Hi,

Attempting to sync against davmail and exchange get the below error. I've emailed you the full debug output of vdirsyncer2 --verbosity=DEBUG sync

This is on Arch Linux with below versions:

davmail 4.6.1-1
python2-vdirsyncer 0.5.1-1

Not sure if I'm doing something wrong though using the same settings in kontact and that manages to fetch the calendar fine.

My config is like so:

[general]
status_path = ~/.vdirsyncer/status/

[pair fnb_calendar]
a = fnb_calendar_local
b = fnb_calendar_remote

[storage fnb_calendar_local]
type = filesystem
path = ~/.calendars/fnb/
fileext = .ics

[storage fnb_calendar_remote]
type = caldav
url =  http://localhost:1080/users/blah@example.co.za/calendar/
username = something
password = blah

debug: </C:calendar-data><D:getetag>2009-01-14T08:46:00Z</D:getetag></D:prop><D:status>HTTP/1.1 200 OK</D:status></D:propstat></D:response><D:response><D:href>/users/blah@example.co.za/calendar/AAMkADFhNGYwNzJkLTE2YjEtNDA0ZC04YmQ2LWQ3YTdiOGYwY2Y2NABGAAAAAABWxfZAMOs7Q4UNDhZ5a505BwCqyWteSHs-RJcaDYcJcinrAAASxFepAAAlZPQvrhafTpQ54lmBac4TAAAE4kcaAAA=.EML</D:href><D:propstat><D:status>HTTP/1.1 404 Not Found</D:status></D:propstat></D:response></D:multistatus>
warning: Partially invalid XML response, some of your items may be corrupted. Check the debug log and consider switching servers. (<string>:63487:358:FATAL:PARSER:ERR_TAG_NAME_MISMATCH: Opening and ending tag mismatch: prop line 41952 and multistatus
warning: <string>:63487:358:FATAL:PARSER:ERR_TAG_NOT_FINISHED: Premature end of data in tag propstat line 41952
warning: <string>:63487:358:FATAL:PARSER:ERR_TAG_NOT_FINISHED: Premature end of data in tag response line 41952
warning: <string>:63487:358:FATAL:PARSER:ERR_TAG_NOT_FINISHED: Premature end of data in tag multistatus line 1)
warning: Skipping u'/users/blah@example.co.za/calendar/AAMkADFhNGYwNzJkLTE2YjEtNDA0ZC04YmQ2LWQ3YTdiOGYwY2Y2NABGAAAAAABWxfZAMOs7Q4UNDhZ5a505BwCqyWteSHs-RJcaDYcJcinrAAASxFepAAAlZPQvrhafTpQ54lmBac4TAAAE4kcaAAA=.EML', etag property is missing.
warning: Skipping /users/blah@example.co.za/calendar/AAMkADFhNGYwNzJkLTE2YjEtNDA0ZC04YmQ2LWQ3YTdiOGYwY2Y2NABGAAAAAABWxfZAMOs7Q4UNDhZ5a505BwCqyWteSHs-RJcaDYcJcinrAAASxFepAAAlZPQvrhafTpQ54lmBac4TAAAE4kcaAAA=.EML, the item content is missing.
warning: Skipping identical href: u'/users/blah@example.co.za/calendar/AAMkADFhNGYwNzJkLTE2YjEtNDA0ZC04YmQ2LWQ3YTdiOGYwY2Y2NABGAAAAAABWxfZAMOs7Q4UNDhZ5a505BwCqyWteSHs-RJcaDYcJcinrAAASxFepAAAlZPQvrhafTpQ54lmBac4TAAAE4kcaAAA=.EML'
Unhandled exception occured for fnb_calendar.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdirsyncer/cli/tasks.py", line 77, in sync_collection
    force_delete=force_delete
  File "/usr/lib/python2.7/site-packages/vdirsyncer/sync.py", line 163, in sync
    b_info.prepare_idents(storage_a.read_only)
  File "/usr/lib/python2.7/site-packages/vdirsyncer/sync.py", line 114, in prepare_idents
    hrefs_to_download else ()):
  File "/usr/lib/python2.7/site-packages/vdirsyncer/storage/dav.py", line 429, in get_multi
    raise exceptions.NotFoundError(href)
NotFoundError: /users/blah@example.co.za/calendar/AAMkADFhNGYwNzJkLTE2YjEtNDA0ZC04YmQ2LWQ3YTdiOGYwY2Y2NABGAAAAAABWxfZAMOs7Q4UNDhZ5a505BwCqyWteSHs-RJcaDYcJcinrAAASxFepAAAlZPQvrhafTpQ54lmBac4TAAAE4kcaAAA=.EML
untitaker commented 9 years ago

Hello @divansantana,

I've replied to your mail.

Technical info/inner monologue: This doesn't seem to be an encoding problem by vdirsyncer (unlike previous ones). DavMail sends us this for our PROPFIND:

<D:response>
    <D:href>/users/dsantana@fnb.co.za/calendar/AAMkADFhNGYwNzJkLTE2YjEtNDA0ZC04YmQ2LWQ3YTdiOGYwY2Y2NABGAAAAAABWxfZAMOs7Q4UNDhZ5a505BwCqyWteSHs-RJcaDYcJcinrAAASxFepAAAlZPQvrhafTpQ54lmBac4TAAAE4kcaAAA%3D.EML</D:href>
    <D:propstat>
        <D:prop>
            <D:getcontenttype>text/calendar; component=vevent</D:getcontenttype>
            <D:getetag>2013-02-18T05:33:13Z</D:getetag>
            <D:resourcetype/>
        </D:prop>
        <D:status>HTTP/1.1 200 OK</D:status>
    </D:propstat>
</D:response>

We then send this tag inside a multiget query:

<D:href>/users/dsantana@fnb.co.za/calendar/AAMkADFhNGYwNzJkLTE2YjEtNDA0ZC04YmQ2LWQ3YTdiOGYwY2Y2NABGAAAAAABWxfZAMOs7Q4UNDhZ5a505BwCqyWteSHs-RJcaDYcJcinrAAASxFepAAAlZPQvrhafTpQ54lmBac4TAAAE4kcaAAA%3D.EML</D:href>

DavDroid sends this:

[...]
</D:response>
<D:response>
    <D:href>/users/dsantana@fnb.co.za/calendar/AAMkADFhNGYwNzJkLTE2YjEtNDA0ZC04YmQ2LWQ3YTdiOGYwY2Y2NABGAAAAAABWxfZAMOs7Q4UNDhZ5a505BwCqyWteSHs-RJcaDYcJcinrAAASxFepAAAlZPQvrhafTpQ54lmBac4TAAAE4kcaAAA%3D.EML</D:href>
    <D:propstat>
        <D:prop>
        <D:response>
            <D:href>/users/dsantana@fnb.co.za/calendar/isnjuf63v3fbv4phiad7iui8d8.ics</D:href>
            <D:propstat>
            <D:prop>
                <C:calendar-data xmlns:C="urn:ietf:params:xml:ns:caldav" C:content-type="text/calendar" C:version="2.0">[...]

The response tag for AAMkADFhNGY[...].ics is never finished, after the calendar-data, the response content continues "normally" until the end of the multistatus:

<D:response>
    <D:href>/users/dsantana@fnb.co.za/calendar/AAMkADFhNGYwNzJkLTE2YjEtNDA0ZC04YmQ2LWQ3YTdiOGYwY2Y2NABGAAAAAABWxfZAMOs7Q4UNDhZ5a505BwCqyWteSHs-RJcaDYcJcinrAAASxFepAAAlZPQvrhafTpQ54lmBac4TAAAE4kcaAAA=.EML</D:href>
    <D:propstat>
        <D:status>HTTP/1.1 404 Not Found</D:status>
    </D:propstat>
</D:response>
</D:multistatus>

Which explains all warnings.

Here's my reply to the email:

Are you able to download                                                                      

    http://localhost:1080/users/dsantana@fnb.co.za/calendar/AAMkADFhNGYwNzJkLTE2YjEtNDA0ZC04Ym
Q2LWQ3YTdiOGYwY2Y2NABGAAAAAABWxfZAMOs7Q4UNDhZ5a505BwCqyWteSHs-RJcaDYcJcinrAAASxFepAAAlZPQvrhaf
TpQ54lmBac4TAAAE4kcaAAA=.EML                                                                  

with your browser or cURL (given the proper username and password)? Does it                   
contain any weird characters (I'm thinking of both nullbytes and unescaped XML                
tags) that might trip DavMail up?
divansantana commented 9 years ago

Thanks for the super quick response. Output below. Does this mean it's a bug in davmail?

With additional verbosity:
   ~   curl -vvv -u 'username:password' http://localhost:1080/users/dsantana@fnb.co.za/calendar/AAMkADFhNGYwNzJkLTE2YjEtNDA0ZC04YmQ2LWQ3YTdiOGYwY2Y2NABGAAAAAABWxfZAMOs7Q4UNDhZ5a505BwCqyWteSHs-RJcaDYcJcinrAAASxFepAAAlZPQvrhafTpQ54lmBac4TAAAE4kcaAAA\=.EML
*   Trying ::1...
* Connected to localhost (::1) port 1080 (#0)
* Server auth using Basic with user 'username'
> GET /users/dsantana@fnb.co.za/calendar/AAMkADFhNGYwNzJkLTE2YjEtNDA0ZC04YmQ2LWQ3YTdiOGYwY2Y2NABGAAAAAABWxfZAMOs7Q4UNDhZ5a505BwCqyWteSHs-RJcaDYcJcinrAAASxFepAAAlZPQvrhafTpQ54lmBac4TAAAE4kcaAAA=.EML HTTP/1.1
> Host: localhost:1080
> Authorization: Basic Zm5iam5iMDFcZjMzMzMwMDA6MFdoYXRTaGFsbE15UGIyZGF5Pw==
> User-Agent: curl/7.42.1
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< Server: DavMail Gateway 4.6.1-2343
< DAV: 1, calendar-access, calendar-schedule, calendarserver-private-events, addressbook
< Date: Sun, 07 Jun 2015 19:10:08 +0000
< Expires: Sun, 07 Jun 2015 19:10:08 +0000
< Cache-Control: private, max-age=0
< Content-Type: text/plain;charset=UTF-8
< Connection: close
< Content-Length: 109
<
* Closing connection 0
Invalid property line: ATTENDEE;CN="an"";PARTSTAT=NEEDS-ACTION;ROLE=REQ-PARTICIPANT:mailto:somebody@somewhere.co.za%                             ~  
untitaker commented 9 years ago

I think there's something wrong with your data too, but DavMail is definetly handling this poorly, yes.

It seems that Microsoft has a tool that is supposed to fix broken events, could you check it out?

https://www.microsoft.com/en-us/download/details.aspx?id=28786

divansantana commented 9 years ago

Wahoo this is working now! :) Thanks!

  1. Used the MS tool to identify which are the problematic events (all really old ones). Searched for them and deleted them.
  2. Attempted a sync again vdirsyncer2 sync which reported another error and advised I run a repair.
  3. Ran the repair, vdirsyncer2 repair fnb_calendar_remote and resync'ed.
  4. All sorted and working nicely now.

Thanks for the help.

untitaker commented 9 years ago

Great! I've added that tool to the documentation.

untitaker commented 9 years ago

Filed bug against DavMail: https://sourceforge.net/p/davmail/bugs/598/