vdirsyncer finds no contacts in Zimbra address book

[davidfraser]

I can successfully synchronize contacts with Zimbra using CardDAV with various other programs (CardDAVSync for Android, cadaver, webdav-sync), but vdirsyncer doesn't find any contacts.

Config (actual email, server etc modified):

[general]
status_path = ~/.vdirsyncer/status/
processes = 1

[pair work_contacts]
a = work_contacts_local
b = work_contacts_remote

conflict_resolution = b wins

[storage work_contacts_local]
type = filesystem
path = ~/work/contacts/Contacts/
fileext = .vcf
create = True
encoding = utf-8

[storage work_contacts_remote]
type = carddav
url = https://zimbra.hostname.com/dav/user@server.com/Contacts/
auth = basic
username = user@server.com

Logs (likewise redacted):

debug: Using 1 processes.
debug: Not using multiprocessing.
Server password for user@server.com at the resource https://zimbra.hostname.com/dav/user@server.com/Contacts/: Syncing work_contacts
debug: OPTIONS https://zimbra.hostname.com/dav/user@server.com/Contacts/
debug: {'Depth': 1, 'Content-Type': 'application/xml; charset=UTF-8', 'User-Agent': 'vdirsyncer'}
debug: None
debug: Sending request...
debug: 200
debug: {'date': 'Thu, 03 Jul 2014 13:25:28 GMT', 'dav': '1, 2, 3, access-control, calendar-access, calendar-schedule, version-control, addressbook, extended-mkcol, calendar-proxy, calendarserver-principal-property-search', 'content-length': '0', 'allow': 'MKCOL, REPORT, MOVE, COPY, HEAD, POST, GET, DELETE, ACL, MKCALENDAR, PROPFIND, LOCK, OPTIONS, UNLOCK, PUT, PROPPATCH'}
debug: 
debug: REPORT https://zimbra.hostname.com/dav/user@server.com/Contacts/
debug: {'Depth': 'infinity', 'Content-Type': 'application/xml; charset=UTF-8', 'User-Agent': 'vdirsyncer'}
debug: <?xml version="1.0" encoding="utf-8" ?>
            <C:addressbook-query xmlns:D="DAV:"
                    xmlns:C="urn:ietf:params:xml:ns:carddav">
                <D:prop>
                    <D:getetag/>
                </D:prop>
                <C:filter>
                    <C:comp-filter name="VCARD"/>
                </C:filter>
            </C:addressbook-query>
debug: Sending request...
debug: 207
debug: {'date': 'Thu, 03 Jul 2014 13:25:28 GMT', 'transfer-encoding': 'chunked', 'dav': '1, 2, 3, calendar-access, calendar-proxy, calendarserver-principal-property-search, access-control, addressbook', 'content-type': 'text/xml;charset=UTF-8'}
debug: <?xml version="1.0" encoding="UTF-8"?>

<D:multistatus xmlns:D="DAV:"/>

[untitaker]

Is this URL directly pointing to an addressbook, or do you have multiple ones?

[davidfraser]

Directly to an address book, but I've also tried specifying a collection with one item, and multiple items, and using from b for the collection

[untitaker]

Not sure what is happening here. Can you try https://github.com/untitaker/vdirsyncer/pull/84 and see if it fixes your problem?

pip2 install --user --force-reinstall git+https://github.com/untitaker/vdirsyncer.git#zimbra

[davidfraser]

Nope, sorry :( 500 error for that request:

debug: Sending request...
debug: 200
debug: {'date': 'Thu, 03 Jul 2014 13:46:13 GMT', 'dav': '1, 2, 3, access-control, calendar-access, calendar-schedule, version-control, addressbook, extended-mkcol, calendar-proxy, calendarserver-principal-property-search', 'content-length': '0', 'allow': 'MKCOL, REPORT, MOVE, COPY, HEAD, POST, GET, DELETE, ACL, MKCALENDAR, PROPFIND, LOCK, OPTIONS, UNLOCK, PUT, PROPPATCH'}
debug: 
debug: REPORT https://zimbra.sjsoft.com/dav/davidf@j5int.com/Contacts/
debug: {'Depth': 'infinity', 'Content-Type': 'application/xml; charset=UTF-8', 'User-Agent': 'vdirsyncer'}
debug: <?xml version="1.0" encoding="utf-8" ?>
            <C:addressbook-query xmlns:D="DAV:"
                    xmlns:C="urn:ietf:params:xml:ns:carddav">
                <D:prop>
                    <D:getetag/>
                </D:prop>
                <C:filter>
                </C:filter>
            </C:addressbook-query>
debug: Sending request...
debug: 500
debug: {'date': 'Thu, 03 Jul 2014 13:46:13 GMT', 'content-length': '0'}
debug: 

traceback:

Traceback (most recent call last):
  File "vdirsyncer/vdirsyncer/cli.py", line 333, in sync_collection
    force_delete=status_name in force_delete
  File "vdirsyncer/vdirsyncer/sync.py", line 129, in sync
    list_b = prepare_list(storage_b, b_href_to_status)
  File "vdirsyncer/vdirsyncer/sync.py", line 66, in prepare_list
    for href, etag in storage.list():
  File "vdirsyncer/vdirsyncer/storage/dav.py", line 417, in _list
    headers=headers
  File "vdirsyncer/vdirsyncer/storage/dav.py", line 203, in request
    session=self._session, **self._settings)
  File "vdirsyncer/vdirsyncer/utils/__init__.py", line 213, in request
    r.raise_for_status()
  File "requests/models.py", line 795, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 500 Server Error: Server Error

[untitaker]

Maybe completely removing the filter tag works then? I think i need to get my hands on a copy of Zimbra to integrate into the testing environment.

[davidfraser]

Zimbra is very heavy ... I don't think you want to install it just for testing this.

It seems very strange that Zimbra is returning this empty set, but on the other hand the other clients work with it... I did some debugging and comparison with pyCardDav (which works) - they do a general PROPFIND with Accept: / and Depth: 1 and that seems to return items if I put it into vdirsyncer, rather than doing an addressbook-query. But that's strange, and I can't see what's missing from our addressbook-query and Zimbra's supposed to support it. :(

[untitaker]

@dmfs Is the behavior with PROPFIND described above implemented because of Zimbra?

[davidfraser]

http://www.ietf.org/mail-archive/web/vcarddav/current/msg01649.html seems to indicate that Zimbra may behave like this (but no replies)

[dmfs]

CardDAV-Sync for Android doesn't use addressbook-query either. I didn't see any benefits in using it. We either use a regular PROPFIND or a sync-collection REPORT if supported. In contrast to calendar-query it usually doesn't make any sense to filter the elements in an address book. As of RFC 6352 an address book collection supports only address resources. Of course that includes other formats than VCard, but I've not seen any server that actually supports other formats. We filter the elements by content-type after we received the list.

[untitaker]

@davidfraser I saw your fork contained a commit re duplicate hrefs. Is it still relevant?

[untitaker]

Can you try again with the latest commit on the zimbra branch, @davidfraser?

[davidfraser]

The latest commit works, but I had to set Depth to 1 otherwise I got a 403 Forbidden with a prop-finite-depth error. Thanks!

The commit re duplicate hrefs was for Zimbra calendar - haven't investigated more but got lots of these (I wonder if they're for recurring appointments?)

[untitaker]

Unfortunately it seems like we can't merge this anytime soon, since it seems to break Radicale.

[untitaker]

@dmfs, i think CardDAV sync for Android then also breaks for Radicale-git with database backend.

[dmfs]

We don't have any reports about problems so far. What's the problem? Does it return an error? How does the request you send look like? How does the response look like?

[untitaker]

See https://travis-ci.org/untitaker/vdirsyncer/jobs/29148128. Radicale simply crashes for reasons i am not even trying to comprehend.

[dmfs]

What was the content of the last request? If I read the log correctly it was empty:

debug: PROPFIND http://127.0.0.1/bob/test.vcf/
debug: {'Depth': 1, 'Content-Type': 'application/xml; charset=UTF-8', 'User-Agent': 'vdirsyncer'}
debug: None

I think that's what None indicates here.

[untitaker]

Yes, precisely. As far as i can see that's what pycarddav does. I'll reread the RFC.

[dmfs]

What's the purpose of an empty PROPFIND? Some servers probably just return a default multistatus response (like an index) but that's not specified and an empty PROPFIND is not valid, afaik.

[geier]

I'll have to admit that I didn't read the RFC properly when implementing this in pycarddav. I poked around a bit and stopped when it seemed to work with all CardDAV servers I had access to.

[geier]

BTW the RFC says:

A client may choose not to submit a request body. An empty PROPFIND request body must be treated as a request for the names and values of all properties.

I'm on the phone, so cannot properly look into this atm.

[untitaker]

Just pushed a fix to the zimbra branch, now using a proper request body for the PROPFIND. @davidfraser Can you try this one too, just to make sure?

[davidfraser]

@unittaker that fixes the synchronization, thanks! I have a problem remaining with address books whose name has a space in them - but will file a separate ticket for that in due course

[untitaker]

Done. Please file a new issue for the addressbook-issue and eventually also re duplicate hrefs.

[untitaker]

Just released 0.2.1.