Shared xDAV (CalDAV/CardDAV) resources are not discoverable

[anatoli26]

Cyrus has an auto-discovery mechanism for CalDAV and CardDAV that works well with user-owned resources, i.e. when a client (like iOS Contacts and Calendar apps) asks about available resources, Cyrus returns the complete list of resources under /dav/calendars/user/<user@domain>/ and /dav/addressbooks/user/<user@domain>/.

However, this functionality doesn't include any shared resources the user has access to. Cyrus does have shared resources for IMAP folders, it's to expect the same behavior for CalDAV and CardDAV resources.

When the client asks for:

PROPFIND /dav/calendars/user/<user@domain>/

<A:propfind xmlns:A="DAV:">
...

The server responds with:

HTTP/1.1 207 Multi-Status

<A:multistatus xmlns:A="DAV:" ...>
  <A:response>
    <A:href>/dav/calendars/user/<user@domain>/</A:href>
    <A:propstat>
...
  </A:response>
  <A:response>
    <A:href>/dav/calendars/user/<user@domain>/Default/</A:href>
    <A:propstat>
      <A:prop>
...

The idea is to include in the returned lists the shared calendars/addressbooks too with the discovery logic based on the IMAP shared folders.

Below goes the initial exchange between the calendar app on iOS 10.2.6 and Cyrus 3.0.5 when the exact URL (/dav/calendars/shared/) for the shared calendar is provided in the advanced settings of the app (the URL finally resets to the user principals folder (/dav/principals/user/t3@domain.com/) as iOS is pointed to it by Cyrus).

---------- t3@domain.com Sun Mar 25 06:05:36 2018

<1521968736<PROPFIND /dav/calendars/shared/ HTTP/1.1
Accept: */*
Content-type: text/xml
Connection: keep-alive
Content-length: 181
Host: mail.domain.com
User-agent: iOS/11.2.6 (15D100) accountsd/1.0
Prefer: return=minimal
Depth: 0
Brief: t
Accept-language: en-us
Authorization: Basic ...
Accept-encoding: br, gzip, deflate

<1521968736<<?xml version="1.0" encoding="UTF-8"?>
<A:propfind xmlns:A="DAV:">
  <A:prop>
    <A:current-user-principal/>
    <A:principal-URL/>
    <A:resourcetype/>
  </A:prop>
</A:propfind>

>1521968736>HTTP/1.1 207 Multi-Status
Date: Sun, 25 Mar 2018 09:05:36 GMT
Strict-Transport-Security: max-age=600
Vary: Accept-Encoding, Brief, Prefer
Preference-Applied: return=minimal
Content-Type: application/xml; charset=utf-8
Content-Length: 546

<?xml version="1.0" encoding="utf-8"?>
<A:multistatus xmlns:A="DAV:" xmlns:C="urn:ietf:params:xml:ns:caldav">
  <A:response>
    <A:href>/dav/calendars/shared/</A:href>
    <A:propstat>
      <A:prop>
        <A:current-user-principal>
          <A:href>/dav/principals/user/t3@domain.com/</A:href>
        </A:current-user-principal>
        <A:resourcetype>
          <A:collection/>
          <C:calendar/>
        </A:resourcetype>
      </A:prop>
      <A:status>HTTP/1.1 200 OK</A:status>
    </A:propstat>
  </A:response>
</A:multistatus>

<1521968736<OPTIONS /dav/principals/user/t3%40domain.com/ HTTP/1.1
Host: mail.domain.com
Connection: keep-alive
Accept: */*
User-Agent: iOS/11.2.6 (15D100) accountsd/1.0
Accept-Language: en-us
Content-Length: 0
Accept-Encoding: br, gzip, deflate

>1521968736>HTTP/1.1 200 OK
Date: Sun, 25 Mar 2018 09:05:36 GMT
Strict-Transport-Security: max-age=600
Cache-Control: no-cache
Link: </dav/principals/.server-info>; rel="server-info"; token="80769c2c66d340ecd178710db26d56b9c4699e3e"
DAV: 1, 2, 3, access-control, extended-mkcol, resource-sharing
DAV: calendar-access, calendar-auto-schedule
DAV: calendar-query-extended, calendar-availability, calendar-managed-attachments
DAV: calendarserver-sharing, inbox-availability
DAV: addressbook
Allow: OPTIONS, GET, HEAD
Allow: PROPFIND, REPORT, COPY
Content-Length: 0

[dilyanpalauzov]

Ideally not only the /dav/calendars/shared/ namespace shall be queried, but also collections owned by other users, who have shared some access with the current principal. This is not entirely clear from the description above.

How is the search supposed to work at the end?

Shall the CARDDAV:addressbook-home-set property return (a root with) all address books, where the user has access (shared and of other users) and a REPORT with CARDDAV:addressbook-home-set returns then a partial list of of all AB-collections, where the user has access? The RFC says former is supposed only to return collections owned by the principal.

[dilyanpalauzov]

When the client asks for: PROPFIND /dav/calendars/user/user\@domain/

One of the questions is how the client comes to this URL for any server software if the user provides only a username@domain.

The returned list above cannot include the shared resources. It depends on the PROPFIND parameters what to return and you have not posted what PROFPIND has exactly asked for.

[dilyanpalauzov]

Here a first try. When the collections within calendar-home-set or addressbook-home-set are returned, the code inserts references to all collections, where the user has access to, including collections to other users. I make some experiments and it works. This might not be the wisest way to implement the feature, it might have side/surprizing effects, but for my case it seems to work.

home-set.txt

[dilyanpalauzov]

I am not sure how to approach this at protocol level. IMAP distinguishes between this user, 'Other users' and 'Shared folders', so it is possible to get shared data not linked to any other user.

CalDAV and CardDAV also have the concept of the current user/principal (calendar-home-set, addressbook-home-set) and other users. Collections of other users can be searched with DAV:principal-property-search. But there is nothing like "shared collection" that does not belong to anybody.

One possibility is to tweak the home-sets to return all collections, to which the user has access to. However this can mean a lot of calendars cluttering the namespace and is not necessary good.

Another posibility is keep as it is and ask the users to search with DAV:principal-property-search for "shared" calendars explicitly, as if "shared" (or something else) was a regular user on the system.

[anatoli26]

I guess the first approach (i.e. to return all collections) is better. If there's a lot of calendars users have access to, that's it, they have a lot of calendars.

Thinking about deployments with lots of users IMO it's better to simplify things for users, they have one user/pass and they use it to access all needed resources. It's easier to indicate the main calendar with a reasonable name, than to explain how to search for something hidden.

[dilyanpalauzov]

It would have been simpler, if clients supported DAV:principal-property-search. This way it would be easier to evaluate, what would be better to return in the home-sets. So the approach shall be to teach clients about DAV:principal-property-search rather than tweaking the server.

Finally a shared calendar could be handled as regular calendar user - e.g. a group.

The patch I provided above returns in the home-set all collections, where the user has access to (kind of).

[dilyanpalauzov]

Clients need to have also the notion, that the calendar-home-set can return calendars belonging to several users.

If one user has shared a calendar collection with DAV:displayname "XYZ" and CALDAV:calendar-description of "ABC", and another user has shared another calendar with the properties of the same value, to the users querying calendar-home-set will show two "XYZ / ABC" and the poor users cannot understand the difference. So it is a matter of adjusting the clients to display in the calendar-home-set results not only the DAV:displayname and CALDAV:calendar-description but also somehow the ownership of the calendar, so that users can distinguisch also based on this criterion.

[anatoli26]

If one user has shared a calendar collection with DAV:displayname "XYZ" and CALDAV:calendar-description of "ABC", and another user has shared another calendar with the properties of the same value, to the users querying calendar-home-set will show two "XYZ / ABC" and the poor users cannot understand the difference.

Maybe to add a description on the server to the shared calendars? Like: shared (john.doe): MySharedCalendar ("MySharedCalendar" being the calendar's name given by it's owner, john.doe@example.com)

Edit: this name would go in the DAV:displayname property

[anatoli26]

And a global calendar (without an owner, created in /dav/calendars/ namespace, e.g. /dav/calendars/GlobalCalendar) could have a name like this: shared (global): GlobalCalendar or shared: GlobalCalendar (I would prefer the later, it's simpler)

[karagian]

@dilyanpalauzov I tried your patch on cyrus 3.0.8 but doesn't seem to work. I create a shared calendar, edited ACLs so the user has access to it, used another client as before (davdroid) to access the shared calendar and successfully added some events to it, and then tried to set up the account on an iPad. iPad still only shows the personal addressbooks. The shared calendar is not available. Am I doing something wrong?

[dilyanpalauzov]

When the calendar has rl right both DavDroid and Evolution find it.

I don't know how exactly iPad works. Perhaps you can provide server-side logs including headers and excluding private information.

[karagian]

I haven't tested Evolution at all, but DavDroid's behavior is exactly the same as without the patch applied: If the account is setup using the principal URL, only the personal calendars can be accessed. When using a shared calendar URL instead, that specific shared calendar is available along with all personal calendars. But that's the way it used to work anyway. The patch didn't change anything.

As for iOS I did the tests again and I attach the logs here. I did 2 tests, the first time using the principal url and the second using the shared calendar url. Note that the second test cannot be done without applying this patch first https://github.com/cyrusimap/cyrus-imapd/pull/2307/commits/7644e0eb2c7ec94fbf403446135022060099ab4d because httpd crashes due to a bug.

Hope this helps. ios_shared_calendar_url.txt ios_principal_url.txt

[karagian]

ok, I just realized that the changes on the patch probably only affect addressbook collections and not calendars? I'll do a test with a shared addressbook and get back with the results...

[karagian]

Done the test with a shared addressbook this time, but the same result. I used the shared addressbook URL to setup the account. Only contacts on personal addressbooks are available.

ios_shared_addressbook_url.txt

[dilyanpalauzov]

The logs you sent don't contain the HTTP body.

I have configured my server according to RFC6764, so in Evolution and DavDroid I only enter the email address and the rest happens in the background. You start with an /dav/addessbooks/Shared, try instead starting with /dav/calendars or /dav/addressbooks . Nevertheless, when I start in DavDroid with a user name, the /dav/principals/user URL and a password I see the shared address books and calendars - the ones shown in cyradm as user/GGG/#calendars/ZZZ with the lr9 right.

[karagian]

Oh, I see. It does work the way you describe it. Other users' calendars or addressbooks are properly discoverable from DavDroid and even iOS! That's great! But besides these collections, there can also be addressbooks and calendars in the shared namespace. That's where /dav/addressbooks/Shared url comes from. There is a collection created on the server outside the users' namespace just like shared imap folders. Here's the mailbox list on my test server:

ctl_mboxlist -d
#addressbooks   64 default anyone       lrs
#addressbooks.Shared    64 default anyone       lrs     karagian        lrswipkxtecdan789
#calendars      32 default anyone       lrs9
#calendars.Public Calendar      32 default anyone       lrs9    karagian        lrswipkxtecd789
#calendars.Shared       32 default anyone       lrs9    karagian        lrswipkxtecdan789
user.karagian   0 default karagian      lrswipkxtecdan
user.karagian.#addressbooks     64 default karagian     lrswipkxtecdan
user.karagian.#addressbooks.Default     64 default karagian     lrswipkxtecdan
user.karagian.#calendars        32 default karagian     lrswipkxtecdan
user.karagian.#calendars.Attachments    128 default karagian    lrswipkxtecdan  anyone  r
user.karagian.#calendars.Default        32 default karagian     lrswipkxtecdan
user.karagian.#calendars.Inbox  32 default karagian     lrswipkxtecdan
user.karagian.#calendars.Outbox 32 default karagian     lrswipkxtecdan
user.karagian.#notifications    128 default karagian    lrswipkxtecdan
user.karagian.Sent      0 default karagian      lrswipkxtecdan
user.karagian2  0 default karagian2     lrswipkxtecdan
user.karagian2.#addressbooks    64 default karagian2    lrswipkxtecdan
user.karagian2.#addressbooks.Default    64 default karagian2    lrswipkxtecdan
user.karagian2.#addressbooks.kara2shared        64 default karagian2    lrswipkxtecdan  karagian        lrswipkxtecdan789
user.karagian2.#calendars       32 default karagian2    lrswipkxtecdan
user.karagian2.#calendars.Default       32 default karagian2    lrswipkxtecdan
user.karagian2.#calendars.Inbox 32 default karagian2    lrswipkxtecdan
user.karagian2.#calendars.Outbox        32 default karagian2    lrswipkxtecdan
user.karagian2.#calendars.sharedcalkara2        32 default karagian2    lrswipkxtecdan789       karagian        lrswipkxtecdan789

With your patch, user karagian2's shared addressbooks and calendars are now properly discoverable by user karagian who has access to them. But shared calendars in the shared namespace as shown above are not. Maybe you could tweak the patch to also include these too? That would be perfect.

[dilyanpalauzov]

How do you create a calendar in the shared namespace and how do you create the #calendars mailbox?

[karagian]

You must use an admin account to login to the server and then issue commands like these:

0000 CREATE #calendars (TYPE CALENDAR)
0001 CREATE #addressbooks (TYPE ADDRESSBOOK)

0002 CREATE #calendars/Shared (TYPE CALENDAR)
0003 CREATE #addressbooks/Shared (TYPE ADDRESSBOOK)

Just use . as separator instead of / if using unixhierarchysep:0

[anatoli26]

Yeah, as @karagian said, you use an undocumented feature to create a global shared folder with a specific type. You can use these commands from admin IMAP (imtest -a admin; instead of /shared there could go any name):

x CREATE #calendars/shared (TYPE CALENDAR) x CREATE #addressbooks/shared (TYPE ADDRESSBOOK)

This one creates a global shared WebDAV folder: x CREATE #drive/shared (TYPE COLLECTION)

Then you should set permissions on these folders with cyradm as on any other folder of the corresponding type.

This was discussed at info-cyrus mailing list, see here the archive: https://lists.andrew.cmu.edu/pipermail/info-cyrus/2018-March/040037.html

[dilyanpalauzov]

home-set-2.txt

The attached patch lists under calendar-home-set the URL /dav/calendars/user/ggg, as before, and under the latter URL it includes as sub-collections all collections of the form #calendars[hierarchy-separator], including collections not belonging to any user, where the user has access to. Likewise for addressbook-home-set.

I do not pretend that the way I do it is the best possible approach, as I neither understand the mailbox code very good, nor am I sure that at protocol level this is very wisely done (e.g. the shared namespace could be returned as a separate URL in the calendar-home-set, rather than pretending that /dav/calendars/shared is contained in /dav/calendars/user/ggg; REPORTs towards the /dav/calendars/user/ggg will likely not return objects in other collections that appear as its children).

Another approach is to return in addressbook-home-set and calendar-home-set all mailboxes ending in #addressbook or #calender that are reable by the user and then expect from the client to iterate over the returned HREFs. In this case to have mailbox magically discovered by the CalDAV/CardDAV clients, both the user/ggg/#addressbook and the user/ggg/#addressbook/xyz need to be reable by the user.

Nevertheless, from users' perspective the provided patch does what it is expected to do and lists the mailboxes in the shared #addressbook/#calendars namespace.

[karagian]

@dilyanpalauzov your patch works like a charm! I tested DavDroid and iOS and both offer all shared calendars and addressbooks the user has access to. This is so great I think until @ksmurchison finds some time to deal with this issue properly, this is a perfect workaround. From the users' perspective they should not experience any difference. Thank you so much!

[dilyanpalauzov]

You likely want also to update somehow the displayname and description of the shared resources.

[dilyanpalauzov]

Today I think the way to go is to unveil other user's home-set on DAV:principal-property-search and show only the ownerless/shared collections as part of the current user home-set, providing that the current user/principal and the shared stuff are in the same domain. But: the clients I use don't know about DAV:principal-property-search, I have only single-domain setup and with the ownerless collections showing users' collections by default is not essential.

That said at protocol level CARDDAV:addressbook-home-set/CALDAV:calendar-home-set should additionally return /dav/addressbooks@domain , it the address exists and contains readable collections and that's it.

[MatthiasPetermann]

Hello all,

I'm porting Cyrus IMAP 3.6 to NetBSD and was just wondering if the local patch we had in there so far in 3.0.x for this issue is still needed. Looking at this ticket, there seems to be no final decision on this issue. Am I correct in this assessment? Currently, as I said before, I use the patch suggested by Дилян Палаузов locally and have been running it well for over three years in a small production system. Overall, CalDAV support in Cyrus is doing very well with it, so it would certainly be worth extending that further.

Kind regards Matthias

[ksmurchison]

I need to do some more research with my calendaring colleagues. While Dilyan's patch will work, its not ideal.

There actually is a protocol for sharing user calendars, and its an invite/accept process. When a sharee accepts an invite, the calendar is added to the user's subscription.db and mapped into the user's calendar-home-set.

I'd like to find a way for a user to opt-in to calendars in a shared namespace, rather than forcing them to see all of them. Perhaps invite could be sent to all users with access upon creation.

[dilyanpalauzov]

I would say I do not use this patch myself anymore. My current practice is to use only collections belonging to some (natural or imaginary) user. By modifying the ACLs on the -home-sets (#calendars/ directories), I control who can detect the (extra) home-sets. Once that homesets are detected, they are iterated and the calendars are extracted and shown to the user.

As far as I remember the above patch presents collections from the shared (no-username) namespace in the individual -home-sets. There were some problems with this approach, but I do not remember what they were. From user’s perspective there is nearly no difference between both approaches.

[ksmurchison]

Do you have a patch for Cyrus which iterates through all home-sets and presents all accessible calendars to the user, or are you advertising multiple URLs in CALDAV:calendar-home-set property and the client is doing the iteration on its own?

[dilyanpalauzov]

I am advertising multiple URLs in CALDAV:calendar-home-set property and the client is doing the iteration in each home-set on its own. Likewise for CARDDAV.

[ksmurchison]

Which client(s) does this work with?

[dilyanpalauzov]

I mean to use a shared collections, the user first need lr access to the home-set of the sharee, and in addition needs lr access to each shared calendar.

Which clients? DAVx5 and Gnome Evoltuon work. CardDAV also recently started working with more than one addressbook-home-sets. If you want to test, I can provide you a setup.

[dilyanpalauzov]

Demo: user aaa@aegee.org , password abc . That user shall see the home-sets of other users. Server: mail.aegee.org and webmail.aegee.org. The former server does not return 401, if no username is submitted, the latter does always return 401, if no username is submitted.

[sdalu]

@dilyanpalauzov I tested the patch against 3.8, works fine in discovering shared calendars (using android DAVx). But, I don't know if it is related, calendar entries are not visible to others.