search

signalapp / Flock

Private contact and calendar sync for Android.
https://signal.org/blog/flock
358 stars 80 forks source link

Zimbra, error "Server doesn't CardDAV well-known URI or DAV: current-user-principal" #79

Open Zeriuno opened 9 years ago

Zeriuno commented 9 years ago

Fixing #73 you were right to underline that it wasn't sure Flock was going to support my Zimbra account! It took me a while but I was able to do a few tests and here is the result: Two logs actually, because I am unsure of whether I should sumbit as url $zimbraurl/dav or just $zimbraurl (first one) https://gist.github.com/anonymous/c6e40bc61a679f7628a0 https://gist.github.com/cbdb3beb9b77c134bef6

Thanks for looking at it when you get a little time!

rhodey commented 9 years ago

Thanks for checking back in with results and opening a detailed issue :) I promise to check this out this weekend!

Zeriuno commented 9 years ago

I have to thank you, rhodey, (and a few people at my hosting provider too, actually) for helping troubleshooting these issues with Zimbra. Take all the time you need!

rhodey commented 9 years ago

@Zeriuno looking at these logs now it seems that you're making it even less far in the server tests than before :[ as in the logs from #73. Are you using the same WebDAV host URL for these tests as you were before for #73?

In #73 the situation was:

  1. flock client executes PROPFIND http request
  2. server response with status 302 "FOUND" and redirects

The situation in these two new logs seems to be:

  1. flock client executes PROPFIND http request
  2. server response 404

OR

  1. flock client executes PROPFIND http request
  2. server response 405 "Method PROPFIND is not defined in RFC 2068 and is not supported by the Servlet API"

If you're sure that you're using the same URL for the tests as before then it would seem that the Zimbra server you're trying to work with has since changed its configuration. Here is some curl stuff you can use to help the testing along...

$ curl -k -i -u <username>:<password> -X PROPFIND https://<WebDAV host>/.well-known/carddav --upload-file - -H "Depth: 0" <<end
<?xml version="1.0"?>
<a:propfind xmlns:a="DAV:">
<a:prop><a:current-user-principal/></a:prop>
</a:propfind>
end

The above curl request mimics the first test run by the Flock client which is a test for the current-user-principal property. The trailing /.well-known/carddav is something called a well-known URI and is defined in rfc 6764. Well-known URIs are nice because they make it possible for a WebDAV client to work with a WebDAV server without having to ask the user for a bunch of configuration information.

The correct response to a PROPFIND request on a well-known URI is a redirect to the actual resource. For example on OwnCloud /.well-known/carddav maps to something like .../remote.php/carddav' and/.well-known/caldavmaps to something like.../remote.php/caldav'. Check out the example request and response to one of my testing Darwin Calendar Server instances below...

Request

$ curl -k -i -u coffee00@flock:test -X PROPFIND http://55.55.55.55/.well-known/carddav --upload-file - -H "Depth: 0" <<end
<?xml version="1.0"?>
<a:propfind xmlns:a="DAV:">
<a:prop><a:current-user-principal/></a:prop>
</a:propfind>
end

Response

HTTP/1.1 301 Moved Permanently
Date: Mon, 01 Dec 2014 21:15:02 GMT
Content-Length: 165
Location: http://55.55.55.55/
Content-Type: text/html;charset=utf-8
Server: Twisted/13.2.0 TwistedWeb/9.0.0+r11995

<html><head><title>Moved Permanently</title></head><body><h1>Moved Permanently</h1><p>Document moved to http://55.55.55.55/.</p></body></html>

Darwin Calendar Server really just does WebDAV stuff so there is no special endpoint for CardDAV vs CalDAV requests, this response tells us that for everything CardDAV we can just talk to the root endpoint at http://55.55.55.55/. So now that we know where to send our CardDAV requests we try the current-user-principal PROPFIND request again but this time with the new target set to http://55.55.55.55/

Request

$ curl -k -i -u coffee00@flock:test -X PROPFIND http://55.55.55.55/ --upload-file - -H "Depth: 0" <<end
<?xml version="1.0"?>
<a:propfind xmlns:a="DAV:">
<a:prop><a:current-user-principal/></a:prop>
</a:propfind>
end

Response

HTTP/1.1 207 Multi-Status
Date: Mon, 01 Dec 2014 21:15:33 GMT
ETag: "44925-1000-546E5445"
Accept-Ranges: bytes
Content-Type: text/xml
Last-Modified: Thu, 20 Nov 2014 20:51:17 GMT
Content-Length: 364
Server: Twisted/13.2.0 TwistedWeb/9.0.0+r11995
DAV: 1, access-control, calendar-access, calendar-schedule, calendar-auto-schedule, calendar-availability, inbox-availability, calendarserver-private-events, calendarserver-private-comments, calendar-query-extended, calendar-default-alarms, calendarserver-partstat-changes, calendar-no-timezone, addressbook, extended-mkcol, calendarserver-principal-property-search, calendarserver-principal-search, calendarserver-home-sync
Vary: accept-encoding

<?xml version='1.0' encoding='UTF-8'?>
<multistatus xmlns='DAV:'>
  <response>
    <href>/</href>
    <propstat>
      <prop>
        <current-user-principal>
          <href>/principals/__uids__/COFFEE00%40FLOCK/</href>
        </current-user-principal>
      </prop>
      <status>HTTP/1.1 200 OK</status>
    </propstat>
  </response>
</multistatus

And there ya go, the current-user-principal property we asked for was returned and as expected it is the same as the user we authorized as coffee00@flock. Whenever you're able to get this sequence of requests working on your Zimbra server just input whatever you used for <WebDAV host> into the Flock client and then we'll see where that takes us.

Zeriuno commented 9 years ago

Thank you @rhodey for your clear and detailed explanation: it is really appreciated.

I tried to test http://webmail.web4all.fr/.well-known/carddav and it redirects with a 302 toward the https version of the same url. But the https version returns me a 411

HTTP/1.1 411 Length Required
Server: nginx
Date: Mon, 01 Dec 2014 22:15:03 GMT
Content-Type: text/html
Content-Length: 174
Set-Cookie: SERVERID=zose-proxy02a; path=/

<html>
<head><title>411 Length Required</title></head>
<body bgcolor="white">
<center><h1>411 Length Required</h1></center>
<hr><center>nginx</center>
</body>
</html>

Any suggestion on how to pursue the investigation?

rhodey commented 9 years ago

can you copy + paste the exact curl command you ran for reference? minus the username & password credentials of course.

Zeriuno commented 9 years ago

Of course, here it is:

curl -k -i -u username:password -X PROPFIND http://webmail.web4all.fr/.well-known/carddav --upload-file - -H "Depth: 0" <<end
<?xml version="1.0"?>
<a:propfind xmlns:a="DAV:">
<a:prop><a:current-user-principal/></a:prop>
</a:propfind>
end

The response being:

HTTP/1.1 302 Found
Cache-Control: no-cache
Content-length: 0
Location: https://webmail.web4all.fr/.well-known/carddav
Connection: close

But testing https://webmail.web4all.fr/.well-known/carddav returned a 411, like I said.

rhodey commented 9 years ago

awesome, thanks!

I would expect curl to calculate the data length header itself and include this automatically but to check this try adding the -v option and this will show us exactly what request you're sending. If you could include the output with the -v option I think we'll be able to figure this 411 thing out.

Another possibility is that the Zimbra server is using some really weird SSL or TLS cipher suites, I've ran into this problem with curl before and adding the parameters to negotiate a more sane set of cipher suites was a pain.

Zeriuno commented 9 years ago

Tested with -v but unfortunately it didn't help Here's the request

curl -v -k -i -u username:test -X PROPFIND https://webmail.web4all.fr/.well-known/carddav --upload-file - -H "Depth: 0" <<end
<?xml version="1.0"?>
<a:propfind xmlns:a="DAV:">
<a:prop><a:current-user-principal/></a:prop>
</a:propfind>
end

And here the complete response:

* Adding handle: conn: 0x7f9993003a00
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7f9993003a00) send_pipe: 1, recv_pipe: 0
* About to connect() to webmail.web4all.fr port 443 (#0)
*   Trying 185.49.20.40...
* Connected to webmail.web4all.fr (185.49.20.40) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
* Server certificate: *.web4all.fr
* Server certificate: Gandi Standard SSL CA
* Server certificate: UTN-USERFirst-Hardware
* Server auth using Basic with user 'username'
> PROPFIND /.well-known/carddav HTTP/1.1
> Authorization: Basic dXNlcm5hbWU6dGVzdA==
> User-Agent: curl/7.30.0
> Host: webmail.web4all.fr
> Accept: */*
> Transfer-Encoding: chunked
> Depth: 0
> Expect: 100-continue
>
< HTTP/1.1 411 Length Required
HTTP/1.1 411 Length Required
* Server nginx is not blacklisted
< Server: nginx
Server: nginx
< Date: Mon, 01 Dec 2014 22:39:05 GMT
Date: Mon, 01 Dec 2014 22:39:05 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 174
Content-Length: 174
< Set-Cookie: SERVERID=zose-proxy02a; path=/
Set-Cookie: SERVERID=zose-proxy02a; path=/

* HTTP error before end of send, stop sending
<
<html>
<head><title>411 Length Required</title></head>
<body bgcolor="white">
<center><h1>411 Length Required</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Closing connection 0

I am asking my host if they know about Zimbra TLS cipher suites: I'll post here if I get anything.

rhodey commented 9 years ago

I think these two lines in the HTTP request are our problem:

Transfer-Encoding: chunked
Expect: 100-continue

Our PROPFIND request is actually an XML document so technically we're sending a file to the WebDAV server, curl has decided that it wants to send this file chunked and after receiving an HTTP status 100 response from the server would then begin the transfer of the XML document. But as we can see the server does not return a 100 because for some reason I guess it doesn't feel like processing this request as chunked.

We need to tell curl to send this PROPFIND request in a single http request, then I expect the length will be included and the server will no longer give us a 411. Give me a few minutes to research how to do this in curl, maybe you'll find the answer too.

The Flock client doesn't do this chunked thing, ftr.

rhodey commented 9 years ago

$ man curl

Remove an internal header by giving a replacement without content on the right side of the colon, as in: -H "Host:"

try adding a -H "Expect:" to your curl command, this should remove the Expect: 100 header and hopefully cause curl to avoid sending this as chunked.

Zeriuno commented 9 years ago

Nope :-/

Request:

curl -v -k -i -u user:password -X PROPFIND https://webmail.web4all.fr/.well-known/carddav --upload-file - -H "Depth: 0" -H "Expect:" <<end
<?xml version="1.0"?>
<a:propfind xmlns:a="DAV:">
<a:prop><a:current-user-principal/></a:prop>
</a:propfind>
end

Response

* Adding handle: conn: 0x7ff082803a00
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7ff082803a00) send_pipe: 1, recv_pipe: 0
* About to connect() to webmail.web4all.fr port 443 (#0)
*   Trying 185.49.20.40...
* Connected to webmail.web4all.fr (185.49.20.40) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
* Server certificate: *.web4all.fr
* Server certificate: Gandi Standard SSL CA
* Server certificate: UTN-USERFirst-Hardware
* Server auth using Basic with user 'daniele%40pitrolo.it'
> PROPFIND /.well-known/carddav HTTP/1.1
> Authorization: Basic ZGFuaWVsZSU0MHBpdHJvbG8uaXQ6NGRlc3NvIHNvbm8gYSBjYXNB
> User-Agent: curl/7.30.0
> Host: webmail.web4all.fr
> Accept: */*
> Transfer-Encoding: chunked
> Depth: 0
>
< HTTP/1.1 411 Length Required
HTTP/1.1 411 Length Required
* Server nginx is not blacklisted
< Server: nginx
Server: nginx
< Date: Mon, 01 Dec 2014 23:17:23 GMT
Date: Mon, 01 Dec 2014 23:17:23 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 174
Content-Length: 174
< Set-Cookie: SERVERID=zose-proxy02a; path=/
Set-Cookie: SERVERID=zose-proxy02a; path=/

<
<html>
<head><title>411 Length Required</title></head>
<body bgcolor="white">
<center><h1>411 Length Required</h1></center>
<hr><center>nginx</center>
</body>
</html>
rhodey commented 9 years ago

got it :D, kinda... :|

request

curl -k -i -u <username>:<password> -X PROPFIND https://webmail.web4all.fr/.well-known/carddav --upload-file - -H "Depth: 0" -H "Content-Length: 105" -H "Transfer-Encoding:" -H "Expect:" <<end
<?xml version="1.0"?>
<a:propfind xmlns:a="DAV:">
<a:prop><a:current-user-principal/></a:prop>
</a:propfind>
end

response

HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Mon, 01 Dec 2014 23:28:51 GMT
Content-Length: 0
Location: http://webmail.web4all.fr:80/dav
Set-Cookie: SERVERID=zose-proxy02a; path=/
Cache-control: private

Weird that we get directed from an https resource to an http resource, but with that aside we should be able to execute our PROPFIND request on http://webmail.web4all.fr:80/dav and finally get back the current-user-principal property. but oddly enough this server gives us another redirect to https://webmail.web4all.fr/dav, see below...

request

curl -k -i -u <username>:<password> -X PROPFIND http://webmail.web4all.fr:80/dav --upload-file - -H "Depth: 0" -H "Content-Length: 105" -H "Transfer-Encoding:" -H "Expect:" <<end
<?xml version="1.0"?>
<a:propfind xmlns:a="DAV:">
<a:prop><a:current-user-principal/></a:prop>
</a:propfind>
end

response

HTTP/1.1 302 Found
Cache-Control: no-cache
Content-length: 0
Location: https://webmail.web4all.fr/dav
Connection: close

only then after two redirects can we finally get the current-user-principal property, like so...

request

curl -k -i -u <username>:<password> -X PROPFIND https://webmail.web4all.fr/dav --upload-file - -H "Depth: 0" -H "Content-Length: 105" -H "Transfer-Encoding:" -H "Expect:" <<end
<?xml version="1.0"?>
<a:propfind xmlns:a="DAV:">
<a:prop><a:current-user-principal/></a:prop>
</a:propfind>
end

except because I don't have a valid login on this server I of course get a 401 unauthorized.

response

HTTP/1.1 401 invalid username/password
Server: nginx
Date: Mon, 01 Dec 2014 23:40:07 GMT
Content-Length: 0
WWW-Authenticate: BASIC realm="Zimbra"
Set-Cookie: SERVERID=zose-proxy02a; path=/

In summary this zimbra server is configured kinda weird because we get redirected from an HTTPS resource to an HTTP resource and then back to HTTPS. The correct response to a PROPFIND on https://webmail.web4all.fr/.well-known/carddav is a simple redirect to https://webmail.web4all.fr/dav, this HTTP step in the middle is weird and unfortunately Flock will not follow nested redirects.

If you have any easy way to give your Zimbra server provider a heads up about this weird triple redirect that would probably be best although it is also possible that you could find an endpoint that does this redirect correctly, for example, like maybe https://webmail.web4all.fr/idk-why-this-works would redirect directly to https://webmail.web4all.fr/dav and skip this silly HTTP middle step.

Zeriuno commented 9 years ago

Ok, the redirection towards http:/webmail.web4all.fr:80/dav/ confirms some tests someone else made. We have the whole picture now, and I hope this will help the provider fix this weird redirects.

Nonetheless, I tried

curl -k -i -u username:'password' -X PROPFIND https://webmail.web4all.fr/dav --upload-file - -H "Depth: 0" -H "Content-Length: 105" -H "Transfer-Encoding:" -H "Expect:" <<end
<?xml version="1.0"?>
<a:propfind xmlns:a="DAV:">
<a:prop><a:current-user-principal/></a:prop>
</a:propfind>
end

And I have been served a 400:

HTTP/1.1 400 Bad Request
Server: nginx
Date: Mon, 01 Dec 2014 23:54:44 GMT
Content-Length: 0
Set-Cookie: SERVERID=zose-proxy02a; path=/

Since my password has blank spaces, I was worried that this might be the reason, and I tried twice, with simple quotes and without password (curl propts you to submit it then, which is perhaps better, since it won't show up in bash history). Seems like we aren't done yet, does it?