Endless loop if server doesn't accept Negotiate authentication

GoogleCodeExporter commented 9 years ago

Serf stuck up in endless loop when trying to authenticate to the server using 
Negotiate protocol and server doesn't accept client credentials. For example if 
server and client are in different domains.

Original issue reported on code.google.com by chemodax@gmail.com on 16 Jun 2011 at 7:42

GoogleCodeExporter commented 9 years ago

You don't happen to happen a wireshark dump of this session?

I'm wondering if client and server keep exchanging requests and 401 responses, 
or if the loop is within serf. In this context, the condition in auth_kerb.c 
line 211-212 looks suspicious.

Original comment by lieven.govaerts@gmail.com on 21 Jun 2011 at 8:46

GoogleCodeExporter commented 9 years ago

Ivan and I went through a Fiddler2 network trace of this issue. What seems to 
happen is this:

C -> S:  GET
S -> C:  401 - WWW-Authenticate Negotiate/NTLM

C -> S:  GET Authorization: Negotiate <type1>
S -> C:  401 - WWW-Authenticate <type 2 challenge>

C -> S: GET Authorization: Negotiate <type3 authn>
S -> C:  401 - WWW-Authenticate Negotiate/NTLM

So, during the second part of the authentication, the server aborts the 
negotiation by sending a WWW-Authenticate header without token (as in the 
initial 401 response).

Our analysis of the cause is that, in auth_kerb.c:do_auth, this response of the 
server is ignored, and the request is retried. The idea behind this response 
being ignored is that an application can send 50 requests on a connection 
before that the first 401-response is received. When the first response is 
handled, the other 49 401-responses are still incoming, all with an 'initial 
state' WWW-Authenticate header.

So when receiving responses with WWW-Authenticate/Negotiate header on a 
connection where negotiate was started, we need to differentiate between:
- a response without token send for the 2nd,3rd... pipeline requests
- a response without token send for the request that contained the type 1 or 
type 3 token.

Original comment by lieven.govaerts@gmail.com on 21 Jun 2011 at 10:49

GoogleCodeExporter commented 9 years ago

s/Our analysis of the cause is that/Our analysis of the cause of the endless 
loop is that/

Original comment by lieven.govaerts@gmail.com on 21 Jun 2011 at 10:51

GoogleCodeExporter commented 9 years ago

Ivan, can you have a look at attached patch? It's completely untested and needs 
cleanup, but should implement what we discussed earlier.

Original comment by lieven.govaerts@gmail.com on 25 Jun 2011 at 12:42

Attachments:

serf_kerb_tag.patch.txt

GoogleCodeExporter commented 9 years ago

This patch definitely has at least on bug: in method 
serf__handle_auth_response() you pass uninitalized variable 'request' to 
dispatch_auth() method. Btw all approach with tag_request return flag for authn 
callbacks debatable for me.

Original comment by chemodax@gmail.com on 27 Jun 2011 at 2:25

GoogleCodeExporter commented 9 years ago

I need to hook into this issue, based on this subversion ticket: 
http://subversion.tigris.org/issues/show_bug.cgi?id=3869#desc35

I do not know if the cause cause is the same but I do have the same result as 
the reporter.

I have compiled serf 1.1.x with GSS-API support and linked it against 
Subversion trunk (1.8.0-dev). Here is my outcome (attachments):

Called a SPNEGO protected resource:
$ svn ls http://blnn728x.ww004.siemens.net:8081/manager/html
^Csubversion/svn/list-cmd.c:309: (apr_err=4)
subversion/libsvn_client/list.c:254: (apr_err=4)
subversion/libsvn_client/ra.c:506: (apr_err=4)
subversion/libsvn_client/ra.c:464: (apr_err=4)
subversion/libsvn_client/ra.c:817: (apr_err=4)
subversion/libsvn_client/revisions.c:73: (apr_err=4)
subversion/libsvn_ra_serf/serf.c:521: (apr_err=4)
subversion/libsvn_ra_serf/property.c:1095: (apr_err=4)
subversion/libsvn_ra_serf/util.c:2195: (apr_err=4)
subversion/libsvn_ra_serf/property.c:698: (apr_err=4)
subversion/libsvn_ra_serf/property.c:674: (apr_err=4)
subversion/libsvn_ra_serf/util.c:806: (apr_err=4)
subversion/libsvn_ra_serf/util.c:773: (apr_err=4)
svn: E000004: Fehler beim Ausführen des Kontextes: Interrupted system call

The process won't stop, I had to Ctrl+C it. 

1. In the first place serf generates a correct SPNEGO ticket (patched source, 
see issue 89) after a 401: serf-spnego-request.png
2. The server accepts this one and responds with OK and the mutual answer: 
tomcat-serf-spnego-response.
3. After that OK, serf does not realize that the response body does not carry 
any valid XML and traps into a OPTIONS <=> 401 loop for ever. No further 
tickets are exchanged. Subversion process has to be killed: 
serf-endless-loop.png

To verify that the server is not faulty, I have issued the same request with 
libsvn_ra_neon. Neon sends the tickets, sees that the response carries no XML 
and aborts further process:
$ svn ls http://blnn728x.ww004.siemens.net:8081/manager/html
subversion/svn/list-cmd.c:302: (apr_err=175002)
subversion/libsvn_client/list.c:153: (apr_err=175002)
subversion/libsvn_client/ra.c:475: (apr_err=175002)
subversion/libsvn_client/ra.c:335: (apr_err=175002)
subversion/libsvn_ra/ra_loader.c:496: (apr_err=175002)
svn: E175002: Unable to connect to a repository at URL 
'http://blnn728x.ww004.siemens.net:8081/manager/html'
subversion/libsvn_ra_neon/util.c:1556: (apr_err=175002)
subversion/libsvn_ra_neon/util.c:1152: (apr_err=175002)
svn: E175002: Die OPTIONS-Anfrage gab in der Antwort ungültiges XML zurück: 
XML parse error at line 1: Extra content at the end of the document
 (http://blnn728x.ww004.siemens.net:8081/manager/html)

See neon-spnego-request.png and tomcat-neon-spnego-response.png

Original comment by 1983-01...@gmx.net on 22 Oct 2012 at 9:18

Attachments:

GoogleCodeExporter commented 9 years ago

Original comment by chemodax@gmail.com on 9 Nov 2012 at 5:32

Added labels: Priority-Critical
Removed labels: Priority-Medium

GoogleCodeExporter commented 9 years ago

I believe this issue is now fixed on trunk, with the implementation of 
per-request authentication which automatically keeps the connection in 
non-pipelined mode.

Original comment by lieven.govaerts@gmail.com on 3 Jan 2013 at 7:40

Changed state: Fixed

GoogleCodeExporter commented 9 years ago

The problem seems to be there in trunk@r1991.

Original comment by chemodax@gmail.com on 3 Jul 2013 at 4:13

Changed state: Accepted

GoogleCodeExporter commented 9 years ago

Can you provide me the stderr output with enabled:AUTH_VERBOSE, CONN_VERBOSE, 
SOCK_VERBOSE and SOCK_MSG_VERBOSE ? From connecting to a http server so I can 
see the headers.

Original comment by lieven.govaerts@gmail.com on 3 Jul 2013 at 7:17

GoogleCodeExporter commented 9 years ago

Hi, Lieven! See attached log (I've truncated auth headers for security reasons)

Original comment by chemodax@gmail.com on 4 Jul 2013 at 9:05

Attachments:

log.txt

GoogleCodeExporter commented 9 years ago

Should be finally fixed by r2112.

Original comment by chemodax@gmail.com on 7 Aug 2013 at 2:09

GoogleCodeExporter commented 9 years ago

Original comment by chemodax@gmail.com on 9 Aug 2013 at 4:45

Changed state: Fixed

Manouchehri / serf

Endless loop if server doesn't accept Negotiate authentication #77