Endless loop with HTTPS proxies returning Connection: close header for CONNECT request

[GoogleCodeExporter]

Serf gets stuck in endless loop if HTTPS proxy return "Connection: close" 
header in response for CONNECT request. Note that tunnel is successfully 
established and connection will be closed when tunnel will be closed. One of 
such proxy is Fiddler.

Reproduction script:
1. Run Fiddler2 proxy [1]
2. Run serf_get.exe -p localhost:8888 https://www.google.com/
serf_get.exe never complete, while in Fiddler shows multiple conversations like 
this:
[[[
C: CONNECT google.com:443 HTTP/1.1
C: Host: google.com:443

S: HTTP/1.1 200 Connection Established
S: FiddlerGateway: Direct
S: StartTime: 14:20:32.465
S: Connection: close
]]]

[1] http://fiddler2.com/

Original issue reported on code.google.com by chemodax@gmail.com on 14 Aug 2013 at 10:26

[GoogleCodeExporter]

I can confirm this issue with MS Forefront TMG too.

Original comment by 1983-01...@gmx.net on 4 Dec 2013 at 8:23

[GoogleCodeExporter]

Since I have upgraded to subversion 1.8 I am experiencing a similar issue with 
the serf library.

Following is the pattern of the messages between me and the proxy:

[[[
C: CONNECT google.com:443 HTTP/1.1
C: Host: google.com:443

S: HTTP/1.1 200 Connection Established
S: Connection: close
S: Proxy-Connection: Keep-Alive
]]]

Unfortunately, as a simple contractor, I could not find out the proxy the 
company is using but I am sure that it is not Fiddler.

With the "Connection: close" in the header, serf keeps closing the connection 
and opening another, over and over again, as it is supposed to do.

The fact is that I remembered having a similar issue with a Firefox upgrade 
last year that stopped working behind the same proxy.

After digging a little, I found the bug 
(https://bugzilla.mozilla.org/show_bug.cgi?id=828236) and also the correction 
(https://bugzilla.mozilla.org/attachment.cgi?id=700613&action=diff&headers=1). 
They have removed the "Proxy-Connection: Keep-Alive" from the header because of 
this previous bug (https://bugzilla.mozilla.org/show_bug.cgi?id=570283). The 
reasoning behind those two bugs are worth reading to understand the change that 
was made. Basically they have included these two new parameters on CONNECT 
requests:

Proxy-Connection: Keep-Alive
Connection: Keep-Alive

I have download the serf source code and included both parameters. Tested using 
serf_get on a linux VM and everything worked as expected.

As I use TortoiseSVN on windows, instead of recompiling everything, I am using 
a proxy (NTLMaps or Fiddler) between my development machine and the enterprise 
proxy to include this headers and everything is also working.

I would suggest including the same parameters in the header for the CONNECT 
requests but I am no expert on this, specially the implications of this change. 
Also, when I changed the serf source code, some tests stopped passing (possible 
because of the change in size of the header).

Original comment by vladlo...@gmail.com on 13 Jan 2014 at 7:53

[GoogleCodeExporter]

Vlad, can you attach a patch?

Original comment by 1983-01...@gmx.net on 14 Jan 2014 at 8:58

[GoogleCodeExporter]

I've tried suggested workaround and it doesn't fix problem at least for Fiddler 
(see attached patch). 

Original comment by i...@visualsvn.com on 14 Jan 2014 at 1:15

Attachments:

• serf-issue-123-v1.patch.txt

[GoogleCodeExporter]

I was trying to create this patch but things were a bit busy for me today.

In my tests I have changed the outgoing.c file as you can see in the attached 
patch.

The patch contains also changes on the tests so they could at least pass by 
just adding the new request parameters.

As I said before, this has worked for me but the company I am working for is 
not using Fiddler. I used it just to include the headers without compiling the 
subversion for windows. For instance, I have attached this rule on the 
OnBeforeRequest:

        if (oSession.HostnameIs("my_proxy_ip") && oSession.port == 443)
        {
            oSession.oRequest["Proxy-Connection"] = "Keep-Alive";
            oSession.oRequest["Connection"] = "Keep-Alive";
        }

Original comment by vladlo...@gmail.com on 14 Jan 2014 at 6:24

Attachments:

• proxy_keepalive.patch.txt

[GoogleCodeExporter]

I don't know if I made myself clear in the last comment but I am using Fiddler 
between subversion and my proxy just to add the parameters without having to 
recompile everything.

My setup is something like this, with both intermediate proxy adding the new 
parameters:

svn -> Fiddler (NTLMaps) -> Company Proxy

Also, I have checked that only by using Proxy-Connection: Keep-Alive my proxy 
also work.

Original comment by vladlo...@gmail.com on 14 Jan 2014 at 6:27

[GoogleCodeExporter]

Hi Vladimir.

thanks for sharing the results of your investigation.

I can't seem to make a lot of sense of your company's proxy:
It replies by saying it supports HTTP/1.1, but requires a non-standard header 
(Proxy-Connection: keep-alive) and a header that is implicit in HTTP/1.1 
(Connection: keep-alive)?

Also, what would this mean?
S: Connection: close
S: Proxy-Connection: Keep-Alive
Both can only concern the connection between client and proxy, so what is this 
proxy trying to say here?

Can you try with only the "Connection: keep-alive" header? I don't think I can 
be convinced to add the Proxy-Connection header, so let's hope it's not needed.

Lieven

Original comment by lieven.govaerts@gmail.com on 15 Jan 2014 at 8:51

[GoogleCodeExporter]

Hello Lieven,

I have just tried with only the Connection: Keep-Alive and the proxy didn't 
work, answering with:

S: HTTP/1.1 200 Connection Established
S: Date: Thu, 16 Jan 2014 11:14:23 GMT
S: Connection: close
S: Proxy-Connection: Keep-Alive
S: Via: 1.1 localhost.localdomain

As I said in the previous comment, my proxy works only if I set the 
Proxy-Connection: Keep-Alive (it don't even require the Connection: Keep-Alive).

It is a shame that no one here can find which proxy the company uses. I will 
try to find someone who does know this information and post it here.

Vladimir

Original comment by vladlo...@gmail.com on 16 Jan 2014 at 11:25

[GoogleCodeExporter]

Vladimir,

what proxy are you behind? I am behind Microsoft Forefront crap. I have tried 
the same constellation with curl and it works. Even enquired on the mailing 
list. Daniel described why it is working with curl though I did not fully 
understand his description. Here is the thread: 
http://curl.haxx.se/mail/archive-2013-12/0005.html

Original comment by 1983-01...@gmx.net on 16 Jan 2014 at 11:28

[GoogleCodeExporter]

I am still trying to find the proxy here.

In the curl thread, I think that Daniel was trying to say that it doesn't 
matter that the response of the connect has a Connection: Close, if it is a 
200, it means that the tunnel was successfull. Note that this is aligned with 
the comment that started this thread.

I inspected the curl source code for lib/http_proxy.c and found that they also 
seem to include the Proxy-Connection: Keep-Alive, so the Connection: close 
wouldn't appear on my proxy.

I have also a older version of subversion (1.7.10) here that had neon. I had a 
wireshark trace of the connection using neon and serf when I first start 
getting this issue. In that trace, they seem to "by-pass" this Connection: 
close on the header if the response is 200 and use the tunnel.

Both solutions are worth trying but I think that, although depreciated, the 
Proxy-Connection: Keep-Alive has made its way through current versions of new 
softwares because there were some proxies that used this.

Have you tried the Fiddler or NTLMaps in your setup to see if it works with 
your Forefront proxy?

Original comment by vladlo...@gmail.com on 16 Jan 2014 at 12:09

[GoogleCodeExporter]

Vladimir, I've tested your patch and it doesn't fix original issue with Fiddler 
proxy. 

In this case your workaround should not committed to trunk and we should fully 
fix issue by ignore Connection: close for CONNECT method when tunnel is 
created. Lieven, do you have any ideas how we can fix this?

Original comment by i...@visualsvn.com on 16 Jan 2014 at 3:15

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Vlad, how can I instruct Fiddler to act as a proxy for my proxy will that 
actually work?
Ivan, did you check the curl source to see what's different to serf?

Btw, my proxy responds with

HTTP/1.1 407 Proxy Authentication Required ( Forefront TMG requires 
authorization to fulfill the request. Access to the Web Proxy filter is denied. 
 )
Via: 1.1 MCHP941X
Proxy-Authenticate: Negotiate
Proxy-Authenticate: Kerberos
Proxy-Authenticate: NTLM
Connection: close
Proxy-Connection: close
Pragma: no-cache
Cache-Control: no-cache
Content-Type: text/html
Content-Length: 896

if I try to call https://www.google.de

Original comment by 1983-01...@gmx.net on 16 Jan 2014 at 3:55

[GoogleCodeExporter]

This is similar to issue 119.

Original comment by 1983-01...@gmx.net on 16 Jan 2014 at 3:59

[GoogleCodeExporter]

How I do my Fiddler setup:
1) Start Fiddler;
2) Change system proxy configuration (Control Panel -> Internet Options -> 
Connections -> Lan Settings) to use the company proxy;
3) After the system proxy change, Fiddler will ask you to click on a yellow 
header to restart. Do it;
4) Include the following rule on Fiddler (Rules -> Customize rules...) inside 
the OnBeforeRequest event, replace my_proxy_ip with your proxy ip:
        if (oSession.HostnameIs("my_proxy_ip") && oSession.port == 443)
        {
            oSession.oRequest["Proxy-Connection"] = "Keep-Alive";
        }
5) Instruct subversion or serf_get to use Fiddler as a proxy (default: 
127.0.0.1:8888)

This is my setup and it is working.

As Igor mentioned, this is just a workaround as it won't solve the issue when 
Fiddler is used as the only proxy.

Original comment by vladlo...@gmail.com on 16 Jan 2014 at 4:10

[GoogleCodeExporter]

Lieven,

As far I understand "Connection: Close" means that connection will be closed 
AFTER receiving response. In case of SSL tunnel connection will be closed when 
tunnel will be closed. I didn't check yet, but most likely Neon and curl close 
connection when SSL tunnel is closed.

Original comment by chemodax@gmail.com on 16 Jan 2014 at 5:01

[GoogleCodeExporter]

@chemodax: yes, I understand it the same way like you. That's at least one bug 
we can fix in the ssl tunnel setup code. I'll have a look.

Original comment by lieven.govaerts@gmail.com on 16 Jan 2014 at 6:52

[GoogleCodeExporter]

r2265,-6 make serf ignore the Connection: close header on a 2xx response to a 
CONNECT request.
This means it won't close the connection immediately after receiving that 
response, if the proxy doesn't close the connection either serf should be able 
to communicate with the server.

Original comment by lieven.govaerts@gmail.com on 16 Jan 2014 at 11:24

[GoogleCodeExporter]

Lieven,

I have retried serf_get with your updates against our proxy:

1. serf_get show in help output -H twice
2. Compile time config of logging is done but serf_get how now logging 
parameter. I had to change serf_get.c #if 0 to enable logging. This could be 
better
3. I still have the endless loop, so no avail here

Original comment by 1983-01...@gmx.net on 17 Jan 2014 at 11:50

[GoogleCodeExporter]

The logging code has changed a lot on trunk, serf_get should get a command-line 
option to enable it, that's a todo.

Can you send me the logs you get? You can change the log level to 
SERF_LOGCOMP_ALL instead of SERF_LOGCOMP_ALL_MSG, no need to see the msg 
content.

Original comment by lieven.govaerts@gmail.com on 17 Jan 2014 at 5:16

[GoogleCodeExporter]

Lieven, I can send you the entire conversion because I am querying Google as a 
proof-of-concept. I won't have access to the machines before tuesday, so be a 
bit patient.

Redirection of stderr and stdout will suffice, right?

Original comment by 1983-01...@gmx.net on 17 Jan 2014 at 6:48

[GoogleCodeExporter]

Lieven,

the latest trunk version solved the issue with my proxy.

Thank you and I hope we can see this patch soon on subversion.

Have you checked issue #119 to see if this version solves it too?

Original comment by vladlo...@gmail.com on 17 Jan 2014 at 11:35

[GoogleCodeExporter]

I made a test with Fiddler (4.4.5.6) and it also worked. The setup in this case 
was the same that opened this issue, without another proxy (FiddlerGateway: 
Direct).

serf_get -> Fiddler -> https://www.google.com

To make sure, I have tested both versions with the same setup: latest trunk and 
the 1.3.3 tag version. The later always entered the endless loop while the 
former worked as expected.

Original comment by vladlo...@gmail.com on 18 Jan 2014 at 1:14

[GoogleCodeExporter]

Lieven, log files have been sent privately.

Original comment by 1983-01...@gmx.net on 23 Jan 2014 at 12:15

[GoogleCodeExporter]

Michael, the log files you sent me point to issue #119: not caused by the 
"Connection: close" header, but because Negotiate authentication isn't 
persistent between multiple connections.

Ivan: can you confirm that r2266 fixes this issue for you? Thanks.

Original comment by lieven.govaerts@gmail.com on 29 Jan 2014 at 10:19

• Changed state: Started

[GoogleCodeExporter]

Lieven, really? I thought this is not connected to the auth at all. Even if 
serf cannot authenticate it should stop after one turnround, shouldn't it?

Original comment by 1983-01...@gmx.net on 30 Jan 2014 at 8:37

[GoogleCodeExporter]

This fix will be included in serf 1.3.4 later this week.
Lieven

Original comment by lieven.govaerts@gmail.com on 4 Feb 2014 at 8:14

• Changed state: Fixed