s.get("https://manage.office.com") fails with _ssl.c:504: EOF occurred in violation of protocol

[Akhpatel]

I am behind a proxy. When I execute sess.get("https://google.com") I get a 200 status_code but when I try the same with sess.get("https://manage.office.com") it fails with _ssl.c:504: EOF occurred in violation of protocol

Also requests is not installed. I have placed it in ./lib/ and import it form there. There is another folder named requests-2.9.1.dist-info. So I guess the requests version is 2.9.1. Let me know if I can provide any other information to debug this.

This is the first time I am submitting an issue on github. Let me know if I have made any mistakes and will rectify it.

---

[root@xcpnhigas003 office365audit]# python test_proxy 200 Traceback (most recent call last): File "test_proxy", line 17, in response = sess.get('https://manage.office.com', proxies=proxies, verify=False) File "/etc/netwitness/ng/logcollection/content/collection/cmdscript/office365audit/lib/requests/sessions.py", line 480, in get return self.request('GET', url, kwargs) File "/etc/netwitness/ng/logcollection/content/collection/cmdscript/office365audit/lib/requests/sessions.py", line 468, in request resp = self.send(prep, send_kwargs) File "/etc/netwitness/ng/logcollection/content/collection/cmdscript/office365audit/lib/requests/sessions.py", line 576, in send r = adapter.send(request, kwargs) File "/etc/netwitness/ng/logcollection/content/collection/cmdscript/office365audit/lib/requests/adapters.py", line 447, in send raise SSLError(e, request=request) requests.exceptions.SSLError: [Errno 8] _ssl.c:504: EOF occurred in violation of protocol ****8

Python code

import os
os.sys.path.append(os.path.dirname(os.path.abspath('__file__')))
os.sys.path.append(os.path.join(os.path.dirname(os.path.abspath('__file__')), "lib"))
import requests
proxies = {
    'http':'http://xxx:xxx@yyy:8080',
    'https':'http://xxx:xxx@yyy:8080'
}
sess = requests.Session()
a = requests.adapters.HTTPAdapter(max_retries=20)
sess.mount('https://', a)
response = sess.get('https://google.com', proxies=proxies, verify=False)
print(response.status_code)
response = sess.get('https://manage.office.com', proxies=proxies, verify=False)
print(response.status_code)

System info:

Python 2.7.5 (default, Jul 10 2014, 16:10:08) [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2

[Lukasa]

It’s highly likely that you are having problems with the TLS connection: you are running an ancient operating system and it will lack a bunch of features we need. I strongly, strongly suggest you try a newer distribution to see if your problem reproduces.

[Akhpatel]

Thanks Lukasa for your comment.

I have tried this from a different box with the same os and openssl version and there it works. This is a production environment where we are trying to run our script and don't have the option to update any software. Is there any workaround I can try.

Also I had a more conceptual question. Since we are using ssl proxy(B), the ssl connection is established between my system(A) and proxy(B), and between B and https://manage.office.com right? So would it be right to say that the actual problem would be with the ssl libraries being used at the proxy and not at A?

I tried wget and that works

---

[root@xcpnhigas003 office365audit]# wget --no-check-certificate https://manage.office.com --2017-10-14 11:06:06-- https://manage.office.com/ Resolving proxy.xxx.com... yyy.yyy.yyy.yyy Connecting to proxy.xxx.com|yyy.yyy.yyy.yyy|:8080... connected. WARNING: cannot verify manage.office.com's certificate, issued by /C=ES/ST=Barcelona/L=yyy/O=xxx/OU=IT/CN=proxybc01.xxx.com/emailAddress=xxx@yyy.com': Self-signed certificate encountered. Proxy request sent, awaiting response... 200 OK Length: 0 Saving to:index.html.2'

[ <=>                                                                                                                                       ] 0           --.-K/s   in 0s

2017-10-14 11:06:07 (0.00 B/s) - `index.html.2' saved [0/0]

---

So is the issue with the python libraries?

[Lukasa]

I have tried this from a different box with the same os and openssl version and there it works. This is a production environment where we are trying to run our script and don't have the option to update any software. Is there any workaround I can try.

Can you try showing me the output of using curl -v to make a similar request on the box with the problem?

Also I had a more conceptual question. Since we are using ssl proxy(B), the ssl connection is established between my system(A) and proxy(B), and between B and https://manage.office.com right? So would it be right to say that the actual problem would be with the ssl libraries being used at the proxy and not at A?

No, Requests doesn’t support that proxy model. Instead Requests is using CONNECT to tunnel a TCP connection through proxy B and then establishing the TLS handshake, so the proxy is only relevant if it is deliberately intercepting that handshake and trying to perform a MITM attack on the connection.

[Akhpatel]

curl output below. Is this what you wanted

---

[root@xcpnhigas003 office365audit]# curl -v https://manage.office.com
* About to connect() to proxy proxy.xxx.com port 8080 (#0)
*   Trying yyy.yyy.yyy.yyy... connected
* Connected to proxy.xxx.com (yyy.yyy.yyy.yyy) port 8080 (#0)
* Establish HTTP proxy tunnel to manage.office.com:443
* Proxy auth using Basic with user 'svc_s_siemllbab'
> CONNECT manage.office.com:443 HTTP/1.1
> Host: manage.office.com:443
> Proxy-Authorization: Basic c3ZjX3Nfc2llbXRzYjAxOlh6eUpRNXU2b05BaGRsN0w=
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
< Set-Cookie: BIGipServerPool_VIPProxys=1779244972.36895.0000; expires=Sat, 14-Oct-2017 18:01:07 GMT; path=/
<
* Proxy replied OK to CONNECT request
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
* Server certificate:
*       subject: CN=manage.office.com,OU=Microsoft Corporation,O=Microsoft Corporation,L=Redmond,ST=WA,C=US
*       start date: Feb 14 21:26:10 2017 GMT
*       expire date: Mar 14 21:26:10 2018 GMT
*       common name: manage.office.com
*       issuer: E=0901seguretatinternet@yyy.com,CN=proxybc03.xxx.com,OU=IT,O=some org,L=Sabadell,ST=Barcelona,C=ES
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: manage.office.com
> Accept: */*
>
< HTTP/1.1 200 OK
< Cache-Control: no-cache
< Pragma: no-cache
< Expires: -1
< Server: Microsoft-IIS/8.5
< X-AspNet-Version: 4.0.30319
< X-Powered-By: ASP.NET
< Date: Sat, 14 Oct 2017 17:56:08 GMT
< Content-Length: 0
< Proxy-Connection: Keep-Alive
< Connection: Keep-Alive
< Age: 0
<
* Connection #0 to host proxy.xxx.com left intact
* Closing connection #0

---

The only difference between the other box where it worked and this one is that the other was not behind a proxy. But I have tested it from behind a proxy as well on a different box but that was a normal proxy and not a ssl proxy. Not sure if this makes a difference

[Lukasa]

You seem to have authenticated with the proxy here: I wonder if we used netrc for the proxy request with Requests. Mind adding your username and password to the proxy URL in Requests to see if that fixes it? That is: http://proxyusername:proxypassword@proxyhost:proxyport

[Akhpatel]

I had added the proxyusername and proxypassword in the original request. I just replaced the actual username and password with xxx before posting it here. Reposting that part of the code.

import os
os.sys.path.append(os.path.dirname(os.path.abspath('__file__')))
os.sys.path.append(os.path.join(os.path.dirname(os.path.abspath('__file__')), "lib"))
import requests
proxies = {
    'http':'http://xxx:xxx@yyy:8080',
    'https':'http://xxx:xxx@yyy:8080'
}
sess = requests.Session()
a = requests.adapters.HTTPAdapter(max_retries=20)
sess.mount('https://', a)
response = sess.get('https://google.com', proxies=proxies, verify=False)
print(response.status_code)
response = sess.get('https://manage.office.com', proxies=proxies, verify=False)
print(response.status_code)

[Lukasa]

Did that work any better for you?

[Akhpatel]

No. same issue. I am able to make a call to https://google.com and https://login.microsoftonline.com but for https://manage.office.com I get the same ssl error.

[bkreider]

I don't know anything about your issue with requests, but you can also test ssl connections using openssl to see what it thinks about it:

openssl s_client -connect $HOSTNAME:$PORT -CAfile $PATH_TO_CA_ROOT

That will be a good sanity check.

[Akhpatel]

Thanks @bkreider. This system has an old version of openssl and it does not have the -proxy option so that I can run openssl s_client from behind a ssl proxy. I no longer have access to the system. We can close this issue. I will reopen it later if required.