Open prodis opened 4 years ago
Interesting that using Mint I got the same connection error with and without proxy. 🤔
# Without proxy
Mint.HTTP.connect(:https, "apps.correios.com.br", 443)
{:error, %Mint.TransportError{reason: :closed}}
# With proxy
Mint.HTTP.connect(:https, "apps.correios.com.br", 443, proxy: {:http, "localhost", 8888, []})
%Mint.TransportError{reason: %Mint.TransportError{reason: :closed}}
Tinyproxy logs:
CONNECT May 01 22:01:47 [11]: Connect (file descriptor 10): 172.18.0.1 [172.18.0.1]
CONNECT May 01 22:01:47 [11]: Request (file descriptor 10): CONNECT apps.correios.com.br:443 HTTP/1.1
INFO May 01 22:01:47 [11]: No upstream proxy for apps.correios.com.br
INFO May 01 22:01:47 [11]: opensock: opening connection to apps.correios.com.br:443
INFO May 01 22:01:47 [11]: opensock: getaddrinfo returned for apps.correios.com.br:443
CONNECT May 01 22:01:48 [11]: Established connection to host "apps.correios.com.br" using file descriptor 11.
INFO May 01 22:01:48 [11]: Not sending client headers to remote machine
INFO May 01 22:01:48 [11]: Closed connection between local client (fd:10) and remote client (fd:11)
I have opened an issue in Mint too: https://github.com/elixir-mint/mint/issues/255
Using httpc happens a similar connection error with and without proxy.
url = 'https://apps.correios.com.br/SigepMasterJPA/AtendeClienteService/AtendeCliente'
headers = [{'Accept', 'text/xmll'}]
content_type = 'text/xml; charset=utf-8'
body =
String.to_charlist("""
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:cli="http://cliente.bean.master.sigep.bsb.correios.com.br/">
<soapenv:Header />
<soapenv:Body>
<cli:consultaCEP>
<cep>13212-070</cep>
</cli:consultaCEP>
</soapenv:Body>
</soapenv:Envelope>
""")
:inets.start()
:ok
:ssl.start()
:ok
# Without proxy
:httpc.request(:post, {url, headers, content_type, body}, [], [])
{:error,
{:failed_connect,
[{:to_address, {'apps.correios.com.br', 443}}, {:inet, [:inet], :closed}]}}
# With proxy
:httpc.set_options([{:proxy, {{'localhost', 8888}, []}}])
:ok
:httpc.request(:post, {url, headers, content_type, body}, [], [])
{:error,
{:failed_connect,
[
{:to_address, {'apps.correios.com.br', 443}},
{:tls, [server_name_indication: 'apps.correios.com.br'], :closed}
]}}
Tinyproxy logs:
CONNECT May 01 23:35:15 [7]: Connect (file descriptor 10): 172.18.0.1 [172.18.0.1]
CONNECT May 01 23:35:15 [7]: Request (file descriptor 10): CONNECT apps.correios.com.br:443 HTTP/1.1
INFO May 01 23:35:15 [7]: No upstream proxy for apps.correios.com.br
INFO May 01 23:35:15 [7]: opensock: opening connection to apps.correios.com.br:443
INFO May 01 23:35:15 [7]: opensock: getaddrinfo returned for apps.correios.com.br:443
CONNECT May 01 23:35:15 [7]: Established connection to host "apps.correios.com.br" using file descriptor 11.
INFO May 01 23:35:15 [7]: Not sending client headers to remote machine
INFO May 01 23:35:16 [7]: Closed connection between local client (fd:10) and remote client (fd:11)
I could make a successful request to the host apps.correios.com.br
in Elixir/Erlang only using hackney
as a HTTP client without proxy.
Just for a reference using ibrowse, without proxy returns {:error, :closed}
, and with proxy {:error, :retry_later}
.
url = 'https://apps.correios.com.br/SigepMasterJPA/AtendeClienteService/AtendeCliente'
headers = [
{'Accept', 'text/xmll'},
{'Content-Type', 'text/xml; charset=utf-8'}
]
body =
String.to_charlist("""
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:cli="http://cliente.bean.master.sigep.bsb.correios.com.br/">
<soapenv:Header />
<soapenv:Body>
<cli:consultaCEP>
<cep>13212-070</cep>
</cli:consultaCEP>
</soapenv:Body>
</soapenv:Envelope>
""")
# Without proxy
:ibrowse.send_req(url, headers, :post, [body])
{:error, {:conn_failed, {:error, :closed}}}
# With proxy
:ibrowse.send_req(url, headers, :post, [body], [proxy_host: 'localhost', proxy_port: 8888])
{:error, :retry_later}
Tinyproxy logs:
CONNECT May 02 20:55:42 [9]: Connect (file descriptor 10): 172.19.0.1 [172.19.0.1]
CONNECT May 02 20:55:42 [9]: Request (file descriptor 10): CONNECT apps.correios.com.br:443 HTTP/1.1
INFO May 02 20:55:42 [9]: No upstream proxy for apps.correios.com.br
INFO May 02 20:55:42 [9]: opensock: opening connection to apps.correios.com.br:443
INFO May 02 20:55:42 [9]: opensock: getaddrinfo returned for apps.correios.com.br:443
CONNECT May 02 20:55:43 [9]: Established connection to host "apps.correios.com.br" using file descriptor 11.
INFO May 02 20:55:43 [9]: Not sending client headers to remote machine
INFO May 02 20:55:43 [9]: Closed connection between local client (fd:10) and remote client (fd:11)
CONNECT May 02 20:55:43 [9]: Connect (file descriptor 10): 172.19.0.1 [172.19.0.1]
CONNECT May 02 20:55:43 [9]: Request (file descriptor 10): CONNECT apps.correios.com.br:443 HTTP/1.1
INFO May 02 20:55:43 [9]: No upstream proxy for apps.correios.com.br
INFO May 02 20:55:43 [9]: opensock: opening connection to apps.correios.com.br:443
INFO May 02 20:55:43 [9]: opensock: getaddrinfo returned for apps.correios.com.br:443
CONNECT May 02 20:55:43 [9]: Established connection to host "apps.correios.com.br" using file descriptor 11.
INFO May 02 20:55:43 [9]: Not sending client headers to remote machine
INFO May 02 20:55:44 [9]: Closed connection between local client (fd:10) and remote client (fd:11)
CONNECT May 02 20:55:44 [9]: Connect (file descriptor 10): 172.19.0.1 [172.19.0.1]
CONNECT May 02 20:55:44 [9]: Request (file descriptor 10): CONNECT apps.correios.com.br:443 HTTP/1.1
INFO May 02 20:55:44 [9]: No upstream proxy for apps.correios.com.br
INFO May 02 20:55:44 [9]: opensock: opening connection to apps.correios.com.br:443
INFO May 02 20:55:44 [9]: opensock: getaddrinfo returned for apps.correios.com.br:443
CONNECT May 02 20:55:44 [9]: Established connection to host "apps.correios.com.br" using file descriptor 11.
INFO May 02 20:55:44 [9]: Not sending client headers to remote machine
INFO May 02 20:55:45 [9]: Closed connection between local client (fd:10) and remote client (fd:11)
Another reference using Gun
, that returns timeout and connection closed.
# Timeout
{:ok, conn} = :gun.open('apps.correios.com.br', 443)
:gun.await_up(conn)
{:error, :timeout}
# Setting a long connection timeout
{:ok, conn} = :gun.open('apps.correios.com.br', 443)
:gun.await_up(conn, 30000)
{:error, {:shutdown, :closed}}
I have opened an issue in Gun too: https://github.com/ninenines/gun/issues/228
I have tested HTTP connection/request to the host apps.correios.com.br
with all known Elixir/Erlang HTTP clients and only hackney
not using proxy works.
HTTP client | Not using proxy | With proxy |
---|---|---|
hackney | :white_check_mark: | :x: |
httpc | :x: | :x: |
ibrowse | :x: | :x: |
Gun | :x: | :x: |
Mint | :x: | :x: |
So, what is "the magic" hackney
does to make it work? 😄 And why using proxy does not work?
I could make it work with proxy using the cipher AES256-SHA256 and TLS version 1.2.
url = "https://apps.correios.com.br/SigepMasterJPA/AtendeClienteService/AtendeCliente"
headers = [{"Content-Type", "text/xml; charset=utf-8"}]
body = """
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:cli="http://cliente.bean.master.sigep.bsb.correios.com.br/">
<soapenv:Header />
<soapenv:Body>
<cli:consultaCEP>
<cep>13212-070</cep>
</cli:consultaCEP>
</soapenv:Body>
</soapenv:Envelope>
"""
options = [
proxy: {"127.0.0.1", 8888},
ssl_options: [ciphers: ['AES256-SHA256'], versions: [:"tlsv1.2"]],
with_body: true
]
:hackney.post(url, headers, body, options)
{:ok, 200,
[
{"Date", "Mon, 04 May 2020 22:08:55 GMT"},
{"Server", "Apache/2.4.10 (Debian)"},
{"X-OPNET-Transaction-Trace", "a2_17dcfce9-5957-4b97-a769-481988faf643"},
{"Content-Type", "text/xml;charset=UTF-8"},
{"Content-Length", "422"},
{"Vary", "Accept-Encoding,User-Agent"},
{"Set-Cookie",
"_op_aixPageId=a2_17dcfce9-5957-4b97-a769-481988faf643; Path=/"},
{"Set-Cookie",
"app-%3FINTERNO%3Fpool_proxy_app_sigep_443=POCCAIAK; Expires=Mon, 04-May-2020 22:28:57 GMT; Path=/"},
{"Set-Cookie",
"sto-id-%3FEXTERNO_2%3Fpool_Proxy_reverso_Apps_443=BIABKIMA; Expires=Mon, 04-May-2020 22:28:57 GMT; Path=/"}
],
"<soap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\"><soap:Body><ns2:consultaCEPResponse xmlns:ns2=\"http://cliente.bean.master.sigep.bsb.correios.com.br/\"><return><bairro>Parque Residencial Eloy Chaves</bairro><cep>13212070</cep><cidade>Jundiaí</cidade><complemento2></complemento2><end>Avenida Benedicto Castilho de Andrade</end><uf>SP</uf></return></ns2:consultaCEPResponse></soap:Body></soap:Envelope>"}
The original solution is from ninenines/gun#228.
The question is: why when using a proxy the behaviour is different from the request not using proxy? 🙂
did you try with the latest master?
@prodis bump.
With hackney 1.16.0 I am receiving {:error, :closed}
for both with and without proxy.
url = "https://apps.correios.com.br/SigepMasterJPA/AtendeClienteService/AtendeCliente"
headers = [{"Content-Type", "text/xml; charset=utf-8"}]
body = """
<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:cli="http://cliente.bean.master.sigep.bsb.correios.com.br/">
<soapenv:Header />
<soapenv:Body>
<cli:consultaCEP>
<cep>13212-070</cep>
</cli:consultaCEP>
</soapenv:Body>
</soapenv:Envelope>
"""
:hackney.post(url, headers, body, with_body: true)
{:error, :closed}
:hackney.post(url, headers, body, with_body: true, proxy: {"127.0.0.1", 8888})
{:error, :closed}
It only works using the option ssl_options: [ciphers: ['AES256-SHA256'], versions: [:"tlsv1.2"]]
.
FWIW, I am using the following ssl options to mitigate some of these closed connections when using a proxy where message.protocol
is the protocol I am using for the connection. Specifying the cipher seemed to be the problem in my case, so I just specify them all when I am most interested in a protocol-specific connection.
ssl_options = [
versions: [:"#{message.protocol}"],
server_name_indication: '#{message.host_name}',
reuse_sessions: false,
verify: :verify_none,
depth: 3,
ciphers: :ssl.cipher_suites(:all, message.protocol |> String.to_atom, :openssl) ++ :ssl.cipher_suites(:all, message.protocol |> String.to_atom, :rfc)
]
hackney
uses ssl
to establish the ssl connection. However, hackney 1.16.0 removes the default ciphers to pass in the ssl_options
. Your cipher is included in the default ciphers. That's why it worked before. (See here)
ssl
will choose one from the own default ciphers. However, AES256-SHA256
is not in that default list. And your website doesn't support any one in the list.
iex> :ssl.cipher_suites(:default, :"tlsv1.2", :openssl) |> Enum.member?('AES256-SHA256')
false
It's been a few years; is this still an issue?
I am using
hackney
via HTTPoison and in a request for a specific host I am getting{:error, :closed}
when making the request through a proxy. In the beginning my assumption was some problem with the proxy, but using different proxies I got the same error. And testing different proxies requesting to the different hosts worked fine.I could not identify what this specific host that I am requesting has different that
hackney
closes the connection. Doing the same request withcurl
using a proxy works fine.The following examples are using Tinyproxy.
hackney without proxy works fine
hackney with proxy returns closed error
Tinyproxy logs:
curl with proxy works fine
Tinyproxy logs:
My system info
More tests
Curious about running
hackney
on Linux with different OTP versions, I created some tests to run in Travis CI making the requests to the URL with the problem and to another URL that works fine using or not proxy.The integration tests are here:
The results in Travis CI are here: https://travis-ci.org/github/prodis/correios-cep-elixir/builds/681723934
The
{:error, :closed}
is the same running on Travis, except for OTP 20.0 that withhackney
using proxy works fine for the URL with the problem, but it fails for the another URL (jsonplaceholder.typicode.com) returning 404 for the request. 🙃Take a look in this result using Elixir 1.9 and OTP Release 20.0: https://travis-ci.org/github/prodis/correios-cep-elixir/jobs/681723939
Any idea what is going on? 🙂
Let me know if you need more information.