Workaround for HTTPS proxies

[lpuglia]

Hello, thanks for the amazing job with this library, it works so much better than HttpURLConnection. I have a question about the HTTPS proxy feature, the first time it was mentioned it was in this issue: https://github.com/square/okhttp/issues/3787

I have been trying to connect to HTTP/HTTPS using the following code:

Authenticator proxyAuthenticator = new Authenticator() {
    @Override public Request authenticate(Route route, Response response) throws IOException {
        String credential = Credentials.basic(username, password);
        return response.request().newBuilder()
                .header("Proxy-Authorization", credential)
                .build();
    }
};

OkHttpClient client = new OkHttpClient.Builder()
        .proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort)))
        .proxyAuthenticator(proxyAuthenticator).build()
        .newCall(new Request.Builder().url("https://api64.ipify.org/?format=json").build()).execute();

I'm using a proxy-provider which offers both HTTP and HTTPS proxies (same hostname different port), the HTTP works without any problem, unfortunately whenever i try to use the HTTPS port i get:

java.net.SocketException: Connection reset

I saw that a pull request (https://github.com/square/okhttp/pull/4333) was merged in OKHTTPClient 3.11.1 but the HTTPS-proxy feature is still a task for the Icebox milestone.

I have been trying the workaround suggested in the issue:

OkHttpClient client = new OkHttpClient.Builder()
        .socketFactory(SSLSocketFactory.getDefault())
        .build();

but I guess that it is probably not valid anymore, in fact it throws the following exception:

java.lang.IllegalArgumentException: socketFactory instanceof SSLSocketFactory

I have been playing around with the following code to use sslSocketFactory instead:

TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
trustManagerFactory.init((KeyStore) null);
TrustManager[] trustManagers = trustManagerFactory.getTrustManagers();
if (trustManagers.length != 1 || !(trustManagers[0] instanceof X509TrustManager)) {
    throw new IllegalStateException("Unexpected default trust managers:" + Arrays.toString(trustManagers));
}
X509TrustManager trustManager = (X509TrustManager) trustManagers[0];
SSLContext sslContext = SSLContext.getInstance("SSL");
sslContext.init(null, new TrustManager[] { trustManager }, null);
SSLSocketFactory sslSocketFactory = sslContext.getSocketFactory();

okHttpClientBuilder.socketFactory(sslSocketFactory.getDefault());

without any luck

My main problem at the moment is that the proxy-provider is phasing out from the HTTP protocol and will only provide HTTPS. I'm wondering if there is an "official" workaround until the milestone is reached and if there isn't any suggestion on the matter will be gladly accepted.

[swankjesse]

Thanks for raising this. Can you tell me a bit more about the use case? There’s a decent amount of complexity in doing this properly... we have a lot of code to make HTTPS work well, and supporting it for proxies as well is a lot of work!

In particular, we should figure out...

• would we surface the proxy handshake?
• do we do HTTP/2 to the proxy?
• pinning?
• connection spec?
• SNI? ALPN?
• do we recover from TLS handshake failures?

[lpuglia]

That is a lot of questions, I hope I did my homeworks correctly but take whatever i say with a pinch of salt. So, my use case (which i suppose it would be the most common beside corporate VPNs) is to have a proxied http client to increase the sercurity of my activities and bypass geolocalization checks. As you may know, most VPN providers have a list of servers which you can access with a monthly/yearly subscription fee. Most of these servers are nothing more than a HTTP/HTTPS/SOCKS proxies. This is very true for most of the notorious VPN providers (if you have ever been on youtube you would know which I am talking about).

Without making any free advertisement, my VPN provider has a lot of proxy servers, some of these servers support both HTTP and HTTPS tunneling, unfortunately the list of servers which support HTTP is shrinking by the month. Just to give you an idea of what I'm talking about i used cURL to do some preliminary testing. This is what I use for the HTTP supported proxy

curl -x http://proxy_server:80 --proxy-user username:password -L url

but you can use the HTTPS port of the server as well (please note the change in protocol and port number):

curl -x https://proxy_server:89 --proxy-user username:password -L url

Now, when i try to input the following command:

curl -x http://proxy_server:89 --proxy-user username:password -L url

Just because I specify the wrong protocol for the proxy I get the same error as with OkHttpclient:

* Recv failure: Connection reset by peer

Now, coming to your points (which are very technical and not at my level):

• would we surface the proxy handshake?
  • for my use case it's not really important, i wouldn't know what to do with it as long as the HTTPS tunneling works as well as the HTTP one.
• do we do HTTP/2 to the proxy?
  • Since i can't find any mention of it on the proxy provider sites I would say it is not widely adopted in the field (they would brag about it if that was the case).
• pinning?
  • isn't it deprecated?
• connection spec?
  • quoting @mgaido91 answer from his issue: " both Chrome and Firefox support it. Here I found an article about Chromium support (https://www.chromium.org/developers/design-documents/secure-web-proxy). Actually it is often referred as SSL proxy (Firefox terminology) or Secure Web Proxy (OSX terminology). I have not been able to find any specification and I don't think that there can be, since there is nothing to specify: it is simply an implementation of a HTTP proxy using SSL."
• SNI? ALPN?
  • ?
• do we recover from TLS handshake failures?
  • ?

[lpuglia]

@swankjesse hey, sorry to bother you again, I was looking https://github.com/square/okhttp/issues/3787 where you were suggesting to use:

OkHttpClient client = new OkHttpClient.Builder()
        .socketFactory(SSLSocketFactory.getDefault())
        .build();

it seems to have worked for the other user, he implemented it here: https://github.com/apache/nifi/commit/37271e82414b9386bb735b61ef54e891300117bf

And I don't see any difference with what I'm doing (you can find it above in my first comment) but I keep getting the following error:

2021-02-18 17:47:09.496 4621-4653/com.channel.tv W/System.err: java.lang.IllegalArgumentException: socketFactory instanceof SSLSocketFactory
2021-02-18 17:47:09.497 4621-4653/com.channel.tv W/System.err:     at okhttp3.OkHttpClient$Builder.socketFactory(OkHttpClient.kt:723)

Any suggestion on how to build the most basic SSLSocketFactory? It would be very appreciated since it seems to have been the perfect workaround in the past. Two days ago my provider shut down all the remaining HTTP port and this would solve my problem completely. Many thanks

[yschimke]

Call this method instead https://square.github.io/okhttp/4.x/okhttp/okhttp3/-ok-http-client/-builder/ssl-socket-factory/

The one you are calling is the low level socket below SSL.

[lpuglia]

@yschimke thanks for the suggestion, this is my full code at the moment:

  Authenticator proxyAuthenticator = new Authenticator() {
      @Override public Request authenticate(Route route, Response response) throws IOException {
          String credential = Credentials.basic(username, password);
          return response.request().newBuilder().header("Proxy-Authorization", credential).build();
      }
  };

  OkHttpClient.Builder clientb = new OkHttpClient.Builder()
          .proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort)))
          .proxyAuthenticator(proxyAuthenticator);

  // Create a trust manager that does not validate certificate chains
  final TrustManager[] trustAllCerts = new TrustManager[] {
          new X509TrustManager() {
              @Override public void checkClientTrusted(java.security.cert.X509Certificate[] chain, String authType) throws CertificateException {}
              @Override public void checkServerTrusted(java.security.cert.X509Certificate[] chain, String authType) throws CertificateException {}
              @Override public java.security.cert.X509Certificate[] getAcceptedIssuers() { return new java.security.cert.X509Certificate[]{}; }
          }
  };

  final SSLContext sslContext = SSLContext.getInstance("SSL");
  sslContext.init(null, trustAllCerts, new java.security.SecureRandom());
  final SSLSocketFactory sslSocketFactory = sslContext.getSocketFactory();

  clientb.sslSocketFactory(sslSocketFactory, (X509TrustManager)trustAllCerts[0]);
  clientb.hostnameVerifier(new HostnameVerifier() {
      @Override public boolean verify(String hostname, SSLSession session) { return true; }
  });

  Request request = new Request.Builder().url("https://api64.ipify.org/?format=json")
          .get().build();

  Log.d("-", clientb.build().newCall(request).execute().body().string());

(I'm on Android if that is relevant) I created a TrustManager that accept all certificates but i still get connection reset. Neither checkServerTrusted nor checkClientTrusted get ever called.

I'm also using HttpLoggingInterceptor to check if there is any useful information but it's almost useless:

2021-02-18 21:27:50.935 6849-6874/com.channel.tv I/okhttp.OkHttpClient: --> GET https://api64.ipify.org/?format=json
2021-02-18 21:27:50.935 6849-6874/com.channel.tv I/okhttp.OkHttpClient: --> END GET
2021-02-18 21:27:51.077 6849-6874/com.channel.tv I/okhttp.OkHttpClient: <-- HTTP FAILED: java.net.SocketException: Connection reset

Just to make the full picture here is the verbose output of cURL when I connect to the very same proxy (didn't set the authentication though):

curl -x https://it146.nordvpn.com:89 https://api64.ipify.org/?format=json -v
*   Trying 212.102.54.108:89...
* TCP_NODELAY set
* Connected to it146.nordvpn.com (212.102.54.108) port 89 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Proxy certificate:
*  subject: CN=*.nordvpn.com
*  start date: Aug 12 14:41:29 2020 GMT
*  expire date: Oct  4 10:49:39 2022 GMT
*  subjectAltName: host "it146.nordvpn.com" matched cert's "*.nordvpn.com"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=AlphaSSL CA - SHA256 - G2
*  SSL certificate verify ok.
* allocate connect buffer!
* Establish HTTP proxy tunnel to api64.ipify.org:443
> CONNECT api64.ipify.org:443 HTTP/1.1
> Host: api64.ipify.org:443
> User-Agent: curl/7.68.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 407 Proxy Authentication Required
< Server: squid
< Mime-Version: 1.0
< Date: Thu, 18 Feb 2021 21:06:17 GMT
< Content-Type: text/html;charset=utf-8
< Content-Length: 5083
< X-Squid-Error: ERR_CACHE_ACCESS_DENIED 0
< Proxy-Authenticate: Basic realm="NordVPN"
< X-Cache: MISS from unn-212-102-54-108.cdn77.com
< X-Cache-Lookup: NONE from unn-212-102-54-108.cdn77.com:89
< Connection: close
<
* Ignore 5083 bytes of response-body
* Received HTTP code 407 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 407 from proxy after CONNECT

and here is the output using the wrong protocol:

curl -x it146.nordvpn.com:89 https://api64.ipify.org/?format=json -v
*   Trying 212.102.54.108:89...
* TCP_NODELAY set
* Connected to it146.nordvpn.com (212.102.54.108) port 89 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to api64.ipify.org:443
> CONNECT api64.ipify.org:443 HTTP/1.1
> Host: api64.ipify.org:443
> User-Agent: curl/7.68.0
> Proxy-Connection: Keep-Alive
>
* Recv failure: Connection reset by peer
* Received HTTP code 0 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

I'm almost sure that OkHTTP is behaving as in this second scenario

[lpuglia]

@swankjesse apparently you suggestion was correct for OkHttp 3.12.0, using the following:

Authenticator proxyAuthenticator = new Authenticator() {
    @Override public Request authenticate(Route route, Response response) throws IOException {
        String credential = Credentials.basic(username, password);
        return response.request().newBuilder().header("Proxy-Authorization", credential).build();
    }
};

OkHttpClient.Builder clientb = new OkHttpClient.Builder()
        .proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort)))
        .proxyAuthenticator(proxyAuthenticator);

clientb.socketFactory(SSLSocketFactory.getDefault());

Request request = new Request.Builder().url("https://api64.ipify.org/?format=json")
        .get().build();

Response response = null;
response = clientb.build().newCall(request).execute();
String string = response.body().string();
response.body().close();

System.out.println(string);

solves the problem, now I'm able to use the HTTPS proxy.

@yschimke while I was using a TCP/IP monitor I noticed that the request to the HTTPS proxy was made in clear text, switching to 3.12.0 and adding socketFactory encrypts the proxy request as well.

Unfortunately, on any version >4.x.x the following exception is thrown:

java.lang.IllegalArgumentException: socketFactory instanceof SSLSocketFactory

Do you have any hint on how to port the code to the newer version? This could be a good workaround for the time being.

P.S.

the previous code only works in Java, there seems to be problems on android, in particular:

javax.net.ssl.SSLHandshakeException: Handshake failed

I think it is because my proxy provider only supports TLSv1.3, here is how to solve: add this to your gradle:

implementation 'org.conscrypt:conscrypt-android:2.5.0'

and this to your app onCreate():

Security.insertProviderAt(Conscrypt.newProvider(), 1);

[yschimke]

OK, I think I understand now. The check I added (that now fails) was because it was a problem that had happened in more typical usage where people called the wrong method.

But your code against 3.12.0 specifically tries to do this "one weird trick"

To test with you could copy this class and hide your implementation with it https://github.com/square/okhttp/blob/480c20e46bb1745e280e42607bbcc73b2c953d97/okhttp/src/test/java/okhttp3/DelegatingSocketFactory.java

[lpuglia]

@yschimke thanks! it finally works with 4.9.0. This is the full code for a client that supports HTTTPS proxies with authentication:

Authenticator proxyAuthenticator = new Authenticator() {
    @Override public Request authenticate(Route route, Response response) throws IOException {
        String credential = Credentials.basic(username, password);
        return response.request().newBuilder().header("Proxy-Authorization", credential).build();
    }
};

OkHttpClient client = new OkHttpClient.Builder()
        .proxy(new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort)))
        .proxyAuthenticator(proxyAuthenticator);
        .socketFactory(new DelegatingSocketFactory(SSLSocketFactory.getDefault()))
        .build();

On Android I still have to use the insertProviderAt trick to avoid error during handshakes, but it finally works.

You can either close the issue or keep it open until you figure out the best way to implement the HTTPS proxy support. Many thanks!

[yschimke]

Closing for now, the workaround seems nice and clean. But the Proxy API doesn't have a clean way to express this, hence Proxy.Type.HTTP for HTTPS.

Thanks for working through this.

[subisueno]

I am working for a client using their VM and their Network. I was facing same problem and following the last update from @lpuglia solved the connectivity issue but started facing a new issue - SslException: Unrecognized SSL message, plaintext connection?

The API url invocation is working absolutely fine from my postman with below set up -

1. Custom proxy set up in postman
2. Proxy type HTTPS, proxy server host and port
3. Switch on proxy authentication with my id, password

@yschimke - Please help me to find what is going wrong in my case, with the above code shared by @lpuglia