HTTPS over HTTPS CONNECT tunnel (proxy) does not work

Elasticsearch Version

8.*

Installed Plugins

No response

Java Version

bundled

OS Version

any

Problem Description

It's possible to configure a proxy in ES in at lest 2 places: xpack.http.proxy.{host,port,scheme} and xpack.security.authc.realms.oidc.<realm-name>.http.proxy.{host,port,scheme}. The contexts in which the requests go through the proxy is outside the scope here. In all the cases when a request goes through any HTTPS proxy (the *.scheme: https setting) the same problem, as described below, may manifests.

The problem manifests when both the proxy and the target of the request use HTTPS. For example, in the oidc realm case, both xpack.security.authc.realms.oidc.<realm-name>.http.proxy.scheme is https and the xpack.security.authc.realms.oidc.<realm-name>.op.token_endpoint is set to a "https://" URL. The problem is that the client library that we use (group: 'org.apache.httpcomponents', name: 'httpasyncclient', version: '4.1.5') cannot handle a TLS session on top of an existing TLS session.

In more details, when the httpAsyncClientBuilder is configured to use a proxy, it will use the HTTP CONNECT protocal. The client will connect over TLS to the proxy, and then issue a HTTP CONNECT request to it, instructing the proxy to open a TCP connection to the host of the target URL. If the proxy is successful in opening the connection to the target, it will reply 200 OK back to the client, and from that point on will forward everything from the client's connection over to the connection with the target. If the target URL also uses HTTPS, then the client must and will attempt another TLS handshake over the same connection, this time with the target host. It is at this point that the client lib barfs an IllegalStateException because the connection already has a TLS context attached. Here's a sample stacktrace:

org.elasticsearch.ElasticsearchSecurityException: Failed to exchange code for Id Token using the Token Endpoint.
    at org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectAuthenticator$2.failed(OpenIdConnectAuthenticator.java:586) [x-pack-security-7.17.6.jar:7.17.6]
    at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137) [httpcore-4.4.12.jar:4.4.12]
    at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:101) [httpasyncclient-4.1.4.jar:4.1.4]
    at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426) [httpasyncclient-4.1.4.jar:4.1.4]
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.exception(HttpAsyncRequestExecutor.java:163) [httpcore-nio-4.4.12.jar:4.4.12]
    at org.apache.http.impl.nio.DefaultNHttpClientConnection.produceOutput(DefaultNHttpClientConnection.java:310) [httpcore-nio-4.4.12.jar:4.4.12]
    at org.apache.http.impl.nio.client.InternalIODispatch.onOutputReady(InternalIODispatch.java:86) [httpasyncclient-4.1.4.jar:4.1.4]
    at org.apache.http.impl.nio.client.InternalIODispatch.onOutputReady(InternalIODispatch.java:39) [httpasyncclient-4.1.4.jar:4.1.4]
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:152) [httpcore-nio-4.4.12.jar:4.4.12]
    at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:187) [httpcore-nio-4.4.12.jar:4.4.12]
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:341) [httpcore-nio-4.4.12.jar:4.4.12]
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) [httpcore-nio-4.4.12.jar:4.4.12]
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) [httpcore-nio-4.4.12.jar:4.4.12]
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) [httpcore-nio-4.4.12.jar:4.4.12]
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) [httpcore-nio-4.4.12.jar:4.4.12]
    at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.IllegalStateException: I/O session is already upgraded to TLS/SSL
    at org.apache.http.util.Asserts.check(Asserts.java:34) ~[?:?]
    at org.apache.http.nio.conn.ssl.SSLIOSessionStrategy.upgrade(SSLIOSessionStrategy.java:164) ~[?:?]
    at org.apache.http.nio.conn.ssl.SSLIOSessionStrategy.upgrade(SSLIOSessionStrategy.java:64) ~[?:?]
    at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.upgrade(PoolingNHttpClientConnectionManager.java:460) ~[?:?]
    at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.onRouteUpgrade(AbstractClientExchangeHandler.java:208) ~[?:?]
    at org.apache.http.impl.nio.client.MainClientExec.generateRequest(MainClientExec.java:181) ~[?:?]
    at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.generateRequest(DefaultClientExchangeHandlerImpl.java:134) ~[?:?]
    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.requestReady(HttpAsyncRequestExecutor.java:193) ~[?:?]
    at org.apache.http.impl.nio.DefaultNHttpClientConnection.produceOutput(DefaultNHttpClientConnection.java:287) ~[?:?]
    ... 10 more","-","-",elasticsearch,"-","-"

Note that curl does work in this HTTPS over HTTPS CONNECT scenario, eg: curl --proxy-cacert conf/certs/test-nginx-CA.crt https://github.com/ -o /dev/null -v -x https://localhost:3128 (the HTTPS connect proxy listens at localhost:3128 and has a cert signed by the test-ngnix-CA.crt).

How I tested this: Firstly, I generated proxy certificates in PEM and JKS formats. Then I used nginx 1.25.2, running locally, as a HTTP CONNECT proxy. It needs an external module to support the CONNECT method: https://github.com/chobits/ngx_http_proxy_connect_module (./configure --add-module=ngx_http_proxy_connect_module --with-http_ssl_module). Here's the config:

    server {
        listen                         3127;
        listen                         3128 ssl;
        server_name                    localhost;

        # dns resolver used by forward proxying
        resolver                       8.8.8.8;

        ssl_certificate_key            certs/test-nginx-proxy.key;
        ssl_certificate                certs/test-nginx-proxy.crt;

        # forward proxy for CONNECT requests
        proxy_connect;
        proxy_connect_allow            443 563;
        proxy_connect_connect_timeout  1000s;
        proxy_connect_data_timeout     1000s;

        # defined by yourself for non-CONNECT requests
        # Example: reverse proxy for non-CONNECT requests
        location / {
            proxy_pass http://$host;
            proxy_set_header Host $host;
        }
    }

Lastly, I extracted the client code in a separate project:

    public static void proxyTheXpackWay() throws Exception {
        URL truststoreResource = Main.class.getClassLoader().getResource("test-nginx-CA.jks");
        if (truststoreResource == null) {
            throw new IllegalStateException("cannot access the truststore");
        }
        final SSLContext clientContext = SSLContexts.custom()
                .loadTrustMaterial(new File(truststoreResource.toURI()), "testtest".toCharArray())
                .build();
        final CloseableHttpAsyncClient client = HttpAsyncClients
                .custom()
                .setSSLContext(clientContext)
                .build();

        client.start();
        final HttpHost proxy = new HttpHost("localhost", 3128, "https");
        final RequestConfig config = RequestConfig.custom().setProxy(proxy).build();
        final HttpGet request = new HttpGet("http://httpforever.com");
        request.setConfig(config);
        final Future<HttpResponse> future = client.execute(request, null);
        final HttpResponse response = future.get();
        System.out.println(response.getStatusLine());
        String result = new BufferedReader(new InputStreamReader(response.getEntity().getContent()))
                .lines().parallel().collect(Collectors.joining("\n"));
        System.out.println(result);
        client.close();
    }

    static void proxyTheOidcWay() throws Exception {
        ConnectingIOReactor ioReactor = new DefaultConnectingIOReactor(
                IOReactorConfig.custom().setSoKeepAlive(true).build()
        );
        URL truststoreResource = Main.class.getClassLoader().getResource("test-nginx-CA.jks");
        if (truststoreResource == null) {
            throw new IllegalStateException("cannot access the truststore");
        }
        final SSLContext clientContext = SSLContexts.custom()
                .loadTrustMaterial(new File(truststoreResource.toURI()), "testtest".toCharArray())
                .build();
        Registry<SchemeIOSessionStrategy> registry = RegistryBuilder.<SchemeIOSessionStrategy>create()
                .register("http", NoopIOSessionStrategy.INSTANCE)
                .register("https", new SSLIOSessionStrategy(clientContext, new DefaultHostnameVerifier()))
                .build();
        PoolingNHttpClientConnectionManager connectionManager = new PoolingNHttpClientConnectionManager(ioReactor, registry);
        connectionManager.setDefaultMaxPerRoute(200);
        connectionManager.setMaxTotal(200);
        final RequestConfig requestConfig = RequestConfig.custom()
                .setConnectTimeout(5000)
                .setConnectionRequestTimeout(5000)
                .setSocketTimeout(5000)
                .build();
        HttpAsyncClientBuilder httpAsyncClientBuilder = HttpAsyncClients.custom()
                .setConnectionManager(connectionManager)
                .setDefaultRequestConfig(requestConfig)
                .setKeepAliveStrategy(getKeepAliveStrategy());

        httpAsyncClientBuilder.setProxy(new HttpHost("localhost", 3128, "https"));
        try (CloseableHttpAsyncClient httpAsyncClient = httpAsyncClientBuilder.build()) {
            httpAsyncClient.start();
            HttpContext context = HttpClientContext.create();
            final Future<HttpResponse> future = httpAsyncClient.execute(new HttpGet("https://github.com"), context, null);
            final HttpResponse response = future.get();
            System.out.println(response.getStatusLine());
            String result = new BufferedReader(new InputStreamReader(response.getEntity().getContent()))
                    .lines().parallel().collect(Collectors.joining("\n"));
            System.out.println(result);
        }

    }

    private static ConnectionKeepAliveStrategy getKeepAliveStrategy() {
        final long userConfiguredKeepAlive = 3 * 60 * 1000;
        return (response, context) -> {
            var serverKeepAlive = DefaultConnectionKeepAliveStrategy.INSTANCE.getKeepAliveDuration(response, context);
            long actualKeepAlive;
            if (serverKeepAlive <= -1) {
                actualKeepAlive = userConfiguredKeepAlive;
            } else if (userConfiguredKeepAlive <= -1) {
                actualKeepAlive = serverKeepAlive;
            } else {
                actualKeepAlive = Math.min(serverKeepAlive, userConfiguredKeepAlive);
            }
            if (actualKeepAlive < -1) {
                actualKeepAlive = -1;
            }
            return actualKeepAlive;
        };
    }

Note that I also tested the latest lib version released (group: 'org.apache.httpcomponents.client5', name: 'httpclient5', version: '5.2.1') and the same problem persists:

Caused by: java.lang.IllegalStateException: TLS already activated
    at org.apache.hc.core5.reactor.InternalDataChannel.startTls(InternalDataChannel.java:252)
    at org.apache.hc.client5.http.impl.nio.DefaultManagedAsyncClientConnection.startTls(DefaultManagedAsyncClientConnection.java:158)
    at org.apache.hc.client5.http.ssl.AbstractClientTlsStrategy.upgrade(AbstractClientTlsStrategy.java:111)
Caused by: java.lang.IllegalStateException: TLS already activated

    at org.apache.hc.client5.http.ssl.DefaultClientTlsStrategy.upgrade(DefaultClientTlsStrategy.java:48)
    at org.apache.hc.client5.http.impl.nio.DefaultAsyncClientConnectionOperator.upgrade(DefaultAsyncClientConnectionOperator.java:179)
    at org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManager.upgrade(PoolingAsyncClientConnectionManager.java:505)
    at org.apache.hc.client5.http.impl.async.InternalHttpAsyncExecRuntime.upgradeTls(InternalHttpAsyncExecRuntime.java:249)
    at org.apache.hc.client5.http.impl.async.AsyncConnectExec.proceedToNextHop(AsyncConnectExec.java:323)
    at org.apache.hc.client5.http.impl.async.AsyncConnectExec.access$000(AsyncConnectExec.java:82)
    at org.apache.hc.client5.http.impl.async.AsyncConnectExec$4.completed(AsyncConnectExec.java:298)
    at org.apache.hc.client5.http.impl.async.AsyncConnectExec$6.completed(AsyncConnectExec.java:423)
    at org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.consumeResponse(HttpAsyncMainClientExec.java:224)
    at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.consumeHeader(ClientHttp1StreamHandler.java:243)
    at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.consumeHeader(ClientHttp1StreamDuplexer.java:348)
    at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.consumeHeader(ClientHttp1StreamDuplexer.java:80)
    at org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onInput(AbstractHttp1StreamDuplexer.java:298)
    at org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.inputReady(AbstractHttp1IOEventHandler.java:64)
    at org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.inputReady(ClientHttp1IOEventHandler.java:41)
    at org.apache.hc.core5.reactor.ssl.SSLIOSession.decryptData(SSLIOSession.java:600)
    at org.apache.hc.core5.reactor.ssl.SSLIOSession.access$200(SSLIOSession.java:74)
    at org.apache.hc.core5.reactor.ssl.SSLIOSession$1.inputReady(SSLIOSession.java:202)
    at org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:142)
    at org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:51)
    at org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(SingleCoreIOReactor.java:178)
    at org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreIOReactor.java:127)
    at org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:86)
    at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44)
    at java.base/java.lang.Thread.run(Thread.java:833

I only found this external issue, https://issues.apache.org/jira/browse/HTTPASYNC-139 that bluntly says this scenario (or one similar to it) is not supported (it doesn't seem concerned about an IllegalStateException case...).

In principle, a HTTPS CONNECT tunnel proxy is not a substantial security improvement, compared to a plain HTTP one, when the target URL is itself HTTPS. I labeled it as a bug because we should at least document this limitation, and even better, error when we detect it. Ideally we should fix it, but it might require using a different HTTP client lib (eg jetty).

elastic / elasticsearch