It's possible to configure a proxy in ES in at lest 2 places: xpack.http.proxy.{host,port,scheme} and xpack.security.authc.realms.oidc.<realm-name>.http.proxy.{host,port,scheme}. The contexts in which the requests go through the proxy is outside the scope here. In all the cases when a request goes through any HTTPS proxy (the *.scheme: https setting) the same problem, as described below, may manifests.
The problem manifests when both the proxy and the target of the request use HTTPS.
For example, in the oidc realm case, both xpack.security.authc.realms.oidc.<realm-name>.http.proxy.scheme is https and the xpack.security.authc.realms.oidc.<realm-name>.op.token_endpoint is set to a "https://" URL.
The problem is that the client library that we use (group: 'org.apache.httpcomponents', name: 'httpasyncclient', version: '4.1.5') cannot handle a TLS session on top of an existing TLS session.
In more details, when the httpAsyncClientBuilder is configured to use a proxy, it will use the HTTP CONNECT protocal.
The client will connect over TLS to the proxy, and then issue a HTTP CONNECT request to it, instructing the proxy to open a TCP connection to the host of the target URL. If the proxy is successful in opening the connection to the target, it will reply 200 OK back to the client, and from that point on will forward everything from the client's connection over to the connection with the target. If the target URL also uses HTTPS, then the client must and will attempt another TLS handshake over the same connection, this time with the target host. It is at this point that the client lib barfs an IllegalStateException because the connection already has a TLS context attached. Here's a sample stacktrace:
org.elasticsearch.ElasticsearchSecurityException: Failed to exchange code for Id Token using the Token Endpoint.
at org.elasticsearch.xpack.security.authc.oidc.OpenIdConnectAuthenticator$2.failed(OpenIdConnectAuthenticator.java:586) [x-pack-security-7.17.6.jar:7.17.6]
at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:137) [httpcore-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.executionFailed(DefaultClientExchangeHandlerImpl.java:101) [httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:426) [httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.exception(HttpAsyncRequestExecutor.java:163) [httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.DefaultNHttpClientConnection.produceOutput(DefaultNHttpClientConnection.java:310) [httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.client.InternalIODispatch.onOutputReady(InternalIODispatch.java:86) [httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.http.impl.nio.client.InternalIODispatch.onOutputReady(InternalIODispatch.java:39) [httpasyncclient-4.1.4.jar:4.1.4]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.outputReady(AbstractIODispatch.java:152) [httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.reactor.BaseIOReactor.writable(BaseIOReactor.java:187) [httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:341) [httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) [httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) [httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) [httpcore-nio-4.4.12.jar:4.4.12]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) [httpcore-nio-4.4.12.jar:4.4.12]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.IllegalStateException: I/O session is already upgraded to TLS/SSL
at org.apache.http.util.Asserts.check(Asserts.java:34) ~[?:?]
at org.apache.http.nio.conn.ssl.SSLIOSessionStrategy.upgrade(SSLIOSessionStrategy.java:164) ~[?:?]
at org.apache.http.nio.conn.ssl.SSLIOSessionStrategy.upgrade(SSLIOSessionStrategy.java:64) ~[?:?]
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.upgrade(PoolingNHttpClientConnectionManager.java:460) ~[?:?]
at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.onRouteUpgrade(AbstractClientExchangeHandler.java:208) ~[?:?]
at org.apache.http.impl.nio.client.MainClientExec.generateRequest(MainClientExec.java:181) ~[?:?]
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.generateRequest(DefaultClientExchangeHandlerImpl.java:134) ~[?:?]
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.requestReady(HttpAsyncRequestExecutor.java:193) ~[?:?]
at org.apache.http.impl.nio.DefaultNHttpClientConnection.produceOutput(DefaultNHttpClientConnection.java:287) ~[?:?]
... 10 more","-","-",elasticsearch,"-","-"
Note that curl does work in this HTTPS over HTTPS CONNECT scenario, eg:
curl --proxy-cacert conf/certs/test-nginx-CA.crt https://github.com/ -o /dev/null -v -x https://localhost:3128 (the HTTPS connect proxy listens at localhost:3128 and has a cert signed by the test-ngnix-CA.crt).
How I tested this:
Firstly, I generated proxy certificates in PEM and JKS formats.
Then I used nginx 1.25.2, running locally, as a HTTP CONNECT proxy. It needs an external module to support the CONNECT method: https://github.com/chobits/ngx_http_proxy_connect_module (./configure --add-module=ngx_http_proxy_connect_module --with-http_ssl_module). Here's the config:
server {
listen 3127;
listen 3128 ssl;
server_name localhost;
# dns resolver used by forward proxying
resolver 8.8.8.8;
ssl_certificate_key certs/test-nginx-proxy.key;
ssl_certificate certs/test-nginx-proxy.crt;
# forward proxy for CONNECT requests
proxy_connect;
proxy_connect_allow 443 563;
proxy_connect_connect_timeout 1000s;
proxy_connect_data_timeout 1000s;
# defined by yourself for non-CONNECT requests
# Example: reverse proxy for non-CONNECT requests
location / {
proxy_pass http://$host;
proxy_set_header Host $host;
}
}
Lastly, I extracted the client code in a separate project:
public static void proxyTheXpackWay() throws Exception {
URL truststoreResource = Main.class.getClassLoader().getResource("test-nginx-CA.jks");
if (truststoreResource == null) {
throw new IllegalStateException("cannot access the truststore");
}
final SSLContext clientContext = SSLContexts.custom()
.loadTrustMaterial(new File(truststoreResource.toURI()), "testtest".toCharArray())
.build();
final CloseableHttpAsyncClient client = HttpAsyncClients
.custom()
.setSSLContext(clientContext)
.build();
client.start();
final HttpHost proxy = new HttpHost("localhost", 3128, "https");
final RequestConfig config = RequestConfig.custom().setProxy(proxy).build();
final HttpGet request = new HttpGet("http://httpforever.com");
request.setConfig(config);
final Future<HttpResponse> future = client.execute(request, null);
final HttpResponse response = future.get();
System.out.println(response.getStatusLine());
String result = new BufferedReader(new InputStreamReader(response.getEntity().getContent()))
.lines().parallel().collect(Collectors.joining("\n"));
System.out.println(result);
client.close();
}
static void proxyTheOidcWay() throws Exception {
ConnectingIOReactor ioReactor = new DefaultConnectingIOReactor(
IOReactorConfig.custom().setSoKeepAlive(true).build()
);
URL truststoreResource = Main.class.getClassLoader().getResource("test-nginx-CA.jks");
if (truststoreResource == null) {
throw new IllegalStateException("cannot access the truststore");
}
final SSLContext clientContext = SSLContexts.custom()
.loadTrustMaterial(new File(truststoreResource.toURI()), "testtest".toCharArray())
.build();
Registry<SchemeIOSessionStrategy> registry = RegistryBuilder.<SchemeIOSessionStrategy>create()
.register("http", NoopIOSessionStrategy.INSTANCE)
.register("https", new SSLIOSessionStrategy(clientContext, new DefaultHostnameVerifier()))
.build();
PoolingNHttpClientConnectionManager connectionManager = new PoolingNHttpClientConnectionManager(ioReactor, registry);
connectionManager.setDefaultMaxPerRoute(200);
connectionManager.setMaxTotal(200);
final RequestConfig requestConfig = RequestConfig.custom()
.setConnectTimeout(5000)
.setConnectionRequestTimeout(5000)
.setSocketTimeout(5000)
.build();
HttpAsyncClientBuilder httpAsyncClientBuilder = HttpAsyncClients.custom()
.setConnectionManager(connectionManager)
.setDefaultRequestConfig(requestConfig)
.setKeepAliveStrategy(getKeepAliveStrategy());
httpAsyncClientBuilder.setProxy(new HttpHost("localhost", 3128, "https"));
try (CloseableHttpAsyncClient httpAsyncClient = httpAsyncClientBuilder.build()) {
httpAsyncClient.start();
HttpContext context = HttpClientContext.create();
final Future<HttpResponse> future = httpAsyncClient.execute(new HttpGet("https://github.com"), context, null);
final HttpResponse response = future.get();
System.out.println(response.getStatusLine());
String result = new BufferedReader(new InputStreamReader(response.getEntity().getContent()))
.lines().parallel().collect(Collectors.joining("\n"));
System.out.println(result);
}
}
private static ConnectionKeepAliveStrategy getKeepAliveStrategy() {
final long userConfiguredKeepAlive = 3 * 60 * 1000;
return (response, context) -> {
var serverKeepAlive = DefaultConnectionKeepAliveStrategy.INSTANCE.getKeepAliveDuration(response, context);
long actualKeepAlive;
if (serverKeepAlive <= -1) {
actualKeepAlive = userConfiguredKeepAlive;
} else if (userConfiguredKeepAlive <= -1) {
actualKeepAlive = serverKeepAlive;
} else {
actualKeepAlive = Math.min(serverKeepAlive, userConfiguredKeepAlive);
}
if (actualKeepAlive < -1) {
actualKeepAlive = -1;
}
return actualKeepAlive;
};
}
Note that I also tested the latest lib version released (group: 'org.apache.httpcomponents.client5', name: 'httpclient5', version: '5.2.1') and the same problem persists:
Caused by: java.lang.IllegalStateException: TLS already activated
at org.apache.hc.core5.reactor.InternalDataChannel.startTls(InternalDataChannel.java:252)
at org.apache.hc.client5.http.impl.nio.DefaultManagedAsyncClientConnection.startTls(DefaultManagedAsyncClientConnection.java:158)
at org.apache.hc.client5.http.ssl.AbstractClientTlsStrategy.upgrade(AbstractClientTlsStrategy.java:111)
Caused by: java.lang.IllegalStateException: TLS already activated
at org.apache.hc.client5.http.ssl.DefaultClientTlsStrategy.upgrade(DefaultClientTlsStrategy.java:48)
at org.apache.hc.client5.http.impl.nio.DefaultAsyncClientConnectionOperator.upgrade(DefaultAsyncClientConnectionOperator.java:179)
at org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManager.upgrade(PoolingAsyncClientConnectionManager.java:505)
at org.apache.hc.client5.http.impl.async.InternalHttpAsyncExecRuntime.upgradeTls(InternalHttpAsyncExecRuntime.java:249)
at org.apache.hc.client5.http.impl.async.AsyncConnectExec.proceedToNextHop(AsyncConnectExec.java:323)
at org.apache.hc.client5.http.impl.async.AsyncConnectExec.access$000(AsyncConnectExec.java:82)
at org.apache.hc.client5.http.impl.async.AsyncConnectExec$4.completed(AsyncConnectExec.java:298)
at org.apache.hc.client5.http.impl.async.AsyncConnectExec$6.completed(AsyncConnectExec.java:423)
at org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.consumeResponse(HttpAsyncMainClientExec.java:224)
at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.consumeHeader(ClientHttp1StreamHandler.java:243)
at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.consumeHeader(ClientHttp1StreamDuplexer.java:348)
at org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.consumeHeader(ClientHttp1StreamDuplexer.java:80)
at org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onInput(AbstractHttp1StreamDuplexer.java:298)
at org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.inputReady(AbstractHttp1IOEventHandler.java:64)
at org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.inputReady(ClientHttp1IOEventHandler.java:41)
at org.apache.hc.core5.reactor.ssl.SSLIOSession.decryptData(SSLIOSession.java:600)
at org.apache.hc.core5.reactor.ssl.SSLIOSession.access$200(SSLIOSession.java:74)
at org.apache.hc.core5.reactor.ssl.SSLIOSession$1.inputReady(SSLIOSession.java:202)
at org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:142)
at org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:51)
at org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(SingleCoreIOReactor.java:178)
at org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreIOReactor.java:127)
at org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:86)
at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44)
at java.base/java.lang.Thread.run(Thread.java:833
I only found this external issue, https://issues.apache.org/jira/browse/HTTPASYNC-139 that bluntly says this scenario (or one similar to it) is not supported (it doesn't seem concerned about an IllegalStateException case...).
In principle, a HTTPS CONNECT tunnel proxy is not a substantial security improvement, compared to a plain HTTP one, when the target URL is itself HTTPS. I labeled it as a bug because we should at least document this limitation, and even better, error when we detect it. Ideally we should fix it, but it might require using a different HTTP client lib (eg jetty).
Elasticsearch Version
8.*
Installed Plugins
No response
Java Version
bundled
OS Version
any
Problem Description
It's possible to configure a proxy in ES in at lest 2 places:
xpack.http.proxy.{host,port,scheme}
andxpack.security.authc.realms.oidc.<realm-name>.http.proxy.{host,port,scheme}
. The contexts in which the requests go through the proxy is outside the scope here. In all the cases when a request goes through any HTTPS proxy (the*.scheme: https
setting) the same problem, as described below, may manifests.The problem manifests when both the proxy and the target of the request use HTTPS. For example, in the oidc realm case, both
xpack.security.authc.realms.oidc.<realm-name>.http.proxy.scheme
ishttps
and thexpack.security.authc.realms.oidc.<realm-name>.op.token_endpoint
is set to a "https://" URL. The problem is that the client library that we use (group: 'org.apache.httpcomponents', name: 'httpasyncclient', version: '4.1.5') cannot handle a TLS session on top of an existing TLS session.In more details, when the
httpAsyncClientBuilder
is configured to use a proxy, it will use the HTTP CONNECT protocal. The client will connect over TLS to the proxy, and then issue a HTTP CONNECT request to it, instructing the proxy to open a TCP connection to the host of the target URL. If the proxy is successful in opening the connection to the target, it will reply 200 OK back to the client, and from that point on will forward everything from the client's connection over to the connection with the target. If the target URL also uses HTTPS, then the client must and will attempt another TLS handshake over the same connection, this time with the target host. It is at this point that the client lib barfs anIllegalStateException
because the connection already has a TLS context attached. Here's a sample stacktrace:Note that curl does work in this HTTPS over HTTPS CONNECT scenario, eg:
curl --proxy-cacert conf/certs/test-nginx-CA.crt https://github.com/ -o /dev/null -v -x https://localhost:3128
(the HTTPS connect proxy listens at localhost:3128 and has a cert signed by the test-ngnix-CA.crt).How I tested this: Firstly, I generated proxy certificates in PEM and JKS formats. Then I used nginx 1.25.2, running locally, as a HTTP CONNECT proxy. It needs an external module to support the CONNECT method: https://github.com/chobits/ngx_http_proxy_connect_module (./configure --add-module=ngx_http_proxy_connect_module --with-http_ssl_module). Here's the config:
Lastly, I extracted the client code in a separate project:
Note that I also tested the latest lib version released (group: 'org.apache.httpcomponents.client5', name: 'httpclient5', version: '5.2.1') and the same problem persists:
I only found this external issue, https://issues.apache.org/jira/browse/HTTPASYNC-139 that bluntly says this scenario (or one similar to it) is not supported (it doesn't seem concerned about an
IllegalStateException
case...).In principle, a HTTPS CONNECT tunnel proxy is not a substantial security improvement, compared to a plain HTTP one, when the target URL is itself HTTPS. I labeled it as a bug because we should at least document this limitation, and even better, error when we detect it. Ideally we should fix it, but it might require using a different HTTP client lib (eg jetty).