How to use SSL proxy with AHC

xargsgrep commented 1 year ago

I have an SSL proxy setup which works fine when using curl.

$ curl -x https://test-proxy.com:443 http://google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

I can't seem to figure out how to configure AHC with a ProxyServer instance to do the same. I've tried an older version of AHC and a newer one (see below for configurations). In both cases when I make a request it just hangs.

Configuration for 1.7.x:

    var proxyServer = new ProxyServer(Protocol.HTTPS, "test-proxy.com", 443);
    var config = new AsyncHttpClientConfig.Builder().setProxyServer(proxyServer).build();
    var client = new AsyncHttpClient(config);
    var request = client.prepareGet("http://google.com").build();

Configuration for 2.12.x:

    var proxyServer = new ProxyServer.Builder("test-proxy.com", 80).setSecuredPort(443).build();
    var config = new Builder().setProxyServer(proxyServer).build();
    var client = new DefaultAsyncHttpClient(config);
    var request = client.prepareGet("http://google.com").build();

hyperxpro commented 1 year ago

What's the error you're getting?

xargsgrep commented 1 year ago

The request just times out. However, it's not a connectivity issue because as mentioned the curl command works just fine.

↳ curl -x https://proxy.xargsgrep.com:443 google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

If you look at the output below you'll see it's trying to connect to proxy.xargsgrep.com/44.219.62.231:80. But I want to use SSL for the proxy always, regardless of the schema of the target URL. How do I force AHC 2.12.x to use SSL for the proxy always? The ProxyServer.Builder class has a setSecuredPort method but that's about it.

15:43:07.941 [AsyncHttpClient-3-1] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.checkAccessible: true
15:43:07.942 [AsyncHttpClient-3-1] DEBUG io.netty.buffer.AbstractByteBuf - -Dio.netty.buffer.checkBounds: true
15:43:07.942 [AsyncHttpClient-3-1] DEBUG i.n.util.ResourceLeakDetectorFactory - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@7fa95969
15:43:12.955 [AsyncHttpClient-3-1] DEBUG o.a.n.channel.NettyConnectListener - Trying to recover from failing to connect channel [id: 0x7f9748f7, L:null ! R:proxy.xargsgrep.com/44.219.62.231:80] with a retry value of true 
15:43:12.962 [AsyncHttpClient-3-1] DEBUG o.a.n.channel.NettyConnectListener - Failed to recover from connect exception: io.netty.channel.ConnectTimeoutException: connection timed out: proxy.xargsgrep.com/44.219.62.231:80 with channel [id: 0x7f9748f7, L:null ! R:proxy.xargsgrep.com/44.219.62.231:80]
15:43:12.964 [AsyncHttpClient-3-1] DEBUG o.a.AsyncCompletionHandler - connection timed out: proxy.xargsgrep.com/44.219.62.231:80
java.net.ConnectException: connection timed out: proxy.xargsgrep.com/44.219.62.231:80
    at org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:179)
    at org.asynchttpclient.netty.channel.NettyChannelConnector$1.onFailure(NettyChannelConnector.java:108)
    at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:28)
    at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:20)
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)
    at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
    at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
    at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)
    at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:262)
    at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
    at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: proxy.xargsgrep.com/44.219.62.231:80
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261)
    ... 9 common frames omitted
Exception in thread "main" java.util.concurrent.ExecutionException: java.net.ConnectException: connection timed out: proxy.xargsgrep.com/44.219.62.231:80
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
    at com.twilio.ahc.test.AsyncHC2x.main(AsyncHC2x.java:28)
Caused by: java.net.ConnectException: connection timed out: proxy.xargsgrep.com/44.219.62.231:80
    at org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:179)
    at org.asynchttpclient.netty.channel.NettyChannelConnector$1.onFailure(NettyChannelConnector.java:108)
    at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:28)
    at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:20)
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)
    at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
    at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
    at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)
    at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:262)
    at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
    at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: proxy.xargsgrep.com/44.219.62.231:80
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261)
    ... 9 more

AshwinPrabhuB commented 1 year ago

In order to work with proxies, you first need to set org.asynchttpclient.useProxyProperties=true in org/asynchttpclient/config/ahc.properties file in your jar.

With the above instructions, AHC would pick up the proxy from the system properties -Dhttp.proxyHost=? -Dhttp.proxyPort=? -Dhttps.proxyHost=? -Dhttps.proxyPort=?

xargsgrep commented 1 year ago

@AshwinPrabhuB The problem is not that AHC is not picking up the proxy settings. I am able to configure it via the code as shown in my original post. The problem is that AHC does not work when an https proxy is configured.

xeron commented 9 months ago

From my testing:

AHC v1 sends plain-text request to HTTPS proxy, this is why it fails.
AHC v2 sends the request to HTTPS proxy (to securedPort) only when request URL protocol is https. But seems like it's also plain-text so it also fails.

Seems like someone tried to fix it in this PR – https://github.com/AsyncHttpClient/async-http-client/pull/1795

AsyncHttpClient / async-http-client

How to use SSL proxy with AHC #1907