AsyncHttpClient / async-http-client

Asynchronous Http and WebSocket Client library for Java
Other
6.26k stars 1.59k forks source link

The code seems to spam org.asynchttpclient.netty.channel.NettyChannelConnector$1.operationComplete() #1891

Open couldbejake opened 1 year ago

couldbejake commented 1 year ago

I am opening a large amount of requests using AsyncHttpClient, and seem to run into a large amount of console spam whilst the program is running.

I have noticed that this occurs on Windows, but not on Linux.

Version: Async-http-client version '3.0.0.Beta2'

Preview:

[AsyncHttpClient-9-6] WARN io.netty.util.concurrent.DefaultPromise - An exception was thrown by org.asynchttpclient.netty.channel.NettyChannelConnector$1.operationComplete()
java.lang.NullPointerException: Cannot invoke "org.asynchttpclient.netty.timeout.TimeoutsHolder.setResolvedRemoteAddress(java.net.InetSocketAddress)" because "timeoutsHolder" is null
    at org.asynchttpclient.netty.channel.NettyConnectListener.onSuccess(NettyConnectListener.java:100)
    at org.asynchttpclient.netty.channel.NettyChannelConnector$1.onSuccess(NettyChannelConnector.java:91)
    at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:28)
    at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:22)
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
    at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
    at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
    at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
    at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105)
    at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:300)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:335)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:1589)
[AsyncHttpClient-17-14] WARN io.netty.util.concurrent.DefaultPromise - An exception was thrown by org.asynchttpclient.netty.channel.NettyChannelConnector$1.operationComplete()
java.lang.NullPointerException: Cannot invoke "org.asynchttpclient.netty.timeout.TimeoutsHolder.setResolvedRemoteAddress(java.net.InetSocketAddress)" because "timeoutsHolder" is null
    at org.asynchttpclient.netty.channel.NettyConnectListener.onSuccess(NettyConnectListener.java:100)
    at org.asynchttpclient.netty.channel.NettyChannelConnector$1.onSuccess(NettyChannelConnector.java:91)
    at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:28)
    at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:22)
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)
    at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)
    at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)
    at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625)
    at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105)
    at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:300)
    at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:335)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:1589)
hyperxpro commented 1 year ago

Is this happening under heavy load?

couldbejake commented 1 year ago

Is this happening under heavy load?

This is indeed happening under heavy load (800+ requests per second). I chose this library as it's performant, but it seems to also causes memory leaks when running continuously. The code https://github.com/Scrapium/scrape-tweets-dev-2 (The memory leak might be caused by the uncaught exception)

couldbejake commented 1 year ago

It's seems to be due to the way AsyncHttpClient is interacting with the netty framework This library is my only option, as it allows me to set the proxy of the request, rather than the client.

hyperxpro commented 1 year ago

Can you share basic reproducer?