Issue with Netty HTTP server

[ashok-mariyala]

Expected behavior

Http client request should work without any issues when more than 50000 requests per minute.

Actual behavior

Http client requests are failing with timeout errors. Following are error messages in my Golang HTTP client. context deadline exceeded (Client.Timeout exceeded while awaiting headers) and dial tcp: i/o timeout Same http client code works for other http servers with 50000+ requests per minute.

Steps to reproduce

Minimal yet complete reproducer code (or URL to code)

Netty HTTP Server initialization code

public void initService() throws Exception {
        if(isNettyFlagEnabled()) {
            NioEventLoopGroup bossGroup = new NioEventLoopGroup();
            NioEventLoopGroup workerGroup = new NioEventLoopGroup();
            ServerBootstrap bootstrap = new ServerBootstrap();

            try {
                bootstrap.group(bossGroup, workerGroup).channel(NioServerSocketChannel.class)
                        .childHandler(new ChannelInitializer<SocketChannel>() {
                            @Override
                            public void initChannel(SocketChannel ch) throws Exception {
                                ch.pipeline().addLast("codec", new HttpServerCodec());
                                ch.pipeline().addLast("aggregator", new HttpObjectAggregator(1048576));
                                ch.pipeline().addLast("handler", new MyHttpHandler());
                            }
                        }).option(ChannelOption.SO_BACKLOG, 1024).option(ChannelOption.SO_KEEPALIVE, true);
                SocketAddress bindAddress = new InetSocketAddress("0.0.0.0", port);
                channelFuture = bootstrap.bind(bindAddress).sync();
                LOG.error("Successfully initialized netty http server on port : {}", port);
            } catch (Exception e) {
                Thread.currentThread().interrupt();
                LOG.error("Failed to initialize netty Http server. Error : {}", e.getMessage(), e);
            }
        } else {
            LOG.error("No need to init netty http server instance.");
        }
    }

Netty version

4.1.109.Final

JVM version (e.g. java -version)

Java 11

OS version (e.g. uname -a)

Ubuntu 20

[hyperxpro]

            NioEventLoopGroup bossGroup = new NioEventLoopGroup();
            NioEventLoopGroup workerGroup = new NioEventLoopGroup();

Increase the thread count and benchmark it again.

[normanmaurer]

@ashok-mariyala please show your MyHttpHandler also did you check if the EventLoop is blocked ?

[ashok-mariyala]

@normanmaurer Here is my http handler

import static io.netty.buffer.Unpooled.copiedBuffer;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import io.netty.channel.ChannelFuture;
import io.netty.channel.ChannelFutureListener;
import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.ChannelInboundHandlerAdapter;
import io.netty.handler.codec.http.DefaultFullHttpResponse;
import io.netty.handler.codec.http.FullHttpRequest;
import io.netty.handler.codec.http.HttpResponseStatus;
import io.netty.handler.codec.http.HttpVersion;
import io.netty.util.ReferenceCountUtil;

public class MyHttpHandler extends ChannelInboundHandlerAdapter {
    private static final Logger LOG = LoggerFactory.getLogger(MyHttpHandler.class);

    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
        if (msg instanceof FullHttpRequest) {
            final FullHttpRequest request = (FullHttpRequest) msg;

            try {
                String uri = HttpRequestUtils.getPath(request.uri());
                HttpServiceInfo serviceInfo = HttpResolver.getInstance().resolveService(uri);
                HttpService service = serviceInfo.getHttpService();
                String requestAction = serviceInfo.getRequestAction();

                try {
                    if (service != null) {
                        HttpResponse response = service.handleRequest(new HttpRequest(request, ctx, requestAction), ctx,
                                request, (FullHttpRequest) msg);
                        if (response != null) {
                            ChannelFuture future = ctx.writeAndFlush(response.getResponse());
                            // Close the non-keep-alive connection after the write operation is done.
                            if (!(service.isKeepAlive() || response.isKeepAlive())) {
                                future.addListener(ChannelFutureListener.CLOSE);
                            }
                        }
                    } else {
                        io.netty.handler.codec.http.DefaultHttpResponse response = new io.netty.handler.codec.http.DefaultHttpResponse(
                                HttpVersion.HTTP_1_1, HttpResponseStatus.NOT_FOUND);
                        /*
                         * Disable cache for any requests on the connection grid HTTP server
                         */
                        response.headers().set("Cache-Control", "max-age=0, must-revalidate");
                        /*
                         * Handling the XSS cross site scripting security headers in the http response
                         * X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN X-XSS-Protection:
                         * 1; mode=block
                         */
                        response.headers().set("X-Content-Type-Options", "nosniff");
                        response.headers().set("X-Frame-Options", "SAMEORIGIN");
                        response.headers().set("X-XSS-Protection", "1; mode=block");
                        ChannelFuture future = ctx.writeAndFlush(response);
                        // Close the non-keep-alive connection after the write operation is done.
                        future.addListener(ChannelFutureListener.CLOSE);
                    }
                } finally {
                    ReferenceCountUtil.release(msg);
                    if (service != null)
                        service.serviceFinalize(ctx.channel());
                }
            } catch (Exception e) {
                LOG.error("Error while reading the data from channel. Error : {}", e.getMessage(), e);
            }
        } else {
            super.channelRead(ctx, msg);
        }
    }

    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) throws Exception {
        ctx.writeAndFlush(new DefaultFullHttpResponse(HttpVersion.HTTP_1_1, HttpResponseStatus.INTERNAL_SERVER_ERROR,
                copiedBuffer(cause.getMessage().getBytes())));
    }

}

Here based on url I am invoking respective method and return the method value to the client.

[ashok-mariyala]

          NioEventLoopGroup bossGroup = new NioEventLoopGroup();
          NioEventLoopGroup workerGroup = new NioEventLoopGroup();

Increase the thread count and benchmark it again.

@hyperxpro What is the default thread count and what is best practice production ready thread count that i need to use in above code?

[hyperxpro]

The default is 1. For production, you should keep it as the number of cores. Use Runtime.getRuntime().availableProcessors() for that.

[hyperxpro]

Also, your handler has

HttpResponse response = service.handleRequest(new HttpRequest(request, ctx, requestAction), ctx,
                                request, (FullHttpRequest) msg);

Are you passing the request somewhere else that maybe result in blocking?

[ashok-mariyala]

Also, your handler has

HttpResponse response = service.handleRequest(new HttpRequest(request, ctx, requestAction), ctx,
                              request, (FullHttpRequest) msg);

Are you passing the request somewhere else that maybe result in blocking?

@hyperxpro Yes I am just redirecting the request to actual implementation of the request. This request simply getting data from data store and return the data to the client.

[ashok-mariyala]

The default is 1. For production, you should keep it as the number of cores. Use Runtime.getRuntime().availableProcessors() for that.

@hyperxpro Here number of processors and number of threads both are different right? You are asking me to increase the thread count in the previous comment.

[normanmaurer]

Also, your handler has

HttpResponse response = service.handleRequest(new HttpRequest(request, ctx, requestAction), ctx,
                                request, (FullHttpRequest) msg);

Are you passing the request somewhere else that maybe result in blocking?

@hyperxpro Yes I am just redirecting the request to actual implementation of the request. This request simply getting data from data store and return the data to the client.

Maybe this data store "blocks" ? Attach profiler and check where the CPU is spent

[hyperxpro]

The default is 1. For production, you should keep it as the number of cores. Use Runtime.getRuntime().availableProcessors() for that.

@hyperxpro Here number of processors and number of threads both are different right? You are asking me to increase the thread count in the previous comment.

I mean, increase thread count to the number of processors (total cores in general).

[XiHR123]

Could you provide the stack trace that occurs when the error happens? Regarding your issue, I speculate that it might stem from configuration discrepancies between the two servers. Try explicitly setting the number of threads for the NioEventLoopGroup and run further tests on both servers to see if the same problem persists.