search

pinpoint-apm / pinpoint

APM, (Application Performance Management) tool for large-scale distributed systems.
https://pinpoint-apm.gitbook.io/
Apache License 2.0
13.29k stars 3.75k forks source link

How to solve the problem of 431 HTTP Code (Request Header Fields Too Large) in the environment using Istio #10016

Open gilbertlim opened 1 year ago

gilbertlim commented 1 year ago

In an environment using Istio, if an envoy container is created as a sidecar in the application container where pinpoint-agent operates, the application container communicates with the pintpoint-collector through the envoy proxy.


version


At this time, An HTTP status code 431 error occurs on the client (application-container). 431 code means 'Request Header Fields Too Large'.

Probably because envoy adds multiple headers.

06-04 10:57:42.042 [-Executor(13-0)] WARN c.n.p.p.r.g.GrpcCommandService -- Failed to command stream, cause=INTERNAL: HTTP status code 431
  invalid content-type: null
  trailers: Metadata(:status=431,date=Sun, 04 Jun 2023 10:57:41 GMT,server=envoy)
  io.grpc.StatusRuntimeException: INTERNAL: HTTP status code 431
  invalid content-type: null
  trailers: Metadata(:status=431,date=Sun, 04 Jun 2023 10:57:41 GMT,server=envoy)
      at io.grpc.Status.asRuntimeException(Status.java:535) ~[grpc-api-1.49.2.jar:1.49.2]
      at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487) ~[grpc-stub-1.49.2.jar:1.49.2]
      at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:563) ~[grpc-core-1.49.2.jar:1.49.2]
      at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) ~[grpc-core-1.49.2.jar:1.49.2]
      at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:744) ~[grpc-core-1.49.2.jar:1.49.2]
      at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723) ~[grpc-core-1.49.2.jar:1.49.2]
      at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.49.2.jar:1.49.2]
      at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.49.2.jar:1.49.2]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
      at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]


In server(pinpoint-collector), 'Header size exceeded max allowed size' error occurs.

06-04 10:57:14.014 [el-Worker(20-0)] WARN i.g.n.NettyServerHandler : Stream Error
  io.netty.handler.codec.http2.Http2Exception$HeaderListSizeException: Header size exceeded max allowed size (1024)
      at io.netty.handler.codec.http2.Http2Exception.headerListSizeError(Http2Exception.java:195) ~[netty-codec-http2-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.handler.codec.http2.Http2CodecUtil.headerListSizeExceeded(Http2CodecUtil.java:233) ~[netty-codec-http2-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.handler.codec.http2.HpackDecoder$Http2HeadersSink.finish(HpackDecoder.java:577) ~[netty-codec-http2-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.handler.codec.http2.HpackDecoder.decode(HpackDecoder.java:132) ~[netty-codec-http2-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.handler.codec.http2.DefaultHttp2HeadersDecoder.decodeHeaders(DefaultHttp2HeadersDecoder.java:126) ~[netty-codec-http2-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.handler.codec.http2.DefaultHttp2FrameReader$HeadersBlockBuilder.headers(DefaultHttp2FrameReader.java:733) ~[netty-codec-http2-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.handler.codec.http2.DefaultHttp2FrameReader$2.processFragment(DefaultHttp2FrameReader.java:476) ~[netty-codec-http2-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:484) ~[netty-codec-http2-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:253) ~[netty-codec-http2-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159) ~[netty-codec-http2-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) ~[netty-codec-http2-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173) ~[netty-codec-http2-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378) ~[netty-codec-http2-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438) ~[netty-codec-http2-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[netty-codec-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[netty-codec-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) ~[netty-codec-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[netty-transport-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) ~[netty-transport-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) ~[netty-transport-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[netty-transport-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[netty-common-4.1.77.Final.jar!/:4.1.77.Final]
      at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.77.Final.jar!/:4.1.77.Final]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
      at java.lang.Thread.run(Unknown Source) ~[?:?]


I confirmed that the problem is caused by the difference in 'header max size' configuration of pinpoint-collector and pinpoint-agent.

The error occurred because the header size that pinpoint-agent sends through enovy proxy is larger than the header size that can be accommodated by pinpoint-collector. pinpoint-agent header size (greater than 1K) -> pinpoint-collector header size (1K)

The key to solving the problem is to increase the 'header_list_size_max' of the pinpoint-collector.



There are two solutions.


1.Temporary solution

Like https://github.com/gilbertlim/pinpoint-docker/pull/1 PR, modify pinpoint-collector.properties. Then, when creating an image, the header_list_size_max value of pinpoint-collector is overridden in the configuration.


pinpoint-collector/build/config/pinpoint-collector.properties

# ex) PINPOINT_ZOOKEEPER_ADDRESS should be commented in docker-compose to use pinpoint.zookeeper.address below
##########
#
#pinpoint.zookeeper.address=

collector.receiver.grpc.agent.header_list_size_max=8KB
collector.receiver.grpc.stat.header_list_size_max=8KB
collector.receiver.grpc.span.header_list_size_max=8KB
cd pinpoint-docker/pinpoint-collector
docker build -t pinpoint-collector:2.5.1 .
docker tag pinpoint-collector:2.5.1 9ilbert/pinpoint-collector:2.5.1


You can use the 9ilbert/pinpoint-collector:2.5.1 public image as shown in the example.


2.Solution to change default configuration https://github.com/gilbertlim/pinpoint/commit/06a1ec6f3e386e9b0db200aacc5c72c7c2a88b98

You can fix the error by changing the 'header max size' of the pinpoint-collector in the repository as shown below. Since the gap between header_list_size_max values of pinpoint-agent and pinpoint-collector is large, it seems desirable to reduce the difference or set it equal.


pinpoint-agent

pinpoint/agent/src/main/resources/pinpoint-root.config

profiler.transport.grpc.agent.sender.headers.size.max=8K
profiler.transport.grpc.metadata.sender.headers.size.max=8K
profiler.transport.grpc.stat.sender.headers.size.max=8K
profiler.transport.grpc.span.sender.headers.size.max=8K

pinpoint-collector pinpoint/collector/src/main/resources/pinpoint-collector-grpc-root.properties

collector.receiver.grpc.agent.header_list_size_max=8KB
collector.receiver.grpc.stat.header_list_size_max=8KB
collector.receiver.grpc.span.header_list_size_max=8KB



Anyway, if you get a 431 HTTP Code error, you can increase the collector's header_list_size_max.

ga-ram commented 1 year ago

As far as I know, default collector header_list_size_max is set to 1K to allow only the size of the headers created by pinpoint agent to prevent possible attack on pinpoint collector servers through headers. If it is necessary to increase the size for your environment, you can always override them with -D options or using external spring properties when starting the collector (or update the values as you did above). If you think it would be valuable to provide additional config values to docker-compose.yml and .env file in our docker project, please send PRs.