ClassCastException: class io.netty.handler.codec.http.LastHttpContent$1 cannot be cast to class io.netty.handler.codec.http.HttpRequest

[musketyr]

Expected Behavior

If LastHttpContent.EMPTY_LAST_CONTENT is passed to the method io.micronaut.http.server.netty.handler.PipeliningServerHandler.MessageInboundHandler#read then the method executes without throwing any error.

Actual Behaviour

java.lang.ClassCastException: class io.netty.handler.codec.http.LastHttpContent$1 cannot be cast to class io.netty.handler.codec.http.HttpRequest 

is thrown in io.micronaut.http.server.netty.handler.PipeliningServerHandler.MessageInboundHandler#read method because of unchecked cast to HttpRequest

Steps To Reproduce

Sadly, I don't have a reproducer. This happens randomly on each of our deployed API. It might but might not be related to the blocked attacks of Contrast Security agent but anyway the method should be ready for this.

Environment Information

No response

Example Application

No response

Version

4.5.0

[graemerocher]

do you have a more complete stack trace

[musketyr]

I do, but I can't match the line numbers for some reasons

java.lang.ClassCastException: class io.netty.handler.codec.http.LastHttpContent$1 cannot be cast to class io.netty.handler.codec.http.HttpRequest (io.netty.handler.codec.http.LastHttpContent$1 and io.netty.handler.codec.http.HttpRequest are in unnamed module of loader 'app')
    at io.micronaut.http.server.netty.handler.PipeliningServerHandler$MessageInboundHandler.read(PipeliningServerHandler.java:345)
    at io.micronaut.http.server.netty.handler.PipeliningServerHandler.channelRead(PipeliningServerHandler.java:213)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
    at io.netty.handler.codec.http.websocketx.extensions.WebSocketServerExtensionHandler.channelRead(WebSocketServerExtensionHandler.java:91)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at java.lang.ContrastRunnableWrapper.run(ContrastRunnableWrapper.java:29)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.lang.ContrastRunnableWrapper.run(ContrastRunnableWrapper.java:29)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:840)

[graemerocher]

@yawkat can you take a look?

[FrogDevelopper]

Hi We also have this issue with a ProxyHttpClient.proxy() when the request contains the Expect: 100-continue header

io.micronaut.http.client.exceptions.HttpClientException: Error occurred reading HTTP response: class io.netty.handler.codec.http.LastHttpContent$1 cannot be cast to class io.netty.handler.codec.http.HttpResponse (io.netty.handler.codec.http.LastHttpContent$1 and io.netty.handler.codec.http.HttpResponse are in unnamed module of loader 'app') at io.micronaut.http.client.netty.DefaultHttpClient$BaseHttpResponseHandler.exceptionCaught(DefaultHttpClient.java:2078) Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: Assembly trace from producer [reactor.core.publisher.FluxCreate] : reactor.core.publisher.Flux.create(Flux.java:646) io.micronaut.http.client.netty.DefaultHttpClient.streamRequestThroughChannel(DefaultHttpClient.java:1544) Error has been observed at the following site(s): ____Flux.create ⇢ at io.micronaut.http.client.netty.DefaultHttpClient.streamRequestThroughChannel(DefaultHttpClient.java:1544) | Flux.flatMap ⇢ at io.micronaut.http.client.netty.DefaultHttpClient.streamRequestThroughChannel(DefaultHttpClient.java:1550) Mono.flatMapMany ⇢ at io.micronaut.http.client.netty.DefaultHttpClient.connectAndStream(DefaultHttpClient.java:1069) | Flux.next ⇢ at io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl.(ReactorExecutionFlowImpl.java:50) | Mono.map ⇢ at io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl.map(ReactorExecutionFlowImpl.java:71) | Mono.flatMap ⇢ at io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl.flatMap(ReactorExecutionFlowImpl.java:59) | Mono.onErrorResume ⇢ at io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl.onErrorResume(ReactorExecutionFlowImpl.java:77) | Mono.map ⇢ at io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl.map(ReactorExecutionFlowImpl.java:71) __Mono.error ⇢ at io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl.toMono(ReactorExecutionFlowImpl.java:145) __Mono.from ⇢ at io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl.(ReactorExecutionFlowImpl.java:50) | Mono.map ⇢ at io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl.map(ReactorExecutionFlowImpl.java:71) | Mono.onErrorResume ⇢ at io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl.onErrorResume(ReactorExecutionFlowImpl.java:77) | Mono.map ⇢ at io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl.map(ReactorExecutionFlowImpl.java:71) | Flux.from ⇢ at io.micronaut.configuration.metrics.binder.web.WebMetricsPublisher.(WebMetricsPublisher.java:115) __Mono.error ⇢ at io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl.toMono(ReactorExecutionFlowImpl.java:145) Mono.from ⇢ at io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl.(ReactorExecutionFlowImpl.java:50) | Mono.map ⇢ at io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl.map(ReactorExecutionFlowImpl.java:71) | Mono.onErrorResume ⇢ at io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl.onErrorResume(ReactorExecutionFlowImpl.java:77) | Mono.flatMap ⇢ at io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl.flatMap(ReactorExecutionFlowImpl.java:59) | Mono.contextWrite ⇢ at io.micronaut.http.client.netty.DefaultHttpClient.applyFilterToResponsePublisher(DefaultHttpClient.java:1253) | Flux.from ⇢ at io.micronaut.http.client.netty.DefaultHttpClient.lambda$proxy$24(DefaultHttpClient.java:1037) *Mono.error ⇢ at io.micronaut.http.reactive.execution.ReactorExecutionFlowImpl.toMono(ReactorExecutionFlowImpl.java:145) ____Flux.flatMap ⇢ at io.micronaut.http.client.netty.DefaultHttpClient.proxy(DefaultHttpClient.java:1027) |_ Mono.from ⇢ at com.xxx.yyy.gateway.http.filter.ApiFilter.lambda$proxy$0(ApiFilter.java:46) ____Mono.flatMap ⇢ at com.xxx.yyy.gateway.http.filter.ApiFilter.doFilter(ApiFilter.java:39) |_ Mono.switchIfEmpty ⇢ at com.xxx.yyy.gateway.http.filter.ApiFilter.doFilter(ApiFilter.java:40) Original Stack Trace: at io.micronaut.http.client.netty.DefaultHttpClient$BaseHttpResponseHandler.exceptionCaught(DefaultHttpClient.java:2078) at io.micronaut.http.client.netty.DefaultHttpClient$StreamHttpResponseHandler.exceptionCaught(DefaultHttpClient.java:2426) ... (removing)

[yawkat]

but anyway the method should be ready for this.

No, it is intentional that this is not accepted. It indicates that there is a difference between the connection state of the PipeliningServerHandler and the HttpServerRequestDecoder. Doing it differently would just hide another bug. I can't tell what that bug is with just this information, but you could try adding a logging handler to find out more.

@FrogDevelopper That's a client error, please report it separately.

[musketyr]

thanks for the response @yawkat. but wouldn't in that case be more convenient to throw some dedicated exception?

but you could try adding a logging handler to find out more.

can you please, more elaborate on this? I'm not that familiar on Netty internals

[musketyr]

here's the breadcrumbs from Sentry

[yawkat]

I find it very likely that the contrast security agent is at fault here. You can see it block an attack immediately before the error. From the product description, it seems they hook into netty to block attacks. Likely they are blocking only some parts of the HTTP request in the netty pipeline (the HttpRequest object), and micronaut only sees the leftovers.

Unfortunately the agent is not open source, and the license specifically forbids reverse engineering, so I cannot look into it further.

If you can reproduce this issue without the agent, feel free to reopen. However an agent bug is very likely, so please talk to the contrast security folks.

[musketyr]

ok, thanks for your feedback. Is there still some chance to throw dedicated exception instead of ClassCastException for these use cases that we can use to filter them out in our error inbox?

[yawkat]

I don't really want to add more failure handling there because the code is very hot