search

micronaut-projects / micronaut-core

Micronaut Application Framework
http://micronaut.io
Apache License 2.0
6.07k stars 1.07k forks source link

Reference Counting Error using eventStream() on DefaultHttpClient #2558

Closed dthoffdev closed 4 years ago

dthoffdev commented 4 years ago

Thanks for reporting an issue for Micronaut, please review the task list below before submitting the issue. Your issue report will be closed if the issue is incomplete and the below tasks not completed.

NOTE: If you are unsure about something and the issue is more of a question a better place to ask questions is on Stack Overflow (http://stackoverflow.com/tags/micronaut) or Gitter (https://gitter.im/micronautfw/). DO NOT use the issue tracker to ask questions.

Task List

Steps to Reproduce

  1. checkout https://github.com/dhoff1985/micronaut-stream-message-size.git
  2. Run ReproduceStreamClientErrorSpec. a. controller serves Observable of 4 180 character events b. test consumes this using DefaultHttpClient.eventStream() c. 4th message throws a reference counting error
  3. See the error

Expected Behaviour

The 4th event should be read without error

Actual Behaviour

I get an error reading the 4th event

Error consuming Server Sent Events: refCnt: 0, decrement: 1
io.micronaut.http.client.exceptions.HttpClientException: Error consuming Server Sent Events: refCnt: 0, decrement: 1
    at io.micronaut.http.client.DefaultHttpClient$4.onError(DefaultHttpClient.java:644)
    at io.reactivex.internal.util.HalfSerializer.onError(HalfSerializer.java:70)
    at io.reactivex.internal.subscribers.StrictSubscriber.onError(StrictSubscriber.java:103)
    at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.lambda$onError$1(InstrumentedSubscriber.java:96)
    at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.onError(InstrumentedSubscriber.java:100)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.checkTerminate(FlowableFlatMap.java:567)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drainLoop(FlowableFlatMap.java:374)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.drain(FlowableFlatMap.java:366)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap$MergeSubscriber.innerError(FlowableFlatMap.java:606)
    at io.reactivex.internal.operators.flowable.FlowableFlatMap$InnerSubscriber.onError(FlowableFlatMap.java:672)
    at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.lambda$onError$1(InstrumentedSubscriber.java:96)
    at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.onError(InstrumentedSubscriber.java:100)
    at io.reactivex.internal.operators.flowable.FlowableSwitchMap$SwitchMapSubscriber.drain(FlowableSwitchMap.java:221)
    at io.reactivex.internal.operators.flowable.FlowableSwitchMap$SwitchMapInnerSubscriber.onError(FlowableSwitchMap.java:403)
    at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.lambda$onError$1(InstrumentedSubscriber.java:96)
    at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.onError(InstrumentedSubscriber.java:100)
    at io.reactivex.internal.subscribers.BasicFuseableSubscriber.onError(BasicFuseableSubscriber.java:101)
    at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.lambda$onError$1(InstrumentedSubscriber.java:96)
    at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.onError(InstrumentedSubscriber.java:100)
    at io.reactivex.internal.subscribers.BasicFuseableSubscriber.onError(BasicFuseableSubscriber.java:101)
    at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.lambda$onError$1(InstrumentedSubscriber.java:96)
    at io.micronaut.reactive.rxjava2.InstrumentedSubscriber.onError(InstrumentedSubscriber.java:100)
    at io.micronaut.http.netty.reactive.HandlerPublisher.exceptionCaught(HandlerPublisher.java:525)
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:297)
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:276)
    at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:268)
    at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
    at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:297)
    at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:831)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:376)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:326)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:313)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:427)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
    at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
    at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
    at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
    at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100)
    at io.netty.handler.codec.http.DefaultHttpContent.release(DefaultHttpContent.java:94)
    at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:88)
    at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:106)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
    ... 28 more

Environment Information

Example Application

https://github.com/dhoff1985/micronaut-stream-message-size.git

vroyer commented 4 years ago

Same issue with micronaut 1.3.3, java "12.0.2" 2019-07-16, macOS.

juliuskrah commented 4 years ago

I get a similar error when I proxy a WS connection through an API gateway (ocelot)

io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
        at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
        at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
        at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100)
        at io.netty.buffer.DefaultByteBufHolder.release(DefaultByteBufHolder.java:115)
        at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:88)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:106)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:834)
graemerocher commented 4 years ago

@juliuskrah can you provide an example that reproduces the issue?

juliuskrah commented 4 years ago

@graemerocher I have added a demo

https://github.com/juliuskrah/playground/tree/websocket

vinhbt commented 4 years ago

I get error when using wss websocket client io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1 at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74) at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138) at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100) at io.netty.buffer.DefaultByteBufHolder.release(DefaultByteBufHolder.java:115) at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:88) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:106) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1518) at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:834)

jameskleeh commented 4 years ago

@dhoff1985 Thanks for the report. I can't reproduce with Micronaut 2. I've submitted a PR to your sample project

juliuskrah commented 4 years ago

@jameskleeh I provided a sample here that reproduces the error

jameskleeh commented 4 years ago

That isn’t the same issue. The original reported issue was about event streams

On Thu, Jul 16, 2020 at 6:05 PM Julius Krah notifications@github.com wrote:

@jameskleeh https://github.com/jameskleeh I provided a sample here https://github.com/micronaut-projects/micronaut-core/issues/2558#issuecomment-632084241 that reproduces the error

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/micronaut-projects/micronaut-core/issues/2558#issuecomment-659700209, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMCVLP7KO6T4XQ2VKNYBB3R352T3ANCNFSM4J6U7CGA .

juliuskrah commented 4 years ago

Should this be opened as a separate issue?

On Thu, Jul 16, 2020, 22:19 James Kleeh notifications@github.com wrote:

That isn’t the same issue. The original reported issue was about event streams

On Thu, Jul 16, 2020 at 6:05 PM Julius Krah notifications@github.com wrote:

@jameskleeh https://github.com/jameskleeh I provided a sample here < https://github.com/micronaut-projects/micronaut-core/issues/2558#issuecomment-632084241

that reproduces the error

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/micronaut-projects/micronaut-core/issues/2558#issuecomment-659700209 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAMCVLP7KO6T4XQ2VKNYBB3R352T3ANCNFSM4J6U7CGA

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/micronaut-projects/micronaut-core/issues/2558#issuecomment-659706044, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACLY5PNLZVU4PPUOFXFVR2TR354IBANCNFSM4J6U7CGA .

vinhbt commented 4 years ago

When i change some code in AbstractNettyWebSocketHandler class

protected void handleWebSocketFrame(ChannelHandlerContext ctx, WebSocketFrame msg) { .... else if (msg instanceof PingWebSocketFrame) { // respond with pong // PingWebSocketFrame frame = (PingWebSocketFrame) msg; -> PingWebSocketFrame frame = (PingWebSocketFrame) msg.copy(); ctx.writeAndFlush(new PongWebSocketFrame(frame.content()));

    } else if (msg instanceof PongWebSocketFrame) {

}

Exception is not occur.