search

alibaba / Sentinel

A powerful flow control component enabling reliability, resilience and monitoring for microservices. (面向云原生微服务的高可用流控防护组件)
https://sentinelguard.io/
Apache License 2.0
22.41k stars 8.03k forks source link

webflux io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1 #2259

Open RekaDowney opened 3 years ago

RekaDowney commented 3 years ago

Type: bug report

Describe what happened (or what feature you want)

We use sentinel in Spring Cloud Gateway, and found that when backend response incorrently,  SentinelReactorSubscriber will produce `IllegalReferenceCountException: refCnt: 0, decrement: 1`.

StackTrace

Operator called default onErrorDropped
reactor.core.Exceptions$BubblingException: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
    at reactor.core.Exceptions.bubble(Exceptions.java:173)
    at reactor.core.publisher.Operators.onErrorDropped(Operators.java:635)
    at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.runFinally(FluxDoFinally.java:159)
    at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onComplete(FluxDoFinally.java:139)
    at com.alibaba.csp.sentinel.adapter.reactor.SentinelReactorSubscriber.hookOnComplete(SentinelReactorSubscriber.java:136)
    at com.alibaba.csp.sentinel.adapter.reactor.InheritableBaseSubscriber.onComplete(InheritableBaseSubscriber.java:191)
    at com.alibaba.csp.sentinel.adapter.reactor.SentinelReactorSubscriber.onComplete(SentinelReactorSubscriber.java:37)
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.secondComplete(MonoFlatMap.java:189)
    at reactor.core.publisher.MonoFlatMap$FlatMapInner.onComplete(MonoFlatMap.java:260)
    at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:252)
    at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:207)
    at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onComplete(MonoPeekTerminal.java:292)
    at org.springframework.http.server.reactive.ChannelSendOperator$WriteCompletionBarrier.onComplete(ChannelSendOperator.java:423)
    at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:81)
    at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onComplete(FluxConcatArray.java:191)
    at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:207)
    at reactor.netty.channel.MonoSendMany$SendManyInner.run(MonoSendMany.java:345)
    at reactor.netty.channel.MonoSendMany$SendManyInner.trySchedule(MonoSendMany.java:419)
    at reactor.netty.channel.MonoSendMany$SendManyInner.trySuccess(MonoSendMany.java:562)
    at reactor.netty.channel.MonoSendMany$SendManyInner.trySuccess(MonoSendMany.java:114)
    at io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48)
    at io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717)
    at io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272)
    at io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352)
    at io.netty.channel.epoll.AbstractEpollChannel.doWriteBytes(AbstractEpollChannel.java:364)
    at io.netty.channel.epoll.AbstractEpollStreamChannel.writeBytes(AbstractEpollStreamChannel.java:260)
    at io.netty.channel.epoll.AbstractEpollStreamChannel.doWriteSingle(AbstractEpollStreamChannel.java:471)
    at io.netty.channel.epoll.AbstractEpollStreamChannel.doWrite(AbstractEpollStreamChannel.java:429)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:953)
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.flush0(AbstractEpollChannel.java:557)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:917)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372)
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742)
    at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.flush(CombinedChannelDuplexHandler.java:531)
    at io.netty.channel.ChannelOutboundHandlerAdapter.flush(ChannelOutboundHandlerAdapter.java:125)
    at io.netty.channel.CombinedChannelDuplexHandler.flush(CombinedChannelDuplexHandler.java:356)
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
    at io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742)
    at io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728)
    at reactor.netty.channel.MonoSendMany$SendManyInner$AsyncFlush.run(MonoSendMany.java:758)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
    at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)
    at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)
    at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100)
    at io.netty.buffer.AbstractPooledDerivedByteBuf.deallocate(AbstractPooledDerivedByteBuf.java:86)
    at io.netty.buffer.AbstractReferenceCountedByteBuf.handleRelease(AbstractReferenceCountedByteBuf.java:110)
    at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100)
    at org.springframework.core.io.buffer.NettyDataBuffer.release(NettyDataBuffer.java:320)
    at org.springframework.cloud.gateway.filter.RemoveCachedBodyFilter.lambda$filter$0(RemoveCachedBodyFilter.java:43)
    at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.runFinally(FluxDoFinally.java:156)
    ... 46 more

Describe what you expected to happen

Can sentinel handle this kind of exception, or should I handle it through Hooks.onErrorDropped by myself.

How to reproduce it (as minimally and precisely as possible)

  1. backend service response incorrent.

Tell us your environment

spring-cloud-starter-gateway: 2.2.8.RELEASE spring-cloud-alibaba-sentinel-gateway: 2.2.3.RELEASE sentinel-reactor-adapter: 1.8.0

sczyh30 commented 3 years ago

Hi, it seems like this error was emitted by Reactor Netty and caught by Sentinel Reactor Adapter (and "bubbled up"). Maybe we could provide an option to drop the exception by default.