search

scalecube / scalecube-gateway

ScaleCube API Gateway is the single entry point for service consumers. handles incoming requests and proxy/route to the appropriate microservice instance.
Apache License 2.0
18 stars 7 forks source link

Unexpected message type #60

Open dmytro-lazebnyi opened 5 years ago

dmytro-lazebnyi commented 5 years ago
Stacktrace ``` E 2019-09-09T11:55:00,269 i.s.s.g.w.WebsocketGatewayAcceptor Exception occurred on request: GatewayMessage {headers: {q=/om2.exchange.auth/createSession, sessionId=1568023078647, sid=1}, data: bb-670}, session=1568023078647 [http-gateway-client-epoll-4] ... 25 more at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final] at io.netty.handler.codec.http.HttpClientCodec$Encoder.encode(HttpClientCodec.java:167) ~[netty-codec-http-4.1.36.Final.jar:4.1.36.Final] at io.netty.handler.codec.http.HttpObjectEncoder.encode(HttpObjectEncoder.java:86) ~[netty-codec-http-4.1.36.Final.jar:4.1.36.Final] Caused by: java.lang.IllegalStateException: unexpected message type: DefaultFullHttpRequest, state: 1 at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.36.Final.jar:4.1.36.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) ~[netty-common-4.1.36.Final.jar:4.1.36.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:338) ~[netty-transport-native-epoll-4.1.36.Final-linux-x86_64.jar:4.1.36.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:405) ~[netty-common-4.1.36.Final.jar:4.1.36.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.36.Final.jar:4.1.36.Final] at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.run(PooledConnectionProvider.java:566) ~[reactor-netty-0.8.8.RELEASE.jar:0.8.8.RELEASE] at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:470) ~[reactor-netty-0.8.8.RELEASE.jar:0.8.8.RELEASE] at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:441) ~[reactor-netty-0.8.8.RELEASE.jar:0.8.8.RELEASE] at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE] at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:51) ~[reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE] at reactor.netty.NettyOutbound.subscribe(NettyOutbound.java:305) ~[reactor-netty-0.8.8.RELEASE.jar:0.8.8.RELEASE] at reactor.core.publisher.Mono.subscribe(Mono.java:3710) ~[reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE] at reactor.core.publisher.MonoSubscriberContext.subscribe(MonoSubscriberContext.java:47) ~[reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE] at reactor.core.publisher.MonoFlatMap.subscribe(MonoFlatMap.java:53) ~[reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE] at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:141) ~[reactor-core-3.2.9.RELEASE.jar:3.2.9.RELEASE] at reactor.netty.http.HttpOperations.lambda$send$0(HttpOperations.java:104) ~[reactor-netty-0.8.8.RELEASE.jar:0.8.8.RELEASE] at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:305) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final] at io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1036) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final] at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:812) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final] at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:757) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:789) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:763) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:716) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final] at io.netty.channel.CombinedChannelDuplexHandler.write(CombinedChannelDuplexHandler.java:348) ~[netty-transport-4.1.36.Final.jar:4.1.36.Final] at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:107) ~[netty-codec-4.1.36.Final.jar:4.1.36.Final] io.netty.handler.codec.EncoderException: java.lang.IllegalStateException: unexpected message type: DefaultFullHttpRequest, state: 1 ```
artem-v commented 5 years ago

Fixed at https://github.com/scalecube/scalecube-gateway/pull/63

artem-v commented 5 years ago

Tried to reproduce it in https://github.com/scalecube/scalecube-gateway/pull/89. Instead found a buffer leak (and fixed it) on new tests. The scenarios (on all gateway-client-transport protocols) to verify a situation when ServiceCall is calling gateway address but on that address nobody is listening.

artem-v commented 5 years ago

This one is related -- https://github.com/spring-cloud/spring-cloud-gateway/issues/280. Looks like tihs is prroblem with reactor-netty bug in ChannelOoperations.onComplete() not calling onOutboundComplete and not sending LastHttpContent to HttpObjectEncoder, and by turn leaving http channel in broken state (i.e still INIT).

artem-v commented 4 years ago

Found very much related issues: https://github.com/reactor/reactor-netty/issues/361, https://github.com/reactor/reactor-netty/issues/177.

Looping @snripa. From what ppl already posting to them in existing issues is very much looks like (IMHO) that reactor-netty doesn't send LastHttpContent neither from HttpClientOperations#onOutboundComplete() neither from HttpServerOperations#onOutboundComplete(). Though both began from reactor.netty.channel.ChannelOperations#onComplete(). Looks like this block in ChannelOperations on lines 197-199 (on reactory-netty master branch) is the root cause: 

    @Override
    public final void onComplete() {
        if (isDisposed()) {
            return;
        }
               ...
    }

ChannelOperations impl. Subscriber, that's why his onComplete() function must adhere to semantics of org.reactivestreams.Subscriber#onComplete(), i.e dispose on exception (among everything else). Hence it very looks like that when exception occurred it doesn't send LastHttpContent to io.netty.handler.codec.http.HttpObjectEncoder there by leaving him in wrong state.

I suggest as a resolution to this ticket is to write test psmv program which would reproduce this issue and send it to them + make sure they accepted it.