scalecube / scalecube-cluster

ScaleCube Cluster is a lightweight Java VM implementation of SWIM: Scalable Weakly-consistent Infection-style Process Group Membership Protocol. features cluster membership, failure detection, and gossip protocol library.
http://scalecube.github.io/
Apache License 2.0
263 stars 88 forks source link

reactor.core.Exceptions$ReactorRejectedExecutionException: Scheduler unavailable #44

Closed artem-v closed 6 years ago

artem-v commented 6 years ago

Just recently were about running c5.2xlarge perf test with 1 client 1 gw and 1 service. Test didn't start and failed with following in logs.

On the client:

I 2018-10-27T20:46:08,358 i.s.b.BenchmarkState Benchmarks settings: BenchmarkSettings{numberThreads=8, concurrency=16, minInterval=PT0.1S, executionTaskDuration=PT6M, executionTaskInterval=PT8S, numOfIterations=9223372036854775807, reporterInterval=PT3S, csvReporterDirectory=reports/benchmarks/i.s.g.b.r.r.RemoteInfiniteStreamBenchmark/2018-10-27-20-46-08, taskName='i.s.g.b.r.r.RemoteInfiniteStreamBenchmark', durationUnit=MILLISECONDS, rateUnit=SECONDS, warmUpDuration=PT1M, rampUpDuration=PT8S, rampUpInterval=PT1S, consoleReporterEnabled=true, injectors=8, messageRate=1, injectorsPerRampUpInterval=1, messagesPerExecutionInterval=1, options={rateLimit=32, gatewayHost=10.200.2.40}} [main]
I 2018-10-27T20:46:09,851 i.s.g.c.r.RSocketClientTransport Connected successfully on 10.200.2.40:9090 [worker-client-sdk-client-epoll-9]
I 2018-10-27T20:46:09,851 i.s.g.c.r.RSocketClientTransport Connected successfully on 10.200.2.40:9090 [worker-client-sdk-client-epoll-10]
E 2018-10-27T20:46:09,970 i.s.b.BenchmarkState Exception occured on setUp at rampUpIteration: 0, cause: io.rsocket.exceptions.ApplicationErrorException: No reachable member with such service: /benchmarks/one, task won't start [worker-client-sdk-client-epoll-9]
io.rsocket.exceptions.ApplicationErrorException: No reachable member with such service: /benchmarks/one
    at io.rsocket.exceptions.Exceptions.from(Exceptions.java:53) ~[rsocket-core-0.11.8.jar:?]
    at io.rsocket.RSocketClient.handleFrame(RSocketClient.java:456) ~[rsocket-core-0.11.8.jar:?]
    at io.rsocket.RSocketClient.handleIncomingFrames(RSocketClient.java:419) ~[rsocket-core-0.11.8.jar:?]
    at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:130) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:238) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.drainRegular(FluxGroupBy.java:554) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.drain(FluxGroupBy.java:630) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.subscribe(FluxGroupBy.java:696) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Flux.subscribe(Flux.java:6877) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:184) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1083) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoProcessor.onNext(MonoProcessor.java:389) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at io.rsocket.internal.ClientServerInputMultiplexer.lambda$new$1(ClientServerInputMultiplexer.java:98) ~[rsocket-core-0.11.8.jar:?]
    at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:130) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.drainLoop(FluxGroupBy.java:380) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.drain(FluxGroupBy.java:316) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.onNext(FluxGroupBy.java:201) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:108) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:108) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:211) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:327) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:310) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:537) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.http.client.WebsocketClientOperations.onInboundNext(WebsocketClientOperations.java:141) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:410) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:310) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.29.Final.jar:4.1.29.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
E 2018-10-27T20:46:09,970 i.s.b.BenchmarkState Exception occured on setUp at rampUpIteration: 1, cause: io.rsocket.exceptions.ApplicationErrorException: No reachable member with such service: /benchmarks/one, task won't start [worker-client-sdk-client-epoll-10]
io.rsocket.exceptions.ApplicationErrorException: No reachable member with such service: /benchmarks/one
    at io.rsocket.exceptions.Exceptions.from(Exceptions.java:53) ~[rsocket-core-0.11.8.jar:?]
    at io.rsocket.RSocketClient.handleFrame(RSocketClient.java:456) ~[rsocket-core-0.11.8.jar:?]
    at io.rsocket.RSocketClient.handleIncomingFrames(RSocketClient.java:419) ~[rsocket-core-0.11.8.jar:?]
    at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:130) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:238) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.drainRegular(FluxGroupBy.java:554) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.drain(FluxGroupBy.java:630) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.subscribe(FluxGroupBy.java:696) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Flux.subscribe(Flux.java:6877) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:184) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1083) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoProcessor.onNext(MonoProcessor.java:389) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at io.rsocket.internal.ClientServerInputMultiplexer.lambda$new$1(ClientServerInputMultiplexer.java:98) ~[rsocket-core-0.11.8.jar:?]
    at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:130) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.drainLoop(FluxGroupBy.java:380) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.drain(FluxGroupBy.java:316) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.onNext(FluxGroupBy.java:201) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:108) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:108) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:211) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:327) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:310) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:537) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.http.client.WebsocketClientOperations.onInboundNext(WebsocketClientOperations.java:141) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:410) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:310) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.29.Final.jar:4.1.29.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
I 2018-10-27T20:46:10,739 i.s.g.c.r.RSocketClientTransport Connected successfully on 10.200.2.40:9090 [worker-client-sdk-client-epoll-11]
E 2018-10-27T20:46:10,746 i.s.b.BenchmarkState Exception occured on setUp at rampUpIteration: 2, cause: io.rsocket.exceptions.ApplicationErrorException: No reachable member with such service: /benchmarks/one, task won't start [worker-client-sdk-client-epoll-11]
io.rsocket.exceptions.ApplicationErrorException: No reachable member with such service: /benchmarks/one
    at io.rsocket.exceptions.Exceptions.from(Exceptions.java:53) ~[rsocket-core-0.11.8.jar:?]
    at io.rsocket.RSocketClient.handleFrame(RSocketClient.java:456) ~[rsocket-core-0.11.8.jar:?]
    at io.rsocket.RSocketClient.handleIncomingFrames(RSocketClient.java:419) ~[rsocket-core-0.11.8.jar:?]
    at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:130) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:238) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.drainRegular(FluxGroupBy.java:554) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.drain(FluxGroupBy.java:630) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.subscribe(FluxGroupBy.java:696) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Flux.subscribe(Flux.java:6877) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:184) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1083) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoProcessor.onNext(MonoProcessor.java:389) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at io.rsocket.internal.ClientServerInputMultiplexer.lambda$new$1(ClientServerInputMultiplexer.java:98) ~[rsocket-core-0.11.8.jar:?]
    at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:130) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.drainLoop(FluxGroupBy.java:380) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.drain(FluxGroupBy.java:316) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.onNext(FluxGroupBy.java:201) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:108) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:108) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:211) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:327) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:310) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:537) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.http.client.WebsocketClientOperations.onInboundNext(WebsocketClientOperations.java:141) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:410) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:310) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.29.Final.jar:4.1.29.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
I 2018-10-27T20:46:11,739 i.s.g.c.r.RSocketClientTransport Connected successfully on 10.200.2.40:9090 [worker-client-sdk-client-epoll-12]
E 2018-10-27T20:46:11,747 i.s.b.BenchmarkState Exception occured on setUp at rampUpIteration: 3, cause: io.rsocket.exceptions.ApplicationErrorException: No reachable member with such service: /benchmarks/one, task won't start [worker-client-sdk-client-epoll-12]
io.rsocket.exceptions.ApplicationErrorException: No reachable member with such service: /benchmarks/one
    at io.rsocket.exceptions.Exceptions.from(Exceptions.java:53) ~[rsocket-core-0.11.8.jar:?]
    at io.rsocket.RSocketClient.handleFrame(RSocketClient.java:456) ~[rsocket-core-0.11.8.jar:?]
    at io.rsocket.RSocketClient.handleIncomingFrames(RSocketClient.java:419) ~[rsocket-core-0.11.8.jar:?]
    at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:130) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:238) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.drainRegular(FluxGroupBy.java:554) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.drain(FluxGroupBy.java:630) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.subscribe(FluxGroupBy.java:696) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Flux.subscribe(Flux.java:6877) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:184) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1083) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoProcessor.onNext(MonoProcessor.java:389) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at io.rsocket.internal.ClientServerInputMultiplexer.lambda$new$1(ClientServerInputMultiplexer.java:98) ~[rsocket-core-0.11.8.jar:?]
    at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:130) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.drainLoop(FluxGroupBy.java:380) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.drain(FluxGroupBy.java:316) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.onNext(FluxGroupBy.java:201) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:108) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:108) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:211) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:327) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:310) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:537) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.http.client.WebsocketClientOperations.onInboundNext(WebsocketClientOperations.java:141) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:410) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:310) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.29.Final.jar:4.1.29.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
I 2018-10-27T20:46:12,738 i.s.g.c.r.RSocketClientTransport Connected successfully on 10.200.2.40:9090 [worker-client-sdk-client-epoll-13]
E 2018-10-27T20:46:12,746 i.s.b.BenchmarkState Exception occured on setUp at rampUpIteration: 4, cause: io.rsocket.exceptions.ApplicationErrorException: No reachable member with such service: /benchmarks/one, task won't start [worker-client-sdk-client-epoll-13]
io.rsocket.exceptions.ApplicationErrorException: No reachable member with such service: /benchmarks/one
    at io.rsocket.exceptions.Exceptions.from(Exceptions.java:53) ~[rsocket-core-0.11.8.jar:?]
    at io.rsocket.RSocketClient.handleFrame(RSocketClient.java:456) ~[rsocket-core-0.11.8.jar:?]
    at io.rsocket.RSocketClient.handleIncomingFrames(RSocketClient.java:419) ~[rsocket-core-0.11.8.jar:?]
    at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:130) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:238) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.drainRegular(FluxGroupBy.java:554) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.drain(FluxGroupBy.java:630) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.subscribe(FluxGroupBy.java:696) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Flux.subscribe(Flux.java:6877) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:184) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1083) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoProcessor.onNext(MonoProcessor.java:389) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at io.rsocket.internal.ClientServerInputMultiplexer.lambda$new$1(ClientServerInputMultiplexer.java:98) ~[rsocket-core-0.11.8.jar:?]
    at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:130) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.drainLoop(FluxGroupBy.java:380) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.drain(FluxGroupBy.java:316) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.onNext(FluxGroupBy.java:201) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:108) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:108) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:211) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:327) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:310) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:537) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.http.client.WebsocketClientOperations.onInboundNext(WebsocketClientOperations.java:141) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:410) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:310) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.29.Final.jar:4.1.29.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
I 2018-10-27T20:46:13,739 i.s.g.c.r.RSocketClientTransport Connected successfully on 10.200.2.40:9090 [worker-client-sdk-client-epoll-14]
E 2018-10-27T20:46:13,750 i.s.b.BenchmarkState Exception occured on setUp at rampUpIteration: 5, cause: io.rsocket.exceptions.ApplicationErrorException: No reachable member with such service: /benchmarks/one, task won't start [worker-client-sdk-client-epoll-14]
io.rsocket.exceptions.ApplicationErrorException: No reachable member with such service: /benchmarks/one
    at io.rsocket.exceptions.Exceptions.from(Exceptions.java:53) ~[rsocket-core-0.11.8.jar:?]
    at io.rsocket.RSocketClient.handleFrame(RSocketClient.java:456) ~[rsocket-core-0.11.8.jar:?]
    at io.rsocket.RSocketClient.handleIncomingFrames(RSocketClient.java:419) ~[rsocket-core-0.11.8.jar:?]
    at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:130) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:238) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.drainRegular(FluxGroupBy.java:554) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.drain(FluxGroupBy.java:630) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.subscribe(FluxGroupBy.java:696) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Flux.subscribe(Flux.java:6877) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:184) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1083) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoProcessor.onNext(MonoProcessor.java:389) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at io.rsocket.internal.ClientServerInputMultiplexer.lambda$new$1(ClientServerInputMultiplexer.java:98) ~[rsocket-core-0.11.8.jar:?]
    at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:130) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.drainLoop(FluxGroupBy.java:380) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.drain(FluxGroupBy.java:316) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.onNext(FluxGroupBy.java:201) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:108) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:108) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:211) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:327) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:310) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:537) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.http.client.WebsocketClientOperations.onInboundNext(WebsocketClientOperations.java:141) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:410) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:310) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.29.Final.jar:4.1.29.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
I 2018-10-27T20:46:14,737 i.s.g.c.r.RSocketClientTransport Connected successfully on 10.200.2.40:9090 [worker-client-sdk-client-epoll-15]
E 2018-10-27T20:46:14,746 i.s.b.BenchmarkState Exception occured on setUp at rampUpIteration: 6, cause: io.rsocket.exceptions.ApplicationErrorException: No reachable member with such service: /benchmarks/one, task won't start [worker-client-sdk-client-epoll-15]
io.rsocket.exceptions.ApplicationErrorException: No reachable member with such service: /benchmarks/one
    at io.rsocket.exceptions.Exceptions.from(Exceptions.java:53) ~[rsocket-core-0.11.8.jar:?]
    at io.rsocket.RSocketClient.handleFrame(RSocketClient.java:456) ~[rsocket-core-0.11.8.jar:?]
    at io.rsocket.RSocketClient.handleIncomingFrames(RSocketClient.java:419) ~[rsocket-core-0.11.8.jar:?]
    at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:130) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:238) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.drainRegular(FluxGroupBy.java:554) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.drain(FluxGroupBy.java:630) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.subscribe(FluxGroupBy.java:696) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Flux.subscribe(Flux.java:6877) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:184) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1083) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoProcessor.onNext(MonoProcessor.java:389) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at io.rsocket.internal.ClientServerInputMultiplexer.lambda$new$1(ClientServerInputMultiplexer.java:98) ~[rsocket-core-0.11.8.jar:?]
    at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:130) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.drainLoop(FluxGroupBy.java:380) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.drain(FluxGroupBy.java:316) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.onNext(FluxGroupBy.java:201) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:108) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:108) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:211) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:327) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:310) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:537) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.http.client.WebsocketClientOperations.onInboundNext(WebsocketClientOperations.java:141) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:410) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:310) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.29.Final.jar:4.1.29.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
I 2018-10-27T20:46:15,739 i.s.g.c.r.RSocketClientTransport Connected successfully on 10.200.2.40:9090 [worker-client-sdk-client-epoll-16]
E 2018-10-27T20:46:15,747 i.s.b.BenchmarkState Exception occured on setUp at rampUpIteration: 7, cause: io.rsocket.exceptions.ApplicationErrorException: No reachable member with such service: /benchmarks/one, task won't start [worker-client-sdk-client-epoll-16]
io.rsocket.exceptions.ApplicationErrorException: No reachable member with such service: /benchmarks/one
    at io.rsocket.exceptions.Exceptions.from(Exceptions.java:53) ~[rsocket-core-0.11.8.jar:?]
    at io.rsocket.RSocketClient.handleFrame(RSocketClient.java:456) ~[rsocket-core-0.11.8.jar:?]
    at io.rsocket.RSocketClient.handleIncomingFrames(RSocketClient.java:419) ~[rsocket-core-0.11.8.jar:?]
    at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:130) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:238) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.drainRegular(FluxGroupBy.java:554) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.drain(FluxGroupBy.java:630) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$UnicastGroupedFlux.subscribe(FluxGroupBy.java:696) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Flux.subscribe(Flux.java:6877) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:184) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1083) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.MonoProcessor.onNext(MonoProcessor.java:389) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at io.rsocket.internal.ClientServerInputMultiplexer.lambda$new$1(ClientServerInputMultiplexer.java:98) ~[rsocket-core-0.11.8.jar:?]
    at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:130) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.drainLoop(FluxGroupBy.java:380) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.drain(FluxGroupBy.java:316) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxGroupBy$GroupByMain.onNext(FluxGroupBy.java:201) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:108) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:108) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:211) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:327) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:310) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:537) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.http.client.WebsocketClientOperations.onInboundNext(WebsocketClientOperations.java:141) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:141) [reactor-netty-0.8.1.RELEASE.jar:0.8.1.RELEASE]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:410) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:310) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.29.Final.jar:4.1.29.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
I 2018-10-27T20:46:16,761 i.s.g.c.r.RSocketClientTransport Connection closed on 10.200.2.40:9090 [worker-client-sdk-client-epoll-14]
I 2018-10-27T20:46:16,761 i.s.g.c.r.RSocketClientTransport Connection closed on 10.200.2.40:9090 [worker-client-sdk-client-epoll-11]
I 2018-10-27T20:46:16,761 i.s.g.c.r.RSocketClientTransport Connection closed on 10.200.2.40:9090 [worker-client-sdk-client-epoll-15]
I 2018-10-27T20:46:16,761 i.s.g.c.r.RSocketClientTransport Connection closed on 10.200.2.40:9090 [worker-client-sdk-client-epoll-13]
I 2018-10-27T20:46:16,761 i.s.g.c.r.RSocketClientTransport Connection closed on 10.200.2.40:9090 [worker-client-sdk-client-epoll-9]
I 2018-10-27T20:46:16,761 i.s.g.c.r.RSocketClientTransport Connection closed on 10.200.2.40:9090 [worker-client-sdk-client-epoll-12]
I 2018-10-27T20:46:16,761 i.s.g.c.r.RSocketClientTransport Connection closed on 10.200.2.40:9090 [worker-client-sdk-client-epoll-10]
I 2018-10-27T20:46:16,762 i.s.g.c.r.RSocketClientTransport Connection closed on 10.200.2.40:9090 [worker-client-sdk-client-epoll-16]

On the gateway:

I 2018-10-27T20:44:32,541 i.s.c.a.Slf4JConfigEventListener Config property changed: [
CA_CERTIFICATES_JAVA_VERSION=null->***, source=null->env_var,   origin=null
DEFAULT_JAVA_OPTS=null->***,    source=null->env_var,   origin=null
DEFAULT_JMX_OPTS=null->***, source=null->env_var,   origin=null
DEFAULT_OOM_OPTS=null->***, source=null->env_var,   origin=null
HOME=null->***, source=null->env_var,   origin=null
HOSTNAME=null->***, source=null->env_var,   origin=null
JAVA_DEBIAN_VERSION=null->***,  source=null->env_var,   origin=null
JAVA_HOME=null->***,    source=null->env_var,   origin=null
JAVA_OPTS=null->***,    source=null->env_var,   origin=null
JAVA_VERSION=null->***, source=null->env_var,   origin=null
LANG=null->***, source=null->env_var,   origin=null
PATH=null->***, source=null->env_var,   origin=null
PROGRAM_ARGS=null->***, source=null->env_var,   origin=null
PWD=null->***,  source=null->env_var,   origin=null
YOURKIT_AGENT=null->***,    source=null->env_var,   origin=null
awt.toolkit=null->***,  source=null->sys_prop,  origin=null
com.sun.management.jmxremote.authenticate=null->***,    source=null->sys_prop,  origin=null
com.sun.management.jmxremote.port=null->***,    source=null->sys_prop,  origin=null
com.sun.management.jmxremote.rmi.port=null->***,    source=null->sys_prop,  origin=null
com.sun.management.jmxremote.ssl=null->***, source=null->sys_prop,  origin=null
file.encoding=null->***,    source=null->sys_prop,  origin=null
file.encoding.pkg=null->***,    source=null->sys_prop,  origin=null
file.separator=null->***,   source=null->sys_prop,  origin=null
io.scalecube.gateway.discoveryPort=null->***,   source=null->cp,    origin=null->config-gateway.properties
io.scalecube.gateway.seeds=null->***,   source=null->sys_prop,  origin=null
io.scalecube.gateway.servicePort=null->***, source=null->cp,    origin=null->config-gateway.properties
java.awt.graphicsenv=null->***, source=null->sys_prop,  origin=null
java.awt.printerjob=null->***,  source=null->sys_prop,  origin=null
java.class.path=null->***,  source=null->sys_prop,  origin=null
java.class.version=null->***,   source=null->sys_prop,  origin=null
java.endorsed.dirs=null->***,   source=null->sys_prop,  origin=null
java.ext.dirs=null->***,    source=null->sys_prop,  origin=null
java.home=null->***,    source=null->sys_prop,  origin=null
java.io.tmpdir=null->***,   source=null->sys_prop,  origin=null
java.library.path=null->***,    source=null->sys_prop,  origin=null
java.rmi.server.hostname=null->***, source=null->sys_prop,  origin=null
java.rmi.server.randomIDs=null->***,    source=null->sys_prop,  origin=null
java.runtime.name=null->***,    source=null->sys_prop,  origin=null
java.runtime.version=null->***, source=null->sys_prop,  origin=null
java.specification.name=null->***,  source=null->sys_prop,  origin=null
java.specification.vendor=null->***,    source=null->sys_prop,  origin=null
java.specification.version=null->***,   source=null->sys_prop,  origin=null
java.vendor=null->***,  source=null->sys_prop,  origin=null
java.vendor.url=null->***,  source=null->sys_prop,  origin=null
java.vendor.url.bug=null->***,  source=null->sys_prop,  origin=null
java.version=null->***, source=null->sys_prop,  origin=null
java.vm.info=null->***, source=null->sys_prop,  origin=null
java.vm.name=null->***, source=null->sys_prop,  origin=null
java.vm.specification.name=null->***,   source=null->sys_prop,  origin=null
java.vm.specification.vendor=null->***, source=null->sys_prop,  origin=null
java.vm.specification.version=null->***,    source=null->sys_prop,  origin=null
java.vm.vendor=null->***,   source=null->sys_prop,  origin=null
java.vm.version=null->***,  source=null->sys_prop,  origin=null
line.separator=null->***,   source=null->sys_prop,  origin=null
log4j.configurationFile=null->***,  source=null->sys_prop,  origin=null
os.arch=null->***,  source=null->sys_prop,  origin=null
os.name=null->***,  source=null->sys_prop,  origin=null
os.version=null->***,   source=null->sys_prop,  origin=null
path.separator=null->***,   source=null->sys_prop,  origin=null
sun.arch.data.model=null->***,  source=null->sys_prop,  origin=null
sun.boot.class.path=null->***,  source=null->sys_prop,  origin=null
sun.boot.library.path=null->***,    source=null->sys_prop,  origin=null
sun.cpu.endian=null->***,   source=null->sys_prop,  origin=null
sun.cpu.isalist=null->***,  source=null->sys_prop,  origin=null
sun.io.unicode.encoding=null->***,  source=null->sys_prop,  origin=null
sun.java.command=null->***, source=null->sys_prop,  origin=null
sun.java.launcher=null->***,    source=null->sys_prop,  origin=null
sun.jnu.encoding=null->***, source=null->sys_prop,  origin=null
sun.management.compiler=null->***,  source=null->sys_prop,  origin=null
sun.os.patch.level=null->***,   source=null->sys_prop,  origin=null
sun.rmi.dgc.client.gcInterval=null->***,    source=null->sys_prop,  origin=null
sun.rmi.dgc.server.gcInterval=null->***,    source=null->sys_prop,  origin=null
user.dir=null->***, source=null->sys_prop,  origin=null
user.home=null->***,    source=null->sys_prop,  origin=null
user.language=null->***,    source=null->sys_prop,  origin=null
user.name=null->***,    source=null->sys_prop,  origin=null
user.timezone=null->***,    source=null->sys_prop,  origin=null
] [main]
I 2018-10-27T20:44:32,557 i.s.c.ConfigRegistryImpl Registered JMX MBean: javax.management.MBeanInfo[description=Information on the management interface of the MBean, attributes=[javax.management.MBeanAttributeInfo[description=Attribute exposed for management, name=Properties, type=java.util.Collection, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Attribute exposed for management, name=Settings, type=java.util.Collection, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Attribute exposed for management, name=Sources, type=java.util.Collection, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Attribute exposed for management, name=Events, type=java.util.Collection, read-only, descriptor={}]], constructors=[javax.management.MBeanConstructorInfo[description=Public constructor of the MBean, name=io.scalecube.config.jmx.JmxConfigRegistry, signature=[javax.management.MBeanParameterInfo[description=, name=p1, type=io.scalecube.config.ConfigRegistry, descriptor={}]], descriptor={}]], operations=[], notifications=[], descriptor={immutableInfo=true, interfaceClassName=io.scalecube.config.jmx.JmxConfigRegistryMBean, mxbean=false}] [main]
I 2018-10-27T20:44:32,577 i.s.g.GatewayRunner ####################################################################### [main]
I 2018-10-27T20:44:32,577 i.s.g.GatewayRunner Starting Gateway on Config{servicePort=5801, discoveryPort=4801, seeds=[10.200.2.40:4801], memberHost=null, memberPort=null} [main]
I 2018-10-27T20:44:32,577 i.s.g.GatewayRunner ####################################################################### [main]
I 2018-10-27T20:44:33,819 i.s.s.d.a.ServiceDiscovery Start scalecube service discovery with config: ClusterConfig{seedMembers=[10.200.2.40:4801], metadata={}, syncInterval=30000, syncTimeout=3000, suspicionMult=5, syncGroup='default', pingInterval=1000, pingTimeout=500, pingReqMembers=3, gossipInterval=200, gossipFanout=3, gossipRepeatMult=3, transportConfig=TransportConfig{listenAddress=null, listenInterface=null, preferIPv6=false, port=4801, connectTimeout=3000, useNetworkEmulator=false, enableEpoll=true, bossThreads=2, workerThreads=0}, memberHost=null, memberPort=null} [rsocket-boss-3-1]
I 2018-10-27T20:44:33,845 i.s.t.BootstrapFactory Use epoll transport [rsocket-boss-3-1]
I 2018-10-27T20:44:33,864 i.s.t.TransportImpl Bound to: 10.200.2.40:4801 [sc-boss-4-1]
I 2018-10-27T20:44:33,936 i.s.g.h.HttpGateway Starting gateway with GatewayConfig{name='http', gatewayClass=io.scalecube.gateway.http.HttpGateway, options={}, port=8080, workerThreadPool=null} [sc-boss-4-1]
I 2018-10-27T20:44:33,974 i.s.g.h.HttpGateway Gateway has been started successfully on /0:0:0:0:0:0:0:0%0:8080 [sc-boss-4-1]
I 2018-10-27T20:44:33,978 i.s.g.w.WebsocketGateway Starting gateway with GatewayConfig{name='ws', gatewayClass=io.scalecube.gateway.websocket.WebsocketGateway, options={}, port=7070, workerThreadPool=null} [sc-boss-4-1]
I 2018-10-27T20:44:33,984 i.s.g.w.WebsocketGateway Gateway has been started successfully on /0:0:0:0:0:0:0:0%0:7070 [sc-boss-4-1]
I 2018-10-27T20:44:33,985 i.s.g.r.w.RSocketWebsocketGateway Starting gateway with GatewayConfig{name='rsws', gatewayClass=io.scalecube.gateway.rsocket.websocket.RSocketWebsocketGateway, options={}, port=9090, workerThreadPool=null} [sc-boss-4-1]
I 2018-10-27T20:44:33,992 i.s.g.r.w.RSocketWebsocketGateway Gateway has been started successfully on /0:0:0:0:0:0:0:0%0:9090 [sc-boss-4-1]
I 2018-10-27T20:44:53,007 i.s.s.d.a.ServiceDiscovery Service Reference was ADDED since new Member has joined the cluster 3B87532DC50721997FED@10.200.2.176:4802{{"id":"6dcdd1be-84c8-45d2-94c8-3450ea4bdb2c","host":"10.200.2.176","port":5802,"contentTypes":["application/json"],"tags":{},"serviceRegistrations":[{"namespace":"benchmarks","tags":{},"methods":[{"action":"failure","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"infiniteStream","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"one","tags":{},"communicationMode":"REQUEST_RESPONSE"}]},{"namespace":"greeting","tags":{},"methods":[{"action":"pojo/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"failing/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"never/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"pojo/many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"delay/many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"manyStream","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"failing/many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"empty/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"empty/many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"delay/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"one","tags":{},"communicationMode":"REQUEST_RESPONSE"}]}]}=service} : ServiceEndpoint{id='6dcdd1be-84c8-45d2-94c8-3450ea4bdb2c', host='10.200.2.176', port=5802, tags={}, serviceRegistrations=[ServiceRegistration{namespace='benchmarks', tags={}, methods=[ServiceMethodDefinition{action='failure', tags={}}, ServiceMethodDefinition{action='infiniteStream', tags={}}, ServiceMethodDefinition{action='one', tags={}}]}, ServiceRegistration{namespace='greeting', tags={}, methods=[ServiceMethodDefinition{action='pojo/one', tags={}}, ServiceMethodDefinition{action='failing/one', tags={}}, ServiceMethodDefinition{action='many', tags={}}, ServiceMethodDefinition{action='never/one', tags={}}, ServiceMethodDefinition{action='pojo/many', tags={}}, ServiceMethodDefinition{action='delay/many', tags={}}, ServiceMethodDefinition{action='manyStream', tags={}}, ServiceMethodDefinition{action='failing/many', tags={}}, ServiceMethodDefinition{action='empty/one', tags={}}, ServiceMethodDefinition{action='empty/many', tags={}}, ServiceMethodDefinition{action='delay/one', tags={}}, ServiceMethodDefinition{action='one', tags={}}]}]} [sc-membership-4801]
E 2018-10-27T20:45:07,669 r.c.p.Operators Operator called default onErrorDropped [sc-io-5-1]
reactor.core.Exceptions$ReactorRejectedExecutionException: Scheduler unavailable
    at reactor.core.Exceptions.failWithRejected(Exceptions.java:249) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Operators.onRejectedExecution(Operators.java:412) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxPublishOn$PublishOnConditionalSubscriber.trySchedule(FluxPublishOn.java:759) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxPublishOn$PublishOnConditionalSubscriber.onNext(FluxPublishOn.java:687) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drainFused(FluxOnBackpressureBuffer.java:275) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drain(FluxOnBackpressureBuffer.java:199) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.onNext(FluxOnBackpressureBuffer.java:164) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.DirectProcessor$DirectInner.onNext(DirectProcessor.java:297) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.DirectProcessor.onNext(DirectProcessor.java:106) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:89) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.DelegateProcessor.onNext(DelegateProcessor.java:64) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxCreate$IgnoreSink.next(FluxCreate.java:573) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxCreate$SerializedSink.next(FluxCreate.java:151) [reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at io.scalecube.transport.MessageHandler.channelRead(MessageHandler.java:33) [scalecube-transport-2.1.1.jar:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:410) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:310) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.29.Final.jar:4.1.29.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: java.util.concurrent.RejectedExecutionException: Scheduler unavailable
    at reactor.core.Exceptions.<clinit>(Exceptions.java:502) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.scheduler.Schedulers.workerSchedule(Schedulers.java:708) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.scheduler.ExecutorServiceWorker.schedule(ExecutorServiceWorker.java:43) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxPublishOn$PublishOnConditionalSubscriber.trySchedule(FluxPublishOn.java:755) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    ... 33 more
W 2018-10-27T20:45:07,688 i.s.t.ExceptionHandler Exception caught for channel [id: 0x13a5c4ac, L:/10.200.2.40:4801 - R:/10.200.2.176:49374], reactor.core.Exceptions$ReactorRejectedExecutionException: Scheduler unavailable [sc-io-5-1]
reactor.core.Exceptions$BubblingException: reactor.core.Exceptions$ReactorRejectedExecutionException: Scheduler unavailable
    at reactor.core.Exceptions.bubble(Exceptions.java:154) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Operators.onErrorDropped(Operators.java:263) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onError(FluxTimeout.java:182) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxTake$TakeFuseableSubscriber.onError(FluxTake.java:415) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onError(FluxFilterFuseable.java:142) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onError(FluxFilterFuseable.java:319) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxPublishOn$PublishOnConditionalSubscriber.trySchedule(FluxPublishOn.java:759) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxPublishOn$PublishOnConditionalSubscriber.onNext(FluxPublishOn.java:687) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drainFused(FluxOnBackpressureBuffer.java:275) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drain(FluxOnBackpressureBuffer.java:199) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.onNext(FluxOnBackpressureBuffer.java:164) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.DirectProcessor$DirectInner.onNext(DirectProcessor.java:297) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.DirectProcessor.onNext(DirectProcessor.java:106) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:89) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.DelegateProcessor.onNext(DelegateProcessor.java:64) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxCreate$IgnoreSink.next(FluxCreate.java:573) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxCreate$SerializedSink.next(FluxCreate.java:151) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at io.scalecube.transport.MessageHandler.channelRead(MessageHandler.java:33) ~[scalecube-transport-2.1.1.jar:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:410) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:310) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.29.Final.jar:4.1.29.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: reactor.core.Exceptions$ReactorRejectedExecutionException: Scheduler unavailable
    at reactor.core.Exceptions.failWithRejected(Exceptions.java:249) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.Operators.onRejectedExecution(Operators.java:412) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    ... 34 more
Caused by: java.util.concurrent.RejectedExecutionException: Scheduler unavailable
    at reactor.core.Exceptions.<clinit>(Exceptions.java:502) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.scheduler.Schedulers.workerSchedule(Schedulers.java:708) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.scheduler.ExecutorServiceWorker.schedule(ExecutorServiceWorker.java:43) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    at reactor.core.publisher.FluxPublishOn$PublishOnConditionalSubscriber.trySchedule(FluxPublishOn.java:755) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]
    ... 33 more
I 2018-10-27T20:45:17,672 i.s.s.d.a.ServiceDiscovery Service Reference was REMOVED since Member have left the cluster 3B87532DC50721997FED@10.200.2.176:4802{{"id":"6dcdd1be-84c8-45d2-94c8-3450ea4bdb2c","host":"10.200.2.176","port":5802,"contentTypes":["application/json"],"tags":{},"serviceRegistrations":[{"namespace":"benchmarks","tags":{},"methods":[{"action":"failure","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"infiniteStream","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"one","tags":{},"communicationMode":"REQUEST_RESPONSE"}]},{"namespace":"greeting","tags":{},"methods":[{"action":"pojo/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"failing/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"never/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"pojo/many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"delay/many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"manyStream","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"failing/many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"empty/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"empty/many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"delay/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"one","tags":{},"communicationMode":"REQUEST_RESPONSE"}]}]}=service} : ServiceEndpoint{id='6dcdd1be-84c8-45d2-94c8-3450ea4bdb2c', host='10.200.2.176', port=5802, tags={}, serviceRegistrations=[ServiceRegistration{namespace='benchmarks', tags={}, methods=[ServiceMethodDefinition{action='failure', tags={}}, ServiceMethodDefinition{action='infiniteStream', tags={}}, ServiceMethodDefinition{action='one', tags={}}]}, ServiceRegistration{namespace='greeting', tags={}, methods=[ServiceMethodDefinition{action='pojo/one', tags={}}, ServiceMethodDefinition{action='failing/one', tags={}}, ServiceMethodDefinition{action='many', tags={}}, ServiceMethodDefinition{action='never/one', tags={}}, ServiceMethodDefinition{action='pojo/many', tags={}}, ServiceMethodDefinition{action='delay/many', tags={}}, ServiceMethodDefinition{action='manyStream', tags={}}, ServiceMethodDefinition{action='failing/many', tags={}}, ServiceMethodDefinition{action='empty/one', tags={}}, ServiceMethodDefinition{action='empty/many', tags={}}, ServiceMethodDefinition{action='delay/one', tags={}}, ServiceMethodDefinition{action='one', tags={}}]}]} [sc-membership-4801]
I 2018-10-27T20:46:09,890 i.s.g.r.w.RSocketWebsocketAcceptor Accepted rsocket websocket: io.rsocket.RSocketClient@63bc74a9, connectionSetup: io.rsocket.ConnectionSetupPayload$DefaultConnectionSetupPayload@2d1a9cf9 [rsocket-worker-1-1]
I 2018-10-27T20:46:09,890 i.s.g.r.w.RSocketWebsocketAcceptor Accepted rsocket websocket: io.rsocket.RSocketClient@15fdff73, connectionSetup: io.rsocket.ConnectionSetupPayload$DefaultConnectionSetupPayload@768c9334 [rsocket-worker-1-2]
E 2018-10-27T20:46:09,947 i.s.s.ServiceCall Failed  to invoke service, No reachable member with such service definition [/benchmarks/one], args [ServiceMessage {headers: {q=/benchmarks/one}, data: null}] [rsocket-worker-1-2]
E 2018-10-27T20:46:09,947 i.s.s.ServiceCall Failed  to invoke service, No reachable member with such service definition [/benchmarks/one], args [ServiceMessage {headers: {q=/benchmarks/one}, data: null}] [rsocket-worker-1-1]
I 2018-10-27T20:46:10,745 i.s.g.r.w.RSocketWebsocketAcceptor Accepted rsocket websocket: io.rsocket.RSocketClient@66086623, connectionSetup: io.rsocket.ConnectionSetupPayload$DefaultConnectionSetupPayload@6237fe3a [rsocket-worker-1-3]
E 2018-10-27T20:46:10,747 i.s.s.ServiceCall Failed  to invoke service, No reachable member with such service definition [/benchmarks/one], args [ServiceMessage {headers: {q=/benchmarks/one}, data: null}] [rsocket-worker-1-3]
I 2018-10-27T20:46:11,746 i.s.g.r.w.RSocketWebsocketAcceptor Accepted rsocket websocket: io.rsocket.RSocketClient@4ecf3ce0, connectionSetup: io.rsocket.ConnectionSetupPayload$DefaultConnectionSetupPayload@4af0a1e1 [rsocket-worker-1-4]
E 2018-10-27T20:46:11,747 i.s.s.ServiceCall Failed  to invoke service, No reachable member with such service definition [/benchmarks/one], args [ServiceMessage {headers: {q=/benchmarks/one}, data: null}] [rsocket-worker-1-4]
I 2018-10-27T20:46:12,745 i.s.g.r.w.RSocketWebsocketAcceptor Accepted rsocket websocket: io.rsocket.RSocketClient@10da0d80, connectionSetup: io.rsocket.ConnectionSetupPayload$DefaultConnectionSetupPayload@14e59b43 [rsocket-worker-1-5]
E 2018-10-27T20:46:12,746 i.s.s.ServiceCall Failed  to invoke service, No reachable member with such service definition [/benchmarks/one], args [ServiceMessage {headers: {q=/benchmarks/one}, data: null}] [rsocket-worker-1-5]
I 2018-10-27T20:46:13,745 i.s.g.r.w.RSocketWebsocketAcceptor Accepted rsocket websocket: io.rsocket.RSocketClient@efc7886, connectionSetup: io.rsocket.ConnectionSetupPayload$DefaultConnectionSetupPayload@4bfee73d [rsocket-worker-1-6]
E 2018-10-27T20:46:13,747 i.s.s.ServiceCall Failed  to invoke service, No reachable member with such service definition [/benchmarks/one], args [ServiceMessage {headers: {q=/benchmarks/one}, data: null}] [rsocket-worker-1-6]
I 2018-10-27T20:46:14,744 i.s.g.r.w.RSocketWebsocketAcceptor Accepted rsocket websocket: io.rsocket.RSocketClient@615b99ad, connectionSetup: io.rsocket.ConnectionSetupPayload$DefaultConnectionSetupPayload@6564015e [rsocket-worker-1-7]
E 2018-10-27T20:46:14,746 i.s.s.ServiceCall Failed  to invoke service, No reachable member with such service definition [/benchmarks/one], args [ServiceMessage {headers: {q=/benchmarks/one}, data: null}] [rsocket-worker-1-7]
I 2018-10-27T20:46:15,746 i.s.g.r.w.RSocketWebsocketAcceptor Accepted rsocket websocket: io.rsocket.RSocketClient@12d12439, connectionSetup: io.rsocket.ConnectionSetupPayload$DefaultConnectionSetupPayload@57d3a807 [rsocket-worker-1-8]
E 2018-10-27T20:46:15,748 i.s.s.ServiceCall Failed  to invoke service, No reachable member with such service definition [/benchmarks/one], args [ServiceMessage {headers: {q=/benchmarks/one}, data: null}] [rsocket-worker-1-8]
I 2018-10-27T20:46:16,765 i.s.g.r.w.RSocketWebsocketAcceptor Client disconnected: io.rsocket.RSocketClient@efc7886 [rsocket-worker-1-6]
I 2018-10-27T20:46:16,765 i.s.g.r.w.RSocketWebsocketAcceptor Client disconnected: io.rsocket.RSocketClient@15fdff73 [rsocket-worker-1-2]
I 2018-10-27T20:46:16,765 i.s.g.r.w.RSocketWebsocketAcceptor Client disconnected: io.rsocket.RSocketClient@66086623 [rsocket-worker-1-3]
I 2018-10-27T20:46:16,765 i.s.g.r.w.RSocketWebsocketAcceptor Client disconnected: io.rsocket.RSocketClient@12d12439 [rsocket-worker-1-8]
I 2018-10-27T20:46:16,765 i.s.g.r.w.RSocketWebsocketAcceptor Client disconnected: io.rsocket.RSocketClient@10da0d80 [rsocket-worker-1-5]
I 2018-10-27T20:46:16,765 i.s.g.r.w.RSocketWebsocketAcceptor Client disconnected: io.rsocket.RSocketClient@63bc74a9 [rsocket-worker-1-1]
I 2018-10-27T20:46:16,765 i.s.g.r.w.RSocketWebsocketAcceptor Client disconnected: io.rsocket.RSocketClient@4ecf3ce0 [rsocket-worker-1-4]
I 2018-10-27T20:46:16,767 i.s.g.r.w.RSocketWebsocketAcceptor Client disconnected: io.rsocket.RSocketClient@615b99ad [rsocket-worker-1-7]

On the service:

I 2018-10-27T20:44:51,112 i.s.c.a.Slf4JConfigEventListener Config property changed: [
CA_CERTIFICATES_JAVA_VERSION=null->***, source=null->env_var,   origin=null
DEFAULT_JAVA_OPTS=null->***,    source=null->env_var,   origin=null
DEFAULT_JMX_OPTS=null->***, source=null->env_var,   origin=null
DEFAULT_OOM_OPTS=null->***, source=null->env_var,   origin=null
HOME=null->***, source=null->env_var,   origin=null
HOSTNAME=null->***, source=null->env_var,   origin=null
JAVA_DEBIAN_VERSION=null->***,  source=null->env_var,   origin=null
JAVA_HOME=null->***,    source=null->env_var,   origin=null
JAVA_OPTS=null->***,    source=null->env_var,   origin=null
JAVA_VERSION=null->***, source=null->env_var,   origin=null
LANG=null->***, source=null->env_var,   origin=null
PATH=null->***, source=null->env_var,   origin=null
PROGRAM_ARGS=null->***, source=null->env_var,   origin=null
PWD=null->***,  source=null->env_var,   origin=null
YOURKIT_AGENT=null->***,    source=null->env_var,   origin=null
awt.toolkit=null->***,  source=null->sys_prop,  origin=null
com.sun.management.jmxremote.authenticate=null->***,    source=null->sys_prop,  origin=null
com.sun.management.jmxremote.port=null->***,    source=null->sys_prop,  origin=null
com.sun.management.jmxremote.rmi.port=null->***,    source=null->sys_prop,  origin=null
com.sun.management.jmxremote.ssl=null->***, source=null->sys_prop,  origin=null
file.encoding=null->***,    source=null->sys_prop,  origin=null
file.encoding.pkg=null->***,    source=null->sys_prop,  origin=null
file.separator=null->***,   source=null->sys_prop,  origin=null
io.scalecube.gateway.examples.discoveryPort=null->***,  source=null->cp,    origin=null->config-examples.properties
io.scalecube.gateway.examples.seeds=null->***,  source=null->sys_prop,  origin=null
io.scalecube.gateway.examples.servicePort=null->***,    source=null->cp,    origin=null->config-examples.properties
java.awt.graphicsenv=null->***, source=null->sys_prop,  origin=null
java.awt.printerjob=null->***,  source=null->sys_prop,  origin=null
java.class.path=null->***,  source=null->sys_prop,  origin=null
java.class.version=null->***,   source=null->sys_prop,  origin=null
java.endorsed.dirs=null->***,   source=null->sys_prop,  origin=null
java.ext.dirs=null->***,    source=null->sys_prop,  origin=null
java.home=null->***,    source=null->sys_prop,  origin=null
java.io.tmpdir=null->***,   source=null->sys_prop,  origin=null
java.library.path=null->***,    source=null->sys_prop,  origin=null
java.rmi.server.hostname=null->***, source=null->sys_prop,  origin=null
java.rmi.server.randomIDs=null->***,    source=null->sys_prop,  origin=null
java.runtime.name=null->***,    source=null->sys_prop,  origin=null
java.runtime.version=null->***, source=null->sys_prop,  origin=null
java.specification.name=null->***,  source=null->sys_prop,  origin=null
java.specification.vendor=null->***,    source=null->sys_prop,  origin=null
java.specification.version=null->***,   source=null->sys_prop,  origin=null
java.vendor=null->***,  source=null->sys_prop,  origin=null
java.vendor.url=null->***,  source=null->sys_prop,  origin=null
java.vendor.url.bug=null->***,  source=null->sys_prop,  origin=null
java.version=null->***, source=null->sys_prop,  origin=null
java.vm.info=null->***, source=null->sys_prop,  origin=null
java.vm.name=null->***, source=null->sys_prop,  origin=null
java.vm.specification.name=null->***,   source=null->sys_prop,  origin=null
java.vm.specification.vendor=null->***, source=null->sys_prop,  origin=null
java.vm.specification.version=null->***,    source=null->sys_prop,  origin=null
java.vm.vendor=null->***,   source=null->sys_prop,  origin=null
java.vm.version=null->***,  source=null->sys_prop,  origin=null
line.separator=null->***,   source=null->sys_prop,  origin=null
log4j.configurationFile=null->***,  source=null->sys_prop,  origin=null
os.arch=null->***,  source=null->sys_prop,  origin=null
os.name=null->***,  source=null->sys_prop,  origin=null
os.version=null->***,   source=null->sys_prop,  origin=null
path.separator=null->***,   source=null->sys_prop,  origin=null
sun.arch.data.model=null->***,  source=null->sys_prop,  origin=null
sun.boot.class.path=null->***,  source=null->sys_prop,  origin=null
sun.boot.library.path=null->***,    source=null->sys_prop,  origin=null
sun.cpu.endian=null->***,   source=null->sys_prop,  origin=null
sun.cpu.isalist=null->***,  source=null->sys_prop,  origin=null
sun.io.unicode.encoding=null->***,  source=null->sys_prop,  origin=null
sun.java.command=null->***, source=null->sys_prop,  origin=null
sun.java.launcher=null->***,    source=null->sys_prop,  origin=null
sun.jnu.encoding=null->***, source=null->sys_prop,  origin=null
sun.management.compiler=null->***,  source=null->sys_prop,  origin=null
sun.os.patch.level=null->***,   source=null->sys_prop,  origin=null
sun.rmi.dgc.client.gcInterval=null->***,    source=null->sys_prop,  origin=null
sun.rmi.dgc.server.gcInterval=null->***,    source=null->sys_prop,  origin=null
user.dir=null->***, source=null->sys_prop,  origin=null
user.home=null->***,    source=null->sys_prop,  origin=null
user.language=null->***,    source=null->sys_prop,  origin=null
user.name=null->***,    source=null->sys_prop,  origin=null
user.timezone=null->***,    source=null->sys_prop,  origin=null
] [main]
I 2018-10-27T20:44:51,127 i.s.c.ConfigRegistryImpl Registered JMX MBean: javax.management.MBeanInfo[description=Information on the management interface of the MBean, attributes=[javax.management.MBeanAttributeInfo[description=Attribute exposed for management, name=Properties, type=java.util.Collection, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Attribute exposed for management, name=Settings, type=java.util.Collection, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Attribute exposed for management, name=Sources, type=java.util.Collection, read-only, descriptor={}], javax.management.MBeanAttributeInfo[description=Attribute exposed for management, name=Events, type=java.util.Collection, read-only, descriptor={}]], constructors=[javax.management.MBeanConstructorInfo[description=Public constructor of the MBean, name=io.scalecube.config.jmx.JmxConfigRegistry, signature=[javax.management.MBeanParameterInfo[description=, name=p1, type=io.scalecube.config.ConfigRegistry, descriptor={}]], descriptor={}]], operations=[], notifications=[], descriptor={immutableInfo=true, interfaceClassName=io.scalecube.config.jmx.JmxConfigRegistryMBean, mxbean=false}] [main]
I 2018-10-27T20:44:51,148 i.s.g.e.ExamplesRunner *********************************************************************** [main]
I 2018-10-27T20:44:51,148 i.s.g.e.ExamplesRunner Starting Examples services on Config{servicePort=5802, discoveryPort=4802, numOfThreads=null, seeds=[10.200.2.40:4801], memberHost=null, memberPort=null} [main]
I 2018-10-27T20:44:51,148 i.s.g.e.ExamplesRunner *********************************************************************** [main]
I 2018-10-27T20:44:51,148 i.s.g.e.ExamplesRunner Number of worker threads: 8 [main]
I 2018-10-27T20:44:52,356 i.s.s.d.a.ServiceDiscovery Start scalecube service discovery with config: ClusterConfig{seedMembers=[10.200.2.40:4801], metadata={{"id":"6dcdd1be-84c8-45d2-94c8-3450ea4bdb2c","host":"10.200.2.176","port":5802,"contentTypes":["application/json"],"tags":{},"serviceRegistrations":[{"namespace":"benchmarks","tags":{},"methods":[{"action":"failure","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"infiniteStream","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"one","tags":{},"communicationMode":"REQUEST_RESPONSE"}]},{"namespace":"greeting","tags":{},"methods":[{"action":"pojo/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"failing/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"never/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"pojo/many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"delay/many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"manyStream","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"failing/many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"empty/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"empty/many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"delay/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"one","tags":{},"communicationMode":"REQUEST_RESPONSE"}]}]}=service}, syncInterval=30000, syncTimeout=3000, suspicionMult=5, syncGroup='default', pingInterval=1000, pingTimeout=500, pingReqMembers=3, gossipInterval=200, gossipFanout=3, gossipRepeatMult=3, transportConfig=TransportConfig{listenAddress=null, listenInterface=null, preferIPv6=false, port=4802, connectTimeout=3000, useNetworkEmulator=false, enableEpoll=true, bossThreads=2, workerThreads=0}, memberHost=null, memberPort=null} [rsocket-boss-3-1]
I 2018-10-27T20:44:52,383 i.s.t.BootstrapFactory Use epoll transport [rsocket-boss-3-1]
I 2018-10-27T20:44:52,403 i.s.t.TransportImpl Bound to: 10.200.2.176:4802 [sc-boss-4-1]
I 2018-10-27T20:44:53,086 i.s.c.m.MembershipProtocolImpl Joined cluster 'default': [{m: 86585974239EFAEE0994@10.200.2.40:4801, s: ALIVE, inc: 0}, {m: 3B87532DC50721997FED@10.200.2.176:4802{{"id":"6dcdd1be-84c8-45d2-94c8-3450ea4bdb2c","host":"10.200.2.176","port":5802,"contentTypes":["application/json"],"tags":{},"serviceRegistrations":[{"namespace":"benchmarks","tags":{},"methods":[{"action":"failure","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"infiniteStream","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"one","tags":{},"communicationMode":"REQUEST_RESPONSE"}]},{"namespace":"greeting","tags":{},"methods":[{"action":"pojo/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"failing/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"never/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"pojo/many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"delay/many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"manyStream","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"failing/many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"empty/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"empty/many","tags":{},"communicationMode":"REQUEST_STREAM"},{"action":"delay/one","tags":{},"communicationMode":"REQUEST_RESPONSE"},{"action":"one","tags":{},"communicationMode":"REQUEST_RESPONSE"}]}]}=service}, s: ALIVE, inc: 0}] [sc-membership-4802]
artem-v commented 6 years ago

Line of interest is

    at reactor.core.scheduler.Schedulers.workerSchedule(Schedulers.java:708) ~[reactor-core-3.1.8.RELEASE.jar:3.1.8.RELEASE]

When one calls Schedulers.fromExecutorService(exexcutorService) he creates DelegateServiceScheduler instance, which has method :

    @Override
    public Worker createWorker() {
        return new ExecutorServiceWorker(executor);
    }

By turn, ExecutorServiceWorker has field Composite tasks = Disposables.composite() ,. So here it goes: in workerSchedule method we see following:

        WorkerTask sr = new WorkerTask(task, tasks);
        if (!tasks.add(sr)) {
            throw Exceptions.failWithRejected();
        }

If look at .add method which by turn would lead to .addEntry method we can see following:

        boolean addEntry(Disposable value) {
            final Disposable[] a = disposables;
            final int m = mask;

            int pos = mix(value.hashCode()) & m;
            Disposable curr = a[pos];
            if (curr != null) {
                if (curr.equals(value)) {
                    return false;
                }
                for (;;) {
                    pos = (pos + 1) & m;
                    curr = a[pos];
                    if (curr == null) {
                        break;
                    }
                    if (curr.equals(value)) {
                        return false;
                    }
                }
            }
            a[pos] = value;
            if (++size >= maxSize) {
                rehash();
            }
            return true;
        }

So, looks like return false; block only called when curr.equals(value) .

To conclude all above it appears that we can't schedule same task again and again and again to the scheduler which was created via reactor.core.scheduler.Schedulers#fromExecutorService .

artem-v commented 6 years ago

To fix all of this the the proposition is to simply not create executor service and wrap it to scheduler but just go straight with one of it factory methods on Schdulers class itself: for example newSingle().

artem-v commented 6 years ago

Reopening. 3.2.1 Did not fix the problem.

E 2018-10-29T13:06:06,274 r.c.p.Operators Operator called default onErrorDropped [sc-io-5-1]reactor.core.Exceptions$ReactorRejectedExecutionException: Scheduler unavailable
    at reactor.core.Exceptions.failWithRejected(Exceptions.java:249) ~[reactor-core-3.2.1.RELEASE.jar:3.2.1.RELEASE]
    at reactor.core.publisher.Operators.onRejectedExecution(Operators.java:799) ~[reactor-core-3.2.1.RELEASE.jar:3.2.1.RELEASE]
    at reactor.core.publisher.FluxPublishOn$PublishOnConditionalSubscriber.trySchedule(FluxPublishOn.java:763) [reactor-core-3.2.1.RELEASE.jar:3.2.1.RELEASE]
    at reactor.core.publisher.FluxPublishOn$PublishOnConditionalSubscriber.onNext(FluxPublishOn.java:691) [reactor-core-3.2.1.RELEASE.jar:3.2.1.RELEASE]
    at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drainFused(FluxOnBackpressureBuffer.java:279) [reactor-core-3.2.1.RELEASE.jar:3.2.1.RELEASE]
    at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.drain(FluxOnBackpressureBuffer.java:203) [reactor-core-3.2.1.RELEASE.jar:3.2.1.RELEASE]
    at reactor.core.publisher.FluxOnBackpressureBuffer$BackpressureBufferSubscriber.onNext(FluxOnBackpressureBuffer.java:168) [reactor-core-3.2.1.RELEASE.jar:3.2.1.RELEASE]
    at reactor.core.publisher.DirectProcessor$DirectInner.onNext(DirectProcessor.java:333) [reactor-core-3.2.1.RELEASE.jar:3.2.1.RELEASE]
    at reactor.core.publisher.DirectProcessor.onNext(DirectProcessor.java:142) [reactor-core-3.2.1.RELEASE.jar:3.2.1.RELEASE]
    at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:89) [reactor-core-3.2.1.RELEASE.jar:3.2.1.RELEASE]
    at reactor.core.publisher.DelegateProcessor.onNext(DelegateProcessor.java:64) [reactor-core-3.2.1.RELEASE.jar:3.2.1.RELEASE]
    at reactor.core.publisher.FluxCreate$IgnoreSink.next(FluxCreate.java:593) [reactor-core-3.2.1.RELEASE.jar:3.2.1.RELEASE]
    at reactor.core.publisher.FluxCreate$SerializedSink.next(FluxCreate.java:151) [reactor-core-3.2.1.RELEASE.jar:3.2.1.RELEASE]
    at io.scalecube.transport.MessageHandler.channelRead(MessageHandler.java:33) [scalecube-transport-2.1.2.jar:?]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:808) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:410) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:310) [netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:4.1.29.Final]
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) [netty-common-4.1.29.Final.jar:4.1.29.Final]
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.29.Final.jar:4.1.29.Final]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: java.util.concurrent.RejectedExecutionException: Scheduler unavailable
    at reactor.core.Exceptions.<clinit>(Exceptions.java:502) ~[reactor-core-3.2.1.RELEASE.jar:3.2.1.RELEASE]
    at reactor.core.scheduler.Schedulers.workerSchedule(Schedulers.java:709) ~[reactor-core-3.2.1.RELEASE.jar:3.2.1.RELEASE]
    at reactor.core.scheduler.ExecutorServiceWorker.schedule(ExecutorServiceWorker.java:43) ~[reactor-core-3.2.1.RELEASE.jar:3.2.1.RELEASE]
    at reactor.core.publisher.FluxPublishOn$PublishOnConditionalSubscriber.trySchedule(FluxPublishOn.java:759) ~[reactor-core-3.2.1.RELEASE.jar:3.2.1.RELEASE]  ... 33 more
artem-v commented 6 years ago

Fixed at https://github.com/scalecube/scalecube-cluster/pull/65