allenhumphreys
commented
4 years ago @kushagraThapar New issue here!
Closed allenhumphreys closed 4 years ago
allenhumphreys
commented
4 years ago @kushagraThapar New issue here!
kushagraThapar
commented
4 years ago @allenhumphreys @timwhit Based on the user agent string in cosmos client initialization, it seems like you are using spring-data-cosmosdb v2.2.2. (which depends on v3.6.0) Can you please confirm your project structure, and share maven dependency tree ? To use v3.7.1, you need to use spring-data-cosmosdb - v2.2.3.FIX1 Check these release notes to make sure you are on the right version : https://github.com/microsoft/spring-data-cosmosdb/releases
allenhumphreys
commented
4 years ago @kushagraThapar Thanks, we pull in the cosmos dependency version in explicitly because we don't use spring-data-cosmosdb (mostly), we interact with the SDK directly (no JPA repositories etc), so we primarily use spring-data-cosmosdb for some of its configuration capabilities, and that's largely for legacy purposes.
```
runtimeClasspath - Runtime classpath of compilation 'main' (target (jvm)).
+---
kushagraThapar
commented
4 years ago @allenhumphreys - thanks for attaching the dependency tree, it looks good overall, except for the spring-data-cosmsodb version. Can you please try upgrading to spring-data-cosmosdb v2.2.3.FIX1 while I investigate more on this ?
Seems like this is coming from reactor-netty library as we don't manage reference count for our Http Client.
allenhumphreys
commented
4 years ago @kushagraThapar I'm a little confused, the original bug was a memory leak due to reference counting in DIRECT mode. Now you're telling me you don't manage reference counting in DIRECT mode. The memory leak seemed to be caused by too few releases of the byte buffer, and this exception seems to be caused by accessing the byte buffer after it was released to a ref count of 0 (or an attempt to release a byte buffer with a reference count that is already zero, I'm unsure since I didn't dig into any sources)
allenhumphreys
commented
4 years ago Actually, I think I understand what you mean now. Beyond the TCP connection, what HTTP calls is RxGatewayStoreModel in charge of? Unfortunately, I don't have any information about the requests that were being made by the SDK at the time of the exceptions.
kushagraThapar
commented
4 years ago Actually, I think I understand what you mean now. Beyond the TCP connection, what HTTP calls is RxGatewayStoreModel in charge of? Unfortunately, I don't have any information about the requests that were being made by the SDK at the time of the exceptions.
Yes, you got it right. All I am saying is, this bug is not related to Direct TCP protocol, since it includes reactor-netty references which suggests that it is coming from Reactor Netty Http client - which is used in RxGatewayStoreModel.
Since you mentioned that this stack trace is being logged by RxGatewayStoreModel, can you please share more logs which are before and after this exception stack trace ?
Do you see this only in DIRECT mode or GATEWAY mode as well ?
If we can figure out the operation (point read, query, insert, upsert, etc.) being carried out before this exception stack trace, that will help us fix this soon.
kushagraThapar
commented
4 years ago @allenhumphreys - I think I have fixed the issue. Can't verify it since the issue is intermittent. But I am running some long running tests right now to make sure this doesn't breaks anything. It would be great if you can check this PR too: https://github.com/Azure/azure-sdk-for-java/issues/9802 Just to get a second set of eyes.
timwhit
commented
4 years ago Looks good to me. I'd like to see what @allenhumphreys thinks when he wakes up tomorrow.
kushagraThapar
commented
4 years ago Looks good to me. I'd like to see what @allenhumphreys thinks when he wakes up tomorrow.
Sounds great. Thanks for taking time and reviewing the code change. I am experiencing some internal CI issues because of some internal capacity issues of cosmos, so the release might take some time .. Will try to get it out as soon as possible. Thanks for your patience.
allenhumphreys
commented
4 years ago @kushagraThapar Yeah, the double wrapping of the byte buffer did seem a bit suspicious but didn't have time to dig into exactly how there might be a "sometimes" nature to the problem and I am not super familiar with netty memory management classes. Hope this solves the sporadic exceptions, and it was unfortunately we received a bunch right after upgrading the SDK, a real red herring.
kushagraThapar
commented
4 years ago @timwhit @allenhumphreys - Release has been done for this fix - v3.7.2 is out.
allenhumphreys
commented
4 years ago @kushagraThapar Thanks for the fix, we've had this in our staging environment for more than a week and have not encountered it again. If it comes up again, we'll ask to reopen.
chaos-panda
commented
4 years ago We still see the same issue happening with version 3.7.4
Adding stacktrace:
{"level":"ERROR","timestamp":"2020-09-28T00:24:30.315+0000","message":"Network failure","thread":"reactor-http-epoll-2","logger":"com.azure.data.cosmos.internal.RxGatewayStoreModel","stackTrace":" io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1\n\tat io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:74)\n\tat io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:138)\n\tat io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:100)\n\tat io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:88)\n\tat reactor.netty.channel.MonoSendMany$SendManyInner.run(MonoSendMany.java:311)\n\tat reactor.netty.channel.MonoSendMany$SendManyInner.trySchedule(MonoSendMany.java:419)\n\tat reactor.netty.channel.MonoSendMany$SendManyInner.onSubscribe(MonoSendMany.java:236)\n\tat reactor.core.publisher.FluxJust.subscribe(FluxJust.java:70)\n\tat reactor.core.publisher.Flux.subscribe(Flux.java:8325)\n\tat reactor.netty.channel.MonoSendMany.subscribe(MonoSendMany.java:102)\n\tat reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:153)\n\tat reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)\n\tat reactor.core.publisher.Mono.subscribe(Mono.java:4213)\n\tat reactor.netty.NettyOutbound.subscribe(NettyOutbound.java:331)\n\tat reactor.core.publisher.MonoSource.subscribe(MonoSource.java:65)\n\tat reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)\n\tat reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:441)\n\tat reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:514)\n\tat reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.run(PooledConnectionProvider.java:612)\n\tat reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onNext(PooledConnectionProvider.java:485)\n\tat reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onNext(PooledConnectionProvider.java:436)\n\tat reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.deliver(AbstractPool.java:400)\n\tat reactor.netty.internal.shaded.reactor.pool.SimplePool.lambda$drainLoop$15(SimplePool.java:297)\n\tat reactor.core.scheduler.ImmediateScheduler.schedule(ImmediateScheduler.java:47)\n\tat reactor.netty.internal.shaded.reactor.pool.SimplePool.drainLoop(SimplePool.java:297)\n\tat reactor.netty.internal.shaded.reactor.pool.SimplePool.drain(SimplePool.java:187)\n\tat reactor.netty.internal.shaded.reactor.pool.SimplePool.doAcquire(SimplePool.java:136)\n\tat reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.request(AbstractPool.java:363)\n\tat reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onSubscribe(PooledConnectionProvider.java:511)\n\tat reactor.netty.internal.shaded.reactor.pool.SimplePool$QueueBorrowerMono.subscribe(SimplePool.java:381)\n\tat reactor.netty.resources.PooledConnectionProvider.disposableAcquire(PooledConnectionProvider.java:197)\n\tat reactor.netty.resources.PooledConnectionProvider.lambda$acquire$3(PooledConnectionProvider.java:161)\n\tat reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:57)\n\tat reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.lambda$subscribe$0(HttpClientConnect.java:326)\n\tat reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:57)\n\tat reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.resubscribe(FluxRetryWhen.java:204)\n\tat reactor.core.publisher.FluxRetryWhen$RetryWhenOtherSubscriber.onNext(FluxRetryWhen.java:250)\n\tat reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.innerNext(FluxConcatMap.java:274)\n\tat reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:851)\n\tat reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)\n\tat reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241)\n\tat reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)\n\tat reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:147)\n\tat reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)\n\tat reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)\n\tat reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782)\n\tat reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:147)\n\tat reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)\n\tat reactor.core.publisher.Mono.subscribe(Mono.java:4213)\n\tat reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.drain(FluxConcatMap.java:441)\n\tat reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onNext(FluxConcatMap.java:243)\n\tat reactor.core.publisher.DirectProcessor$DirectInner.onNext(DirectProcessor.java:333)\n\tat reactor.core.publisher.DirectProcessor.onNext(DirectProcessor.java:142)\n\tat reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)\n\tat reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onError(FluxRetryWhen.java:180)\n\tat reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:185)\n\tat reactor.netty.http.client.HttpClientConnect$HttpObserver.onUncaughtException(HttpClientConnect.java:402)\n\tat reactor.netty.ReactorNetty$CompositeConnectionObserver.onUncaughtException(ReactorNetty.java:507)\n\tat reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onUncaughtException(PooledConnectionProvider.java:528)\n\tat reactor.netty.resources.PooledConnectionProvider$PooledConnection.onUncaughtException(PooledConnectionProvider.java:364)\n\tat reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:225)\n\tat reactor.netty.channel.FluxReceive.onInboundError(FluxReceive.java:430)\n\tat reactor.netty.channel.ChannelOperations.onInboundError(ChannelOperations.java:442)\n\tat reactor.netty.channel.ChannelOperationsHandler.exceptionCaught(ChannelOperationsHandler.java:129)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273)\n\tat io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireExceptionCaught(CombinedChannelDuplexHandler.java:424)\n\tat io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:92)\n\tat io.netty.channel.CombinedChannelDuplexHandler$1.fireExceptionCaught(CombinedChannelDuplexHandler.java:145)\n\tat io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)\n\tat io.netty.channel.CombinedChannelDuplexHandler.exceptionCaught(CombinedChannelDuplexHandler.java:231)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273)\n\tat io.netty.handler.ssl.SslHandler.exceptionCaught(SslHandler.java:1144)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:273)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:302)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)\n\tat io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907)\n\tat io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.handleReadException(AbstractEpollStreamChannel.java:728)\n\tat io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:818)\n\tat io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Unknown Source)\n"}
Describe the bug Sporadically we're receiving exceptions from Cosmos due to an illegal referent count on a netty byte buffer. We are using direct connectivity.
Exception or Stack Trace
Stack Trace
``` io.netty.util.IllegalReferenceCountException: refCnt: 0 at io.netty.buffer.AbstractByteBuf.ensureAccessible(AbstractByteBuf.java:1489) at io.netty.buffer.AbstractByteBuf.slice(AbstractByteBuf.java:1221) at io.netty.buffer.AbstractByteBuf.slice(AbstractByteBuf.java:1211) at io.netty.buffer.Unpooled.wrappedBuffer(Unpooled.java:229) at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:100) at jdk.internal.reflect.GeneratedMethodAccessor99.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at.common.logging.hooks.CoreSubscriberProxyInvocationHandler.wrappedExceptionInvoke(CoreSubscriberProxyInvocationHandler.java:61)
at .common.logging.hooks.CoreSubscriberProxyInvocationHandler.lambda$handleInvocation$0(CoreSubscriberProxyInvocationHandler.java:48)
at .common.logging.hooks.SubscriberProxyingInterceptor$proxySubscribe$proxy$1.accept(SubscriberProxyingInterceptor.kt:25)
at .common.logging.hooks.SubscriberProxyingInterceptor$proxySubscribe$proxy$1.accept(SubscriberProxyingInterceptor.kt:11)
at .common.logging.hooks.CoreSubscriberProxyInvocationHandler.handleInvocation(CoreSubscriberProxyInvocationHandler.java:45)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:86)
at reactor.core.publisher.$Proxy217.onNext(Unknown Source)
at reactor.core.publisher.FluxJust$WeakScalarSubscription.request(FluxJust.java:99)
at reactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:155)
at reactor.netty.channel.MonoSendMany$SendManyInner.onSubscribe(MonoSendMany.java:226)
at reactor.core.publisher.FluxMap$MapSubscriber.onSubscribe(FluxMap.java:86)
at jdk.internal.reflect.GeneratedMethodAccessor97.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at .common.logging.hooks.CoreSubscriberProxyInvocationHandler.wrappedExceptionInvoke(CoreSubscriberProxyInvocationHandler.java:61)
at .common.logging.hooks.CoreSubscriberProxyInvocationHandler.lambda$handleInvocation$0(CoreSubscriberProxyInvocationHandler.java:48)
at .common.logging.hooks.SubscriberProxyingInterceptor$proxySubscribe$proxy$1.accept(SubscriberProxyingInterceptor.kt:25)
at .common.logging.hooks.SubscriberProxyingInterceptor$proxySubscribe$proxy$1.accept(SubscriberProxyingInterceptor.kt:11)
at .common.logging.hooks.CoreSubscriberProxyInvocationHandler.handleInvocation(CoreSubscriberProxyInvocationHandler.java:45)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:86)
at reactor.core.publisher.$Proxy217.onSubscribe(Unknown Source)
at reactor.core.publisher.FluxJust.subscribe(FluxJust.java:70)
at .common.logging.hooks.FluxInterceptor.subscribe(FluxInterceptor.kt:13)
at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:53)
at .common.logging.hooks.FluxInterceptor.subscribe(FluxInterceptor.kt:13)
at reactor.core.publisher.Flux.subscribe(Flux.java:8186)
at reactor.netty.channel.MonoSendMany.subscribe(MonoSendMany.java:82)
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.drain(MonoIgnoreThen.java:153)
at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:56)
at .common.logging.hooks.MonoInterceptor.subscribe(MonoInterceptor.kt:13)
at reactor.core.publisher.Mono.subscribe(Mono.java:4110)
at reactor.netty.NettyOutbound.subscribe(NettyOutbound.java:329)
at reactor.core.publisher.MonoSource.subscribe(MonoSource.java:58)
at .common.logging.hooks.MonoInterceptor.subscribe(MonoInterceptor.kt:13)
at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
at .common.logging.hooks.MonoInterceptor.subscribe(MonoInterceptor.kt:13)
at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:430)
at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:503)
at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.run(PooledConnectionProvider.java:571)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
```
To Reproduce Steps to reproduce the behavior: Unsure
Screenshots If applicable, add screenshots to help explain your problem.
Setup (please complete the following information):
Additional context
This is a continuation of #9023 but may be entirely unrelated.
Cosmos Client Initialization Logs
--2020-04-02 00:52:16.453 - INFO -- [ main] c.a.d.c.internal.RxDocumentClientImpl : Initializing DocumentClient with serviceEndpoint [https://stgsupstgcu0-cosmosdb0.documents.azure.com:443/], connectionPolicy [ConnectionPolicy{requestTimeoutInMillis=60000, mediaRequestTimeoutInMillis=300000, connectionMode=DIRECT, maxPoolSize=1000, idleConnectionTimeoutInMillis=60000, userAgentSuffix=';spring-data/2.2.0;bc335097171716bed6ad27b6b11228fd5aa4ff258612c0545b460fa678b628f9;;spring-data/2.2.0;bc335097171716bed6ad27b6b11228fd5aa4ff258612c0545b460fa678b628f9;', retryOptions=RetryOptions{maxRetryAttemptsOnThrottledRequests=9, maxRetryWaitTimeInSeconds=30}, enableEndpointDiscovery=true, preferredLocations=[], usingMultipleWriteLocations=false, inetSocketProxyAddress=null}], consistencyLevel [null], directModeProtocol [Tcp] --2020-04-02 00:52:16.464 - INFO -- [ parallel-1] c.a.d.c.internal.RxDocumentClientImpl : Getting database account endpoint from https://:443/
--2020-04-02 00:52:16.563 - INFO -- [ main] c.a.d.c.internal.RxDocumentClientImpl : Initializing DocumentClient with serviceEndpoint [https://:443/], connectionPolicy [ConnectionPolicy{requestTimeoutInMillis=60000, mediaRequestTimeoutInMillis=300000, connectionMode=DIRECT, maxPoolSize=1000, idleConnectionTimeoutInMillis=60000, userAgentSuffix=';spring-data/2.2.0;bc335097171716bed6ad27b6b11228fd5aa4ff258612c0545b460fa678b628f9;;spring-data/2.2.0;bc335097171716bed6ad27b6b11228fd5aa4ff258612c0545b460fa678b628f9;;spring-data/2.2.0;bc335097171716bed6ad27b6b11228fd5aa4ff258612c0545b460fa678b628f9;', retryOptions=RetryOptions{maxRetryAttemptsOnThrottledRequests=9, maxRetryWaitTimeInSeconds=30}, enableEndpointDiscovery=true, preferredLocations=[], usingMultipleWriteLocations=false, inetSocketProxyAddress=null}], consistencyLevel [null], directModeProtocol [Tcp]
--2020-04-02 00:52:16.564 - INFO -- [ parallel-1] c.a.d.c.internal.RxDocumentClientImpl : Getting database account endpoint from https://:443/
--2020-04-02 00:52:16.609 - INFO -- [ main] c.a.d.c.internal.RxDocumentClientImpl : Initializing DocumentClient with serviceEndpoint [https://:443/], connectionPolicy [ConnectionPolicy{requestTimeoutInMillis=60000, mediaRequestTimeoutInMillis=300000, connectionMode=DIRECT, maxPoolSize=1000, idleConnectionTimeoutInMillis=60000, userAgentSuffix=';spring-data/2.2.0;bc335097171716bed6ad27b6b11228fd5aa4ff258612c0545b460fa678b628f9;;spring-data/2.2.0;bc335097171716bed6ad27b6b11228fd5aa4ff258612c0545b460fa678b628f9;;spring-data/2.2.0;bc335097171716bed6ad27b6b11228fd5aa4ff258612c0545b460fa678b628f9;;spring-data/2.2.0;bc335097171716bed6ad27b6b11228fd5aa4ff258612c0545b460fa678b628f9;', retryOptions=RetryOptions{maxRetryAttemptsOnThrottledRequests=9, maxRetryWaitTimeInSeconds=30}, enableEndpointDiscovery=true, preferredLocations=[], usingMultipleWriteLocations=false, inetSocketProxyAddress=null}], consistencyLevel [null], directModeProtocol [Tcp]
Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report