[BUG] Cosmos SDK unhandled exceptions in direct connectivity "Operator called default onErrorDropped"

[allenhumphreys]

Describe the bug We are seeing very frequent error logs from Reactor's onErrorDropped global error handler. I believe the situation is resulting in an uncompleted Mono. The HTTP requests that trigger the cosmos calls then reach their configured timeout and users receive 500 exceptions.

Exception or Stack Trace Add the exception log and stack trace if available

Stack Trace

``` c.a.d.c.i.d.r.RntbdClientChannelPool : channel acquisition failed due to: - io.netty.channel.ConnectTimeoutException: connection timed out: /:14473 at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261) at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:834) -2020-04-21 19:16:49.939 -ERROR -- [rntbd-nio-2-1] reactor.core.publisher.Operators : Operator called default onErrorDropped - java.util.concurrent.CompletionException: GoneException{error=null, resourceAddress='/apps/9d960ad9-f20b-4902-93d4-459ab6603ec2/services/7aa56944-5b55-455b-85b8-cc3dcd17de9e/partitions/705842c7-2d2a-4249-aa2d-efb29d8ceb1c/replicas/132294879738889176s', statusCode=410, message=failed to establish connection to :14473: connection timed out: /:14473, causeInfo=[class: class io.netty.channel.ConnectTimeoutException, message: connection timed out: /:14473], responseHeaders={x-ms-activity-id=82ea9a4f-8404-11ea-8625-2947298209af}, requestHeaders={Accept=application/json, x-ms-client-retry-attempt-count=0, x-ms-continuation=null, x-ms-documentdb-query-enablecrosspartition=true, x-ms-documentdb-query-enable-scan=true, authorization=, x-ms-date=Tue, 21 Apr 2020 19:15:49 GMT, x-ms-documentdb-collection-rid=oiIHAN4yIyA=, x-ms-documentdb-partitionkeyrangeid=oiIHAN4yIyA=,3, x-ms-documentdb-isquery=true, x-ms-max-item-count=1, x-ms-remaining-time-in-ms-on-client=60000, x-ms-documentdb-populatequerymetrics=true, Content-Type=application/query+json}} at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:870) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) at com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdServiceEndpoint.lambda$write$2(RntbdServiceEndpoint.java:266) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) at io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:109) at com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdClientChannelPool$AcquireListener.fail(RntbdClientChannelPool.java:592) at com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdClientChannelPool$AcquireListener.operationComplete(RntbdClientChannelPool.java:582) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:551) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) at io.netty.channel.pool.SimpleChannelPool.notifyConnect(SimpleChannelPool.java:216) at io.netty.channel.pool.SimpleChannelPool.access$000(SimpleChannelPool.java:43) at io.netty.channel.pool.SimpleChannelPool$2.operationComplete(SimpleChannelPool.java:184) at io.netty.channel.pool.SimpleChannelPool$2.operationComplete(SimpleChannelPool.java:181) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577) at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490) at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615) at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:608) at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:263) at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: com.azure.data.cosmos.GoneException: failed to establish connection to :14473: connection timed out: /:14473 at com.azure.data.cosmos.internal.directconnectivity.rntbd.RntbdServiceEndpoint.lambda$write$2(RntbdServiceEndpoint.java:262) ... 35 common frames omitted Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: /:14473 at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:261) ... 9 common frames omitted ```

To Reproduce Steps to reproduce the behavior: Unsure

Expected behavior Exceptions should be handled, the request should be retried or failed. Setup (please complete the following information):

• OS: Linux/macOS
• IDE : IntelliJ
• Version of the Library used: 3.7.2

Additional context Add any other context about the problem here.

Gradle Dependencies

``` +--- com.:-common:1.0.451 | +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.5 -> 1.3.3 | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.61 -> 1.3.71 | | | +--- org.jetbrains.kotlin:kotlin-stdlib-common:1.3.71 | | | \--- org.jetbrains:annotations:13.0 | | \--- org.jetbrains.kotlin:kotlin-stdlib-common:1.3.61 -> 1.3.71 | +--- org.jetbrains.kotlinx:kotlinx-coroutines-reactor:1.3.5 -> 1.3.3 | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.61 -> 1.3.71 (*) | | +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.3 (*) | | +--- io.projectreactor:reactor-core:3.2.5.RELEASE -> 3.3.3.RELEASE | | | \--- org.reactivestreams:reactive-streams:1.0.3 | | \--- org.jetbrains.kotlinx:kotlinx-coroutines-reactive:1.3.3 | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.61 -> 1.3.71 (*) | | +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.3 (*) | | \--- org.reactivestreams:reactive-streams:1.0.2 -> 1.0.3 | +--- org.jetbrains.kotlinx:kotlinx-coroutines-slf4j:1.3.5 -> 1.3.3 | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.61 -> 1.3.71 (*) | | +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.3 (*) | | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | +--- org.jetbrains.kotlinx:kotlinx-coroutines-rx2:1.3.5 -> 1.3.3 | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.61 -> 1.3.71 (*) | | +--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.3 (*) | | +--- org.jetbrains.kotlinx:kotlinx-coroutines-reactive:1.3.3 (*) | | \--- io.reactivex.rxjava2:rxjava:2.2.8 -> 2.2.17 | | \--- org.reactivestreams:reactive-streams:1.0.3 | +--- org.jetbrains.kotlinx:kotlinx-coroutines-jdk8:1.3.5 -> 1.3.3 | | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.61 -> 1.3.71 (*) | | \--- org.jetbrains.kotlinx:kotlinx-coroutines-core:1.3.3 (*) | +--- com.fasterxml.jackson.module:jackson-module-kotlin:2.10.3 -> 2.10.2 | | +--- com.fasterxml.jackson.core:jackson-databind:2.10.2 | | | +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2 | | | \--- com.fasterxml.jackson.core:jackson-core:2.10.2 | | +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2 | | \--- org.jetbrains.kotlin:kotlin-reflect:1.3.61 -> 1.3.71 | | \--- org.jetbrains.kotlin:kotlin-stdlib:1.3.71 (*) | +--- com.microsoft.azure:azure-servicebus-spring-boot-starter:2.2.1 | | +--- org.springframework.boot:spring-boot-starter:2.2.0.RELEASE -> 2.2.4.RELEASE | | | +--- org.springframework.boot:spring-boot:2.2.4.RELEASE | | | | +--- org.springframework:spring-core:5.2.3.RELEASE | | | | | \--- org.springframework:spring-jcl:5.2.3.RELEASE | | | | \--- org.springframework:spring-context:5.2.3.RELEASE | | | | +--- org.springframework:spring-aop:5.2.3.RELEASE | | | | | +--- org.springframework:spring-beans:5.2.3.RELEASE | | | | | | \--- org.springframework:spring-core:5.2.3.RELEASE (*) | | | | | \--- org.springframework:spring-core:5.2.3.RELEASE (*) | | | | +--- org.springframework:spring-beans:5.2.3.RELEASE (*) | | | | +--- org.springframework:spring-core:5.2.3.RELEASE (*) | | | | \--- org.springframework:spring-expression:5.2.3.RELEASE | | | | \--- org.springframework:spring-core:5.2.3.RELEASE (*) | | | +--- org.springframework.boot:spring-boot-autoconfigure:2.2.4.RELEASE | | | | \--- org.springframework.boot:spring-boot:2.2.4.RELEASE (*) | | | +--- org.springframework.boot:spring-boot-starter-logging:2.2.4.RELEASE | | | | +--- ch.qos.logback:logback-classic:1.2.3 | | | | | +--- ch.qos.logback:logback-core:1.2.3 | | | | | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | | | +--- org.apache.logging.log4j:log4j-to-slf4j:2.12.1 | | | | | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | | | | \--- org.apache.logging.log4j:log4j-api:2.12.1 | | | | \--- org.slf4j:jul-to-slf4j:1.7.30 | | | | \--- org.slf4j:slf4j-api:1.7.30 | | | +--- jakarta.annotation:jakarta.annotation-api:1.3.5 | | | +--- org.springframework:spring-core:5.2.3.RELEASE (*) | | | \--- org.yaml:snakeyaml:1.25 | | +--- org.springframework.boot:spring-boot-starter-validation:2.2.0.RELEASE -> 2.2.4.RELEASE | | | +--- org.springframework.boot:spring-boot-starter:2.2.4.RELEASE (*) | | | +--- jakarta.validation:jakarta.validation-api:2.0.2 | | | +--- org.apache.tomcat.embed:tomcat-embed-el:9.0.30 | | | \--- org.hibernate.validator:hibernate-validator:6.0.18.Final | | | +--- org.jboss.logging:jboss-logging:3.3.2.Final -> 3.4.1.Final | | | \--- com.fasterxml:classmate:1.3.4 -> 1.5.1 | | +--- com.microsoft.azure:azure-spring-boot:2.2.1 | | | +--- org.springframework.boot:spring-boot-autoconfigure:2.2.0.RELEASE -> 2.2.4.RELEASE (*) | | | +--- org.slf4j:slf4j-api:1.7.28 -> 1.7.30 | | | +--- org.springframework:spring-web:5.2.0.RELEASE -> 5.2.3.RELEASE | | | | +--- org.springframework:spring-beans:5.2.3.RELEASE (*) | | | | \--- org.springframework:spring-core:5.2.3.RELEASE (*) | | | \--- com.fasterxml.jackson.core:jackson-databind:2.10.0 -> 2.10.2 (*) | | +--- com.microsoft.azure:azure-servicebus:1.2.15 -> 1.2.18 | | | +--- org.apache.qpid:proton-j:0.31.0 | | | +--- org.slf4j:slf4j-api:1.7.0 -> 1.7.30 | | | \--- com.microsoft.azure:adal4j:1.3.0 -> 1.6.4 | | | +--- com.nimbusds:oauth2-oidc-sdk:6.5 | | | | +--- com.sun.mail:javax.mail:1.6.1 | | | | | \--- javax.activation:activation:1.1 | | | | +--- com.github.stephenc.jcip:jcip-annotations:1.0-1 | | | | +--- net.minidev:json-smart:[1.3.1,2.3] -> 2.3 | | | | | \--- net.minidev:accessors-smart:1.2 | | | | | \--- org.ow2.asm:asm:5.0.4 | | | | +--- com.nimbusds:lang-tag:[1.4.3,) -> 1.4.4 | | | | \--- com.nimbusds:nimbus-jose-jwt:[6.0.1,) -> 8.9 | | | | +--- com.github.stephenc.jcip:jcip-annotations:1.0-1 | | | | \--- net.minidev:json-smart:[1.3.1,2.3] -> 2.3 (*) | | | +--- com.google.code.gson:gson:2.8.0 -> 2.8.6 | | | +--- org.slf4j:slf4j-api:1.7.21 -> 1.7.30 | | | +--- commons-codec:commons-codec:1.11 -> 1.13 | | | \--- org.apache.commons:commons-lang3:3.5 -> 3.9 | | +--- com.google.code.findbugs:annotations:2.0.1 | | +--- com.fasterxml.jackson.core:jackson-core:2.10.0 -> 2.10.2 | | \--- commons-codec:commons-codec:1.13 | +--- com.microsoft.azure:azure-servicebus:1.2.18 (*) | +--- com.microsoft.azure:applicationinsights-logging-logback:2.5.1 | | +--- com.microsoft.azure:applicationinsights-core:2.5.1 | | +--- ch.qos.logback:logback-classic:1.2.3 (*) | | \--- ch.qos.logback:logback-core:1.2.3 | +--- org.springframework.boot:spring-boot-starter-security:2.2.4.RELEASE | | +--- org.springframework.boot:spring-boot-starter:2.2.4.RELEASE (*) | | +--- org.springframework:spring-aop:5.2.3.RELEASE (*) | | +--- org.springframework.security:spring-security-config:5.2.1.RELEASE | | | +--- org.springframework.security:spring-security-core:5.2.1.RELEASE | | | | +--- org.springframework:spring-aop:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | | | +--- org.springframework:spring-beans:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | | | +--- org.springframework:spring-context:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | | | +--- org.springframework:spring-core:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | | | \--- org.springframework:spring-expression:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | | +--- org.springframework:spring-aop:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | | +--- org.springframework:spring-beans:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | | +--- org.springframework:spring-context:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | | \--- org.springframework:spring-core:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | \--- org.springframework.security:spring-security-web:5.2.1.RELEASE | | +--- org.springframework.security:spring-security-core:5.2.1.RELEASE (*) | | +--- org.springframework:spring-aop:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | +--- org.springframework:spring-beans:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | +--- org.springframework:spring-context:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | +--- org.springframework:spring-core:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | +--- org.springframework:spring-expression:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | \--- org.springframework:spring-web:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | +--- io.projectreactor:reactor-core:3.3.3.RELEASE (*) | +--- io.projectreactor.addons:reactor-adapter:3.3.2.RELEASE | | \--- io.projectreactor:reactor-core:3.3.2.RELEASE -> 3.3.3.RELEASE (*) | +--- io.projectreactor.addons:reactor-extra:3.3.2.RELEASE | | \--- io.projectreactor:reactor-core:3.3.2.RELEASE -> 3.3.3.RELEASE (*) | +--- io.projectreactor.kotlin:reactor-kotlin-extensions:1.0.2.RELEASE | | \--- io.projectreactor:reactor-core:3.3.2.RELEASE -> 3.3.3.RELEASE (*) | +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.71 (*) | +--- org.jetbrains.kotlin:kotlin-reflect:1.3.71 (*) | +--- net.minidev:json-smart:2.3 (*) | +--- com.microsoft.azure:azure-eventgrid:1.3.0 | | \--- com.microsoft.azure:azure-client-runtime:1.5.0 -> 1.6.13 | | \--- com.microsoft.rest:client-runtime:1.6.13 | | +--- com.google.guava:guava:20.0 -> 28.2-jre | | | +--- com.google.guava:failureaccess:1.0.1 | | | +--- com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava | | | +--- com.google.code.findbugs:jsr305:3.0.2 | | | +--- org.checkerframework:checker-qual:2.10.0 | | | +--- com.google.errorprone:error_prone_annotations:2.3.4 | | | \--- com.google.j2objc:j2objc-annotations:1.3 | | +--- com.squareup.retrofit2:retrofit:2.5.0 | | | \--- com.squareup.okhttp3:okhttp:3.12.0 -> 3.12.2 | | | \--- com.squareup.okio:okio:1.15.0 | | +--- com.squareup.okhttp3:okhttp:3.12.2 (*) | | +--- com.squareup.okhttp3:logging-interceptor:3.12.2 -> 3.12.0 | | | \--- com.squareup.okhttp3:okhttp:3.12.0 -> 3.12.2 (*) | | +--- com.squareup.okhttp3:okhttp-urlconnection:3.12.2 -> 3.14.6 | | | \--- com.squareup.okhttp3:okhttp:3.14.6 -> 3.12.2 (*) | | +--- com.squareup.retrofit2:converter-jackson:2.5.0 | | | +--- com.squareup.retrofit2:retrofit:2.5.0 (*) | | | \--- com.fasterxml.jackson.core:jackson-databind:2.9.4 -> 2.10.2 (*) | | +--- com.fasterxml.jackson.datatype:jackson-datatype-joda:2.9.8 -> 2.10.2 | | | +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2 | | | +--- com.fasterxml.jackson.core:jackson-core:2.10.2 | | | +--- com.fasterxml.jackson.core:jackson-databind:2.10.2 (*) | | | \--- joda-time:joda-time:2.9.9 -> 2.10.5 | | +--- org.apache.commons:commons-lang3:3.4 -> 3.9 | | +--- io.reactivex:rxjava:1.3.8 | | +--- com.squareup.retrofit2:adapter-rxjava:2.4.0 | | | +--- com.squareup.retrofit2:retrofit:2.4.0 -> 2.5.0 (*) | | | \--- io.reactivex:rxjava:1.3.0 -> 1.3.8 | | +--- org.slf4j:slf4j-api:1.7.22 -> 1.7.30 | | \--- com.microsoft.azure:azure-annotations:1.7.0 | +--- com.microsoft.azure:azure-cosmos:3.7.2 | | +--- com.fasterxml.jackson.core:jackson-databind:2.10.1 -> 2.10.2 (*) | | +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.1 -> 2.10.2 | | | +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2 | | | +--- com.fasterxml.jackson.core:jackson-core:2.10.2 | | | \--- com.fasterxml.jackson.core:jackson-databind:2.10.2 (*) | | +--- com.fasterxml.uuid:java-uuid-generator:3.2.0 | | +--- commons-io:commons-io:2.5 | | +--- commons-validator:commons-validator:1.6 | | | +--- commons-digester:commons-digester:1.8.1 | | | +--- commons-logging:commons-logging:1.2 | | | \--- commons-collections:commons-collections:3.2.2 | | +--- io.projectreactor:reactor-core:3.3.0.RELEASE -> 3.3.3.RELEASE (*) | | +--- io.netty:netty-codec-http:4.1.44.Final -> 4.1.45.Final | | | +--- io.netty:netty-common:4.1.45.Final | | | +--- io.netty:netty-buffer:4.1.45.Final | | | | \--- io.netty:netty-common:4.1.45.Final | | | +--- io.netty:netty-transport:4.1.45.Final | | | | +--- io.netty:netty-common:4.1.45.Final | | | | +--- io.netty:netty-buffer:4.1.45.Final (*) | | | | \--- io.netty:netty-resolver:4.1.45.Final | | | | \--- io.netty:netty-common:4.1.45.Final | | | +--- io.netty:netty-codec:4.1.45.Final | | | | +--- io.netty:netty-common:4.1.45.Final | | | | +--- io.netty:netty-buffer:4.1.45.Final (*) | | | | \--- io.netty:netty-transport:4.1.45.Final (*) | | | \--- io.netty:netty-handler:4.1.45.Final | | | +--- io.netty:netty-common:4.1.45.Final | | | +--- io.netty:netty-buffer:4.1.45.Final (*) | | | +--- io.netty:netty-transport:4.1.45.Final (*) | | | \--- io.netty:netty-codec:4.1.45.Final (*) | | +--- io.netty:netty-handler:4.1.44.Final -> 4.1.45.Final (*) | | +--- org.slf4j:slf4j-api:1.7.28 -> 1.7.30 | | +--- org.apache.commons:commons-lang3:3.8.1 -> 3.9 | | +--- org.apache.commons:commons-collections4:4.2 | | +--- org.apache.commons:commons-text:1.6 | | | \--- org.apache.commons:commons-lang3:3.8.1 -> 3.9 | | +--- com.google.guava:guava:25.0-jre -> 28.2-jre (*) | | +--- io.dropwizard.metrics:metrics-core:4.1.0 -> 4.1.2 | | | \--- org.slf4j:slf4j-api:1.7.29 -> 1.7.30 | | +--- io.micrometer:micrometer-core:1.2.0 -> 1.3.2 | | | +--- org.hdrhistogram:HdrHistogram:2.1.11 | | | \--- org.latencyutils:LatencyUtils:2.0.3 | | +--- io.reactivex.rxjava2:rxjava:2.2.4 -> 2.2.17 (*) | | \--- io.projectreactor.netty:reactor-netty:0.9.4.RELEASE | | +--- io.netty:netty-codec-http:4.1.44.Final -> 4.1.45.Final (*) | | +--- io.netty:netty-codec-http2:4.1.44.Final -> 4.1.45.Final | | | +--- io.netty:netty-common:4.1.45.Final | | | +--- io.netty:netty-buffer:4.1.45.Final (*) | | | +--- io.netty:netty-transport:4.1.45.Final (*) | | | +--- io.netty:netty-codec:4.1.45.Final (*) | | | +--- io.netty:netty-handler:4.1.45.Final (*) | | | \--- io.netty:netty-codec-http:4.1.45.Final (*) | | +--- io.netty:netty-handler:4.1.44.Final -> 4.1.45.Final (*) | | +--- io.netty:netty-handler-proxy:4.1.44.Final -> 4.1.45.Final | | | +--- io.netty:netty-common:4.1.45.Final | | | +--- io.netty:netty-buffer:4.1.45.Final (*) | | | +--- io.netty:netty-transport:4.1.45.Final (*) | | | +--- io.netty:netty-codec:4.1.45.Final (*) | | | +--- io.netty:netty-codec-socks:4.1.45.Final | | | | +--- io.netty:netty-common:4.1.45.Final | | | | +--- io.netty:netty-buffer:4.1.45.Final (*) | | | | +--- io.netty:netty-transport:4.1.45.Final (*) | | | | \--- io.netty:netty-codec:4.1.45.Final (*) | | | \--- io.netty:netty-codec-http:4.1.45.Final (*) | | +--- io.netty:netty-transport-native-epoll:4.1.44.Final -> 4.1.45.Final | | | +--- io.netty:netty-common:4.1.45.Final | | | +--- io.netty:netty-buffer:4.1.45.Final (*) | | | +--- io.netty:netty-transport:4.1.45.Final (*) | | | \--- io.netty:netty-transport-native-unix-common:4.1.45.Final | | | +--- io.netty:netty-common:4.1.45.Final | | | +--- io.netty:netty-buffer:4.1.45.Final (*) | | | \--- io.netty:netty-transport:4.1.45.Final (*) | | \--- io.projectreactor:reactor-core:3.3.2.RELEASE -> 3.3.3.RELEASE (*) | +--- com.microsoft.azure:azure-cosmosdb-spring-boot-starter:2.2.1 | | +--- org.springframework.boot:spring-boot-starter:2.2.0.RELEASE -> 2.2.4.RELEASE (*) | | +--- org.springframework.boot:spring-boot-starter-validation:2.2.0.RELEASE -> 2.2.4.RELEASE (*) | | +--- com.microsoft.azure:azure-spring-boot:2.2.1 (*) | | \--- com.microsoft.azure:spring-data-cosmosdb:2.2.0 | | +--- org.springframework:spring-core:5.2.0.RELEASE -> 5.2.3.RELEASE (*) | | +--- org.springframework:spring-web:5.2.0.RELEASE -> 5.2.3.RELEASE (*) | | +--- org.springframework:spring-beans:5.2.0.RELEASE -> 5.2.3.RELEASE (*) | | +--- org.springframework:spring-context:5.2.0.RELEASE -> 5.2.3.RELEASE (*) | | +--- org.springframework:spring-tx:5.2.0.RELEASE -> 5.2.3.RELEASE | | | +--- org.springframework:spring-beans:5.2.3.RELEASE (*) | | | \--- org.springframework:spring-core:5.2.3.RELEASE (*) | | +--- org.springframework.data:spring-data-commons:2.2.0.RELEASE -> 2.2.5.RELEASE | | | +--- org.springframework:spring-core:5.2.4.RELEASE -> 5.2.3.RELEASE (*) | | | +--- org.springframework:spring-beans:5.2.4.RELEASE -> 5.2.3.RELEASE (*) | | | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30 | | +--- org.springframework:spring-expression:5.2.0.RELEASE -> 5.2.3.RELEASE (*) | | +--- com.microsoft.azure:azure-cosmos:3.3.1 -> 3.7.2 (*) | | +--- com.fasterxml.jackson.module:jackson-module-parameter-names:2.9.5 -> 2.10.2 | | | +--- com.fasterxml.jackson.core:jackson-core:2.10.2 | | | \--- com.fasterxml.jackson.core:jackson-databind:2.10.2 (*) | | +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.9.5 -> 2.10.2 | | | +--- com.fasterxml.jackson.core:jackson-core:2.10.2 | | | \--- com.fasterxml.jackson.core:jackson-databind:2.10.2 (*) | | +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.9.5 -> 2.10.2 (*) | | +--- org.json:json:20140107 | | \--- org.javatuples:javatuples:1.2 | +--- com.microsoft.azure:applicationinsights-spring-boot-starter:2.5.1 | | +--- com.microsoft.azure:applicationinsights-core:2.5.1 | | \--- com.microsoft.azure:applicationinsights-web:2.5.1 | +--- com.microsoft.azure:azure-keyvault:1.2.2 | | +--- com.microsoft.azure:azure-keyvault-cryptography:1.2.2 | | | +--- com.microsoft.azure:azure-keyvault-webkey:1.2.2 | | | | +--- com.fasterxml.jackson.core:jackson-databind:2.9.9 -> 2.10.2 (*) | | | | +--- com.fasterxml.jackson.core:jackson-core:2.9.9 -> 2.10.2 | | | | +--- com.fasterxml.jackson.core:jackson-annotations:2.9.9 -> 2.10.2 | | | | +--- commons-codec:commons-codec:1.10 -> 1.13 | | | | \--- com.google.guava:guava:24.1.1-jre -> 28.2-jre (*) | | | +--- com.microsoft.azure:azure-keyvault-core:1.2.2 | | | | +--- org.apache.commons:commons-lang3:3.8.1 -> 3.9 | | | | \--- com.google.guava:guava:24.1.1-jre -> 28.2-jre (*) | | | +--- com.fasterxml.jackson.core:jackson-databind:2.9.9 -> 2.10.2 (*) | | | +--- org.apache.commons:commons-lang3:3.8.1 -> 3.9 | | | +--- commons-codec:commons-codec:1.10 -> 1.13 | | | \--- com.google.guava:guava:24.1.1-jre -> 28.2-jre (*) | | +--- com.microsoft.azure:azure-keyvault-webkey:1.2.2 (*) | | +--- com.microsoft.azure:azure-client-runtime:1.6.12 -> 1.6.13 (*) | | \--- com.microsoft.rest:client-runtime:1.6.12 -> 1.6.13 (*) | +--- org.springframework.boot:spring-boot-starter-actuator:2.2.4.RELEASE | | +--- org.springframework.boot:spring-boot-starter:2.2.4.RELEASE (*) | | +--- org.springframework.boot:spring-boot-actuator-autoconfigure:2.2.4.RELEASE | | | +--- org.springframework.boot:spring-boot-actuator:2.2.4.RELEASE | | | | +--- org.springframework.boot:spring-boot:2.2.4.RELEASE (*) | | | | \--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.2 (*) | | | +--- org.springframework.boot:spring-boot-autoconfigure:2.2.4.RELEASE (*) | | | +--- com.fasterxml.jackson.core:jackson-databind:2.10.2 (*) | | | +--- org.springframework:spring-core:5.2.3.RELEASE (*) | | | +--- org.springframework:spring-context:5.2.3.RELEASE (*) | | | \--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.2 (*) | | \--- io.micrometer:micrometer-core:1.3.2 (*) | +--- org.springframework.boot:spring-boot-starter-webflux:2.2.4.RELEASE | | +--- org.springframework.boot:spring-boot-starter:2.2.4.RELEASE (*) | | +--- org.springframework.boot:spring-boot-starter-json:2.2.4.RELEASE | | | +--- org.springframework.boot:spring-boot-starter:2.2.4.RELEASE (*) | | | +--- org.springframework:spring-web:5.2.3.RELEASE (*) | | | +--- com.fasterxml.jackson.core:jackson-databind:2.10.2 (*) | | | +--- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.10.2 (*) | | | +--- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.10.2 (*) | | | \--- com.fasterxml.jackson.module:jackson-module-parameter-names:2.10.2 (*) | | +--- org.springframework.boot:spring-boot-starter-reactor-netty:2.2.4.RELEASE | | | +--- io.projectreactor.netty:reactor-netty:0.9.4.RELEASE (*) | | | \--- org.glassfish:jakarta.el:3.0.3 | | +--- org.springframework.boot:spring-boot-starter-validation:2.2.4.RELEASE (*) | | +--- org.springframework:spring-web:5.2.3.RELEASE (*) | | +--- org.springframework:spring-webflux:5.2.3.RELEASE | | | +--- org.springframework:spring-beans:5.2.3.RELEASE (*) | | | +--- org.springframework:spring-core:5.2.3.RELEASE (*) | | | +--- org.springframework:spring-web:5.2.3.RELEASE (*) | | | \--- io.projectreactor:reactor-core:3.3.2.RELEASE -> 3.3.3.RELEASE (*) | | \--- org.synchronoss.cloud:nio-multipart-parser:1.1.0 | | +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.30 | | \--- org.synchronoss.cloud:nio-stream-storage:1.1.3 | | \--- org.slf4j:slf4j-api:1.7.12 -> 1.7.30 | +--- io.springfox:springfox-swagger2:3.0.0-SNAPSHOT | | +--- io.swagger:swagger-annotations:1.5.20 | | +--- io.swagger:swagger-models:1.5.20 | | | +--- com.fasterxml.jackson.core:jackson-annotations:2.9.5 -> 2.10.2 | | | +--- org.slf4j:slf4j-api:1.7.22 -> 1.7.30 | | | \--- io.swagger:swagger-annotations:1.5.20 | | +--- io.springfox:springfox-spi:3.0.0-SNAPSHOT | | | \--- io.springfox:springfox-core:3.0.0-SNAPSHOT | | | +--- net.bytebuddy:byte-buddy:1.9.10 -> 1.10.6 | | | +--- com.fasterxml:classmate:1.4.0 -> 1.5.1 | | | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | | +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE | | | | +--- org.springframework:spring-beans:5.2.0.RELEASE -> 5.2.3.RELEASE (*) | | | | +--- org.springframework:spring-context:5.2.0.RELEASE -> 5.2.3.RELEASE (*) | | | | +--- org.springframework:spring-aop:5.2.0.RELEASE -> 5.2.3.RELEASE (*) | | | | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | | \--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE | | | +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*) | | | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | +--- io.springfox:springfox-schema:3.0.0-SNAPSHOT | | | +--- io.springfox:springfox-core:3.0.0-SNAPSHOT (*) | | | \--- io.springfox:springfox-spi:3.0.0-SNAPSHOT (*) | | +--- io.springfox:springfox-swagger-common:3.0.0-SNAPSHOT | | | +--- io.swagger:swagger-annotations:1.5.20 | | | +--- io.swagger:swagger-models:1.5.20 (*) | | | +--- io.springfox:springfox-spi:3.0.0-SNAPSHOT (*) | | | +--- io.springfox:springfox-schema:3.0.0-SNAPSHOT (*) | | | +--- io.springfox:springfox-spring-web:3.0.0-SNAPSHOT | | | | +--- io.github.classgraph:classgraph:4.1.7 | | | | +--- com.fasterxml:classmate:1.4.0 -> 1.5.1 | | | | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | | | +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*) | | | | +--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE (*) | | | | \--- io.springfox:springfox-spi:3.0.0-SNAPSHOT (*) | | | +--- com.fasterxml:classmate:1.4.0 -> 1.5.1 | | | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | | +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*) | | | \--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE (*) | | +--- io.springfox:springfox-spring-web:3.0.0-SNAPSHOT (*) | | +--- com.fasterxml:classmate:1.4.0 -> 1.5.1 | | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*) | | +--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE (*) | | \--- org.mapstruct:mapstruct:1.3.1.Final | +--- io.springfox:springfox-spring-webflux:3.0.0-SNAPSHOT | | +--- com.fasterxml:classmate:1.4.0 -> 1.5.1 | | +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 | | +--- org.springframework.plugin:spring-plugin-core:2.0.0.RELEASE (*) | | +--- org.springframework.plugin:spring-plugin-metadata:2.0.0.RELEASE (*) | | +--- io.springfox:springfox-spi:3.0.0-SNAPSHOT (*) | | \--- io.springfox:springfox-spring-web:3.0.0-SNAPSHOT (*) | +--- org.springframework.security:spring-security-oauth2-jose:5.3.0.RELEASE -> 5.2.1.RELEASE | | +--- com.nimbusds:nimbus-jose-jwt:7.8 -> 8.9 (*) | | +--- org.springframework.security:spring-security-core:5.2.1.RELEASE (*) | | +--- org.springframework.security:spring-security-oauth2-core:5.2.1.RELEASE | | | +--- org.springframework.security:spring-security-core:5.2.1.RELEASE (*) | | | +--- org.springframework:spring-core:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | | \--- org.springframework:spring-web:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | | \--- org.springframework:spring-core:5.2.1.RELEASE -> 5.2.3.RELEASE (*) | +--- com.nimbusds:nimbus-jose-jwt:8.9 (*) | +--- com.onelogin:java-saml:2.5.0 | | +--- com.onelogin:java-saml-core:2.5.0 | | | +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.30 | | | +--- joda-time:joda-time:2.10.3 -> 2.10.5 | | | +--- org.apache.commons:commons-lang3:3.4 -> 3.9 | | | +--- org.apache.santuario:xmlsec:2.1.4 | | | | +--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30 | | | | +--- com.fasterxml.woodstox:woodstox-core:5.0.3 | | | | | \--- org.codehaus.woodstox:stax2-api:3.1.4 | | | | \--- commons-codec:commons-codec:1.12 -> 1.13 | | | \--- commons-codec:commons-codec:1.12 -> 1.13 | | +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.30 | | +--- joda-time:joda-time:2.10.3 -> 2.10.5 | | +--- org.apache.commons:commons-lang3:3.4 -> 3.9 | | +--- org.apache.santuario:xmlsec:2.1.4 (*) | | \--- commons-codec:commons-codec:1.12 -> 1.13 | \--- io.reactivex:rxjava-reactive-streams:1.2.1 | +--- io.reactivex:rxjava:1.2.2 -> 1.3.8 | \--- org.reactivestreams:reactive-streams:1.0.0 -> 1.0.3 +--- org.jetbrains.kotlin:kotlin-stdlib:1.3.71 (*) +--- org.jetbrains.kotlin:kotlin-reflect:1.3.71 (*) +--- org.springframework.boot:spring-boot-starter-actuator:2.2.4.RELEASE (*) +--- org.springframework.boot:spring-boot-starter-webflux:2.2.4.RELEASE (*) +--- org.springframework.cloud:spring-cloud-starter-config:2.2.2.RELEASE | +--- org.springframework.cloud:spring-cloud-starter:2.2.2.RELEASE | | +--- org.springframework.boot:spring-boot-starter:2.2.5.RELEASE -> 2.2.4.RELEASE (*) | | +--- org.springframework.cloud:spring-cloud-context:2.2.2.RELEASE | | | \--- org.springframework.security:spring-security-crypto:5.2.2.RELEASE -> 5.2.1.RELEASE | | +--- org.springframework.cloud:spring-cloud-commons:2.2.2.RELEASE | | | \--- org.springframework.security:spring-security-crypto:5.2.2.RELEASE -> 5.2.1.RELEASE | | \--- org.springframework.security:spring-security-rsa:1.0.9.RELEASE | | \--- org.bouncycastle:bcpkix-jdk15on:1.64 | | \--- org.bouncycastle:bcprov-jdk15on:1.64 | +--- org.springframework.cloud:spring-cloud-config-client:2.2.2.RELEASE | | +--- org.springframework.boot:spring-boot-autoconfigure:2.2.5.RELEASE -> 2.2.4.RELEASE (*) | | +--- org.springframework.cloud:spring-cloud-commons:2.2.2.RELEASE (*) | | +--- org.springframework.cloud:spring-cloud-context:2.2.2.RELEASE (*) | | +--- org.springframework:spring-web:5.2.4.RELEASE -> 5.2.3.RELEASE (*) | | +--- com.fasterxml.jackson.core:jackson-annotations:2.10.2 | | \--- com.fasterxml.jackson.core:jackson-databind:2.10.2 (*) | \--- com.fasterxml.jackson.core:jackson-databind:2.10.2 (*) +--- org.springframework.cloud:spring-cloud-config-server:2.2.2.RELEASE | +--- org.springframework.cloud:spring-cloud-config-client:2.2.2.RELEASE (*) | +--- org.springframework.boot:spring-boot-starter-actuator:2.2.5.RELEASE -> 2.2.4.RELEASE (*) | +--- org.springframework.boot:spring-boot-starter-web:2.2.5.RELEASE -> 2.2.4.RELEASE | | +--- org.springframework.boot:spring-boot-starter:2.2.4.RELEASE (*) | | +--- org.springframework.boot:spring-boot-starter-json:2.2.4.RELEASE (*) | | +--- org.springframework.boot:spring-boot-starter-tomcat:2.2.4.RELEASE | | | +--- jakarta.annotation:jakarta.annotation-api:1.3.5 | | | +--- org.apache.tomcat.embed:tomcat-embed-core:9.0.30 | | | +--- org.apache.tomcat.embed:tomcat-embed-el:9.0.30 | | | \--- org.apache.tomcat.embed:tomcat-embed-websocket:9.0.30 | | | \--- org.apache.tomcat.embed:tomcat-embed-core:9.0.30 | | +--- org.springframework.boot:spring-boot-starter-validation:2.2.4.RELEASE (*) | | +--- org.springframework:spring-web:5.2.3.RELEASE (*) | | \--- org.springframework:spring-webmvc:5.2.3.RELEASE | | +--- org.springframework:spring-aop:5.2.3.RELEASE (*) | | +--- org.springframework:spring-beans:5.2.3.RELEASE (*) | | +--- org.springframework:spring-context:5.2.3.RELEASE (*) | | +--- org.springframework:spring-core:5.2.3.RELEASE (*) | | +--- org.springframework:spring-expression:5.2.3.RELEASE (*) | | \--- org.springframework:spring-web:5.2.3.RELEASE (*) | +--- org.springframework.boot:spring-boot-starter-validation:2.2.5.RELEASE -> 2.2.4.RELEASE (*) | +--- org.springframework.security:spring-security-crypto:5.2.2.RELEASE -> 5.2.1.RELEASE | +--- org.springframework.security:spring-security-rsa:1.0.9.RELEASE (*) | +--- org.eclipse.jgit:org.eclipse.jgit:5.1.3.201810200350-r | | +--- com.jcraft:jsch:0.1.54 | | +--- com.jcraft:jzlib:1.1.1 | | +--- com.googlecode.javaewah:JavaEWAH:1.1.6 | | \--- org.slf4j:slf4j-api:1.7.2 -> 1.7.30 | +--- org.eclipse.jgit:org.eclipse.jgit.http.apache:5.1.3.201810200350-r | | +--- org.eclipse.jgit:org.eclipse.jgit:5.1.3.201810200350-r (*) | | +--- org.apache.httpcomponents:httpclient:4.5.5 -> 4.5.10 | | | +--- org.apache.httpcomponents:httpcore:4.4.12 -> 4.4.13 | | | +--- commons-logging:commons-logging:1.2 | | | \--- commons-codec:commons-codec:1.11 -> 1.13 | | \--- org.apache.httpcomponents:httpcore:4.4.9 -> 4.4.13 | +--- org.yaml:snakeyaml:1.25 | \--- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:2.10.2 | +--- org.yaml:snakeyaml:1.24 -> 1.25 | \--- com.fasterxml.jackson.core:jackson-core:2.10.2 +--- org.springframework.cloud:spring-cloud-starter-openfeign:2.2.2.RELEASE | +--- org.springframework.cloud:spring-cloud-starter:2.2.2.RELEASE (*) | +--- org.springframework.cloud:spring-cloud-openfeign-core:2.2.2.RELEASE | | +--- org.springframework.boot:spring-boot-autoconfigure:2.2.5.RELEASE -> 2.2.4.RELEASE (*) | | +--- org.springframework.cloud:spring-cloud-netflix-ribbon:2.2.2.RELEASE | | | \--- org.springframework.cloud:spring-cloud-netflix-archaius:2.2.2.RELEASE | | +--- org.springframework.boot:spring-boot-starter-aop:2.2.5.RELEASE -> 2.2.4.RELEASE | | | +--- org.springframework.boot:spring-boot-starter:2.2.4.RELEASE (*) | | | +--- org.springframework:spring-aop:5.2.3.RELEASE (*) | | | \--- org.aspectj:aspectjweaver:1.9.5 | | \--- io.github.openfeign.form:feign-form-spring:3.8.0 | | +--- io.github.openfeign.form:feign-form:3.8.0 | | | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30 | | +--- org.springframework:spring-web:5.1.5.RELEASE -> 5.2.3.RELEASE (*) | | +--- commons-fileupload:commons-fileupload:1.4 | | | \--- commons-io:commons-io:2.2 -> 2.5 | | \--- org.slf4j:slf4j-api:1.7.26 -> 1.7.30 | +--- org.springframework:spring-web:5.2.4.RELEASE -> 5.2.3.RELEASE (*) | +--- org.springframework.cloud:spring-cloud-commons:2.2.2.RELEASE (*) | +--- io.github.openfeign:feign-core:10.7.4 | +--- io.github.openfeign:feign-slf4j:10.7.4 | | +--- io.github.openfeign:feign-core:10.7.4 | | \--- org.slf4j:slf4j-api:1.7.13 -> 1.7.30 | \--- io.github.openfeign:feign-hystrix:10.7.4 | +--- io.github.openfeign:feign-core:10.7.4 | +--- com.netflix.archaius:archaius-core:0.7.6 | | +--- com.google.code.findbugs:jsr305:3.0.1 -> 3.0.2 | | +--- commons-configuration:commons-configuration:1.8 | | | +--- commons-lang:commons-lang:2.6 | | | \--- commons-logging:commons-logging:1.1.1 -> 1.2 | | +--- org.slf4j:slf4j-api:1.6.4 -> 1.7.30 | | +--- com.google.guava:guava:16.0 -> 28.2-jre (*) | | +--- com.fasterxml.jackson.core:jackson-annotations:2.4.3 -> 2.10.2 | | +--- com.fasterxml.jackson.core:jackson-core:2.4.3 -> 2.10.2 | | \--- com.fasterxml.jackson.core:jackson-databind:2.4.3 -> 2.10.2 (*) | \--- com.netflix.hystrix:hystrix-core:1.5.18 | +--- org.slf4j:slf4j-api:1.7.0 -> 1.7.30 | +--- com.netflix.archaius:archaius-core:0.4.1 -> 0.7.6 (*) | +--- io.reactivex:rxjava:1.2.0 -> 1.3.8 | \--- org.hdrhistogram:HdrHistogram:2.1.9 -> 2.1.11 +--- org.springframework.data:spring-data-commons:2.2.5.RELEASE (*) +--- io.springfox:springfox-swagger2:3.0.0-SNAPSHOT (*) +--- io.springfox:springfox-spring-webflux:3.0.0-SNAPSHOT (*) +--- io.springfox:springfox-swagger-ui:3.0.0-SNAPSHOT +--- javax.persistence:persistence-api:1.0.2 +--- io.projectreactor:reactor-core:3.3.3.RELEASE (*) +--- com.github.fge:json-patch:1.9 | \--- com.github.fge:jackson-coreutils:1.6 | +--- com.github.fge:msg-simple:1.1 | | +--- com.github.fge:btf:1.2 | | | \--- com.google.code.findbugs:jsr305:2.0.1 -> 3.0.2 | | \--- com.google.code.findbugs:jsr305:2.0.1 -> 3.0.2 | +--- com.fasterxml.jackson.core:jackson-databind:2.2.3 -> 2.10.2 (*) | +--- com.google.guava:guava:16.0.1 -> 28.2-jre (*) | \--- com.google.code.findbugs:jsr305:2.0.1 -> 3.0.2 +--- com.microsoft.azure:azure-cosmosdb-spring-boot-starter:2.2.1 (*) +--- com.microsoft.azure:azure-keyvault-secrets-spring-boot-starter:2.2.1 | +--- org.springframework.boot:spring-boot-starter:2.2.0.RELEASE -> 2.2.4.RELEASE (*) | +--- org.springframework.boot:spring-boot-starter-validation:2.2.0.RELEASE -> 2.2.4.RELEASE (*) | +--- com.microsoft.azure:azure-spring-boot:2.2.1 (*) | +--- com.microsoft.azure:azure-keyvault:1.2.2 (*) | +--- com.microsoft.azure:azure-client-authentication:1.6.13 | | +--- com.microsoft.azure:azure-client-runtime:1.6.13 (*) | | +--- commons-codec:commons-codec:1.11 -> 1.13 | | +--- com.microsoft.azure:adal4j:1.6.4 (*) | | \--- com.microsoft.azure:azure-annotations:1.7.0 | +--- com.microsoft.azure:adal4j:1.6.4 (*) | \--- commons-io:commons-io:2.3 -> 2.5 +--- com.microsoft.azure:azure-cosmos:3.7.2 (*) +--- com.microsoft.azure:applicationinsights-spring-boot-starter:2.5.1 (*) +--- com.squareup.okhttp3:okhttp:3.12.2 (*) +--- com.google.guava:guava:28.2-jre (*) +--- org.jsmpp:jsmpp:2.3.7 | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 +--- org.springframework.boot:spring-boot-starter-cache:2.2.4.RELEASE | +--- org.springframework.boot:spring-boot-starter:2.2.4.RELEASE (*) | \--- org.springframework:spring-context-support:5.2.3.RELEASE | +--- org.springframework:spring-beans:5.2.3.RELEASE (*) | +--- org.springframework:spring-context:5.2.3.RELEASE (*) | \--- org.springframework:spring-core:5.2.3.RELEASE (*) +--- javax.cache:cache-api:1.1.1 \--- org.ehcache:ehcache:3.8.1 +--- org.slf4j:slf4j-api:1.7.25 -> 1.7.30 \--- org.glassfish.jaxb:jaxb-runtime:2.3.1 -> 2.3.2 +--- jakarta.xml.bind:jakarta.xml.bind-api:2.3.2 | \--- jakarta.activation:jakarta.activation-api:1.2.1 +--- org.glassfish.jaxb:txw2:2.3.2 +--- com.sun.istack:istack-commons-runtime:3.0.8 | \--- jakarta.activation:jakarta.activation-api:1.2.1 +--- org.jvnet.staxex:stax-ex:1.8.1 | +--- jakarta.activation:jakarta.activation-api:1.2.1 | \--- jakarta.xml.bind:jakarta.xml.bind-api:2.3.2 (*) +--- com.sun.xml.fastinfoset:FastInfoset:1.2.16 \--- jakarta.activation:jakarta.activation-api:1.2.1 ```

[joshfree]

Thanks for reporting this @allenhumphreys. @kushagraThapar can you please follow up?

/cc @anuchandy

[kushagraThapar]

@David-Noble-at-work - Seems like this error is not getting handled gracefully. Any ideas ?

[David-Noble-at-work]

Not off-hand. I’ll take a closer look.

[timwhit]

It's been 7 days and we are experiencing this issue in production. Do I need to escalate this issue or is anyone looking at it?

[kushagraThapar]

@timwhit If this is a production blocker, for future, feel free to escalate these type of issues. It takes a bit more time on our side to prioritize github issues because of unknown urgency from end user's end. For this one, we will take a look at it today and will provide an update soon.

[kushagraThapar]

@allenhumphreys - As we are investigating the issue, we have some ideas on how to reproduce and test it, but it would be great if you can provide a faithful reproducible code ?

[allenhumphreys]

@kushagraThapar This exception is happening very sporadically in production. I'd love it if I had a repro case but I just don't.

[kushagraThapar]

@allenhumphreys We were able to test it today with our upcoming v4 SDK, and the issue doesn't happen there. SDK re-tries by the number of times configured in the connection policy, and fails if the connection keeps timing out with CosmosClientException.

Would it be possible for you guys to wait for a couple of weeks for v4 SDK. The timeline looks around 1st week of May for GA. Meanwhile, as a workaround, I will suggest increasing the connection timeout for Direct TCP mode. Since this error is happening because of connection timeout, increasing that should help us.

You can increase the timeout by passing this json key value pair in jvm args :

-Dazure.cosmos.directTcp.defaultOptions={"connectionTimeout\":"PT2M"}

[timwhit]

We put in some mitigations on our side already, we'll upgrade to v4 when it's released.

[kushagraThapar]

Sounds good, thanks for the update.

[allenhumphreys]

@kushagraThapar This exception is still occurring in v4.1.0 and we think that it is leading to an unrecoverable SDK state that requires restarting the application to fix. We'll be opening a support ticket as well.

[kushagraThapar]

@allenhumphreys - thanks for the update. @David-Noble-at-work - please take a look at this.

[kushagraThapar]

@allenhumphreys - we have released 4.5.0 - which specifically targets this issue, can you please test the new version on DIRECT mode ?

[FabianMeiswinkel]

Just to be specific - the onErrorDropped events are kind of expected and can be ignored. But the release 4.5.0 will address teh symptoms (unresponsive SDK, need to restart application etc.)

[allenhumphreys]

@kushagraThapar Thank you for letting me know. I will get this out relatively quickly and re-enable direct mode for our lower environments.

@FabianMeiswinkel I would argue against the notion that onErrorDropped should be expected within a project reactor application. Unfortunately, there isn't a lot of specific guidance in Project Reactor's documentation, but the biggest indication to me that such dropped errors should generally be resolved is that the default hook logs an error when this happens and not a warning. Logged exceptions result in alerts for our application, and while I could attempt to ignore only those from the Cosmos SDK, that would be fraught with difficulty.

[FabianMeiswinkel]

Yes - maybe a misunderstanding. It is understood that the occurence of onErrorDropped events indicate violations of the Mono contract (attempt to complete a Mono (exceptionally in this case) that has been completed or canceled already). What I meant with "kind of expected" above is that it is an intentional trade-off - safe-guarding against the race between cancellation (timeout) and exceptional completion (due to error like RequestTimeout) would be expensive (performance wise) - and the only problem with the approach chosen today (first call wins and is signaled to the application layer) is that the onErrorDropped events are logged (functionally the behaviors is as expected). Reactor makes it very easy to decide how to handle these onErrorDropped events (see sample below which would log them as debug instead of error). This work item in the Reactor repo also shows the discussions around which default log level should be used. https://github.com/reactor/reactor-core/issues/1431

Hooks.onErrorDropped(throwable -> { logger.debug("Extra error - on error dropped - operator called : " , throwable); });

[FabianMeiswinkel]

@allenhumphreys - just to be clear - I get your point and will look into whether there are options how we can completely avoid the onErrorDropped events - just wanted to explain why adding the overhead to safe-guard against the race between cancel and completeExceptionally is not an option.

[allenhumphreys]

Thanks @FabianMeiswinkel , that's a fair point. Maybe a note can be added somewhere explaining that, perhaps the README or other SDK integration documentation?

[FabianMeiswinkel]

@allenhumphreys - FYI - I have created a PR to address the noisy error logs here: https://github.com/Azure/azure-sdk-for-java/pull/15337 - I am basically applying a custom hook with just debug logging to the Mono created in our transport. For the scope of this Mono we know that it is safe to ignore the onErrorDropped events - so that seems to be a reasonable way to avoid confusion and noise-level in the log without other side effects.

[kushagraThapar]

This change will be released in upcoming release, closing this issue.