search

Azure / azure-sdk-for-java

This repository is for active development of the Azure SDK for Java. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/java/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-java.
MIT License
2.35k stars 1.99k forks source link

[QUERY] CosmosDB library not able to process any request, throwing TransportException #41773

Closed gataricd closed 1 week ago

gataricd commented 1 month ago

Query/Question We experienced the situation that an application in production suddenly was not able to process any request to the CosmosDB with the following exception:

com.azure.cosmos.implementation.directconnectivity.TransportException: RntbdTransportClient({"id":1,"isClosed":true,"configuration":{"wireLogLevel":null,"tcpKeepIdle":1,"tcpKeepIntvl":1,"timeoutDetectionHighFrequencyTimeLimitInNanos":10000000000,"nonRespondingChannelReadDelayTimeLimitInNanos":10000000000,"timeoutDetectionHighFrequencyThreshold":3,"cancellationCountSinceLastReadThreshold":3,"timeoutDetectionOnWriteThreshold":1,"maxRequestsPerChannel":30,"maxChannelsPerEndpoint":130,"sslHandshakeTimeoutInMillis":5000,"shutdownTimeoutInNanos":15000000000,"timeoutDetectionDisableCPUThreshold":90.0,"maxConcurrentRequestsPerEndpoint":10000,"tcpNetworkRequestTimeoutInNanos":5000000000,"preferTcpNative":true,"bufferPageSize":8192,"requestTimerResolutionInNanos":100000000,"connectionAcquisitionTimeoutInNanos":5000000000,"idleConnectionTimerResolutionInNanos":100000000,"receiveHangDetectionTimeInNanos":65000000000,"timeoutDetectionTimeLimitInNanos":60000000000,"connectTimeoutInMillis":5000,"idleConnectionTimeoutInNanos":0,"idleEndpointTimeoutInNanos":3600000000000,"sendHangDetectionTimeInNanos":10000000000,"timeoutDetectionOnWriteTimeLimitInNanos":6000000000,"channelAcquisitionContextLatencyThresholdInMillis":1000,"minConnectionPoolSizePerEndpoint":1,"connectionEndpointRediscoveryEnabled":true,"timeoutDetectionEnabled":true,"userAgent":{"suffix":"","userAgent":"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3"},"maxBufferCapacity":8388608},"serviceEndpoints":{"count":0,"items":[]}}) is closed

This happened suddenly during early morning hours when it doesn't get much traffic. It kept retrying, but none of the requests succeeded until we restarted the app. We experienced one million such exception, so the app was completely blocked. After investigating it, we concluded that the CosmosDB normally, but the problem was with the CosmosDB library that we use. We never experienced such problem before and it would be useful to get some insight of what could cause this.

CosmosDB Setup

CosmosClientBuilder()
                .endpoint(cosmosEndpointUri)
                .credential(DefaultAzureCredentialBuilder().build())
                .consistencyLevel(ConsistencyLevel.EVENTUAL)
                .contentResponseOnWriteEnabled(true)
                .buildAsyncClient()

Why is this not a Bug or a feature Request? We are not sure if it is a bug in the library, or it is something else.

Setup (please complete the following information if applicable):

github-actions[bot] commented 1 month ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @kushagraThapar @pjohari-ms @TheovanKraay.

kushagraThapar commented 1 month ago

@gataricd from the one liner provided above, it looks like the RntbdTransportClient is closed and once closed, it won't be able to process any requests further, unless the CosmosClient is restarted. I am curious, do you have any logic on closing the CosmosClient (which will internally close the RntbdTransportClient). Can you provide any other stack trace?

gataricd commented 1 month ago

@kushagraThapar there is no logic for closing the CosmosClient, we never do it. I will check if I can find any other stack trace.

gataricd commented 1 month ago

There are more details here:

ThresholdViolation - CTX: {"spanName":"queryItems.tenantMappings","account":"cosmosdb","db":"tenantMapping","container":"tenantMappings","resource":"Document","operation":"Query","operationId":"queryItems.tenantMappings","sequenceNumber":1,"consistency":"Eventual","status":200,"RUs":0.0,"maxRequestSizeInBytes":109,"maxResponseSizeInBytes":421,"maxItems":100,"diagnostics":[{"cosmosDiagnostics":{"userAgent":"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3","activityId":"c9288e6c-e93d-4026-892a-bdd622701e32","requestLatencyInMs":1,"requestStartTimeUTC":"2024-09-09T13:01:48.123762495Z","requestEndTimeUTC":"2024-09-09T13:01:48.125596315Z","responseStatisticsList":[],"supplementalResponseStatisticsList":[],"addressResolutionStatistics":{},"regionsContacted":["west europe"],"retryContext":{"statusAndSubStatusCodes":null,"retryLatency":0,"retryCount":0},"metadataDiagnosticsContext":{"metadataDiagnosticList":null},"serializationDiagnosticsContext":{"serializationDiagnosticsList":null},"gatewayStatisticsList":[{"sessionToken":null,"operationType":"QueryPlan","resourceType":"Document","statusCode":200,"subStatusCode":0,"requestCharge":0.0,"requestTimeline":[{"eventName":"connectionAcquired","startTimeUTC":"2024-09-09T13:01:48.123970083Z","durationInMilliSecs":0.195579},{"eventName":"connectionConfigured","startTimeUTC":"2024-09-09T13:01:48.124165662Z","durationInMilliSecs":0.024415},{"eventName":"requestSent","startTimeUTC":"2024-09-09T13:01:48.124190077Z","durationInMilliSecs":0.373137},{"eventName":"transitTime","startTimeUTC":"2024-09-09T13:01:48.124563214Z","durationInMilliSecs":0.842204},{"eventName":"received","startTimeUTC":"2024-09-09T13:01:48.125405418Z","durationInMilliSecs":0.111238}],"partitionKeyRangeId":null,"responsePayloadSizeInBytes":421,"locationToLocationSpecificHealthContext":{"v":null}}],"samplingRateSnapshot":1.0,"bloomFilterInsertionCountSnapshot":0,"systemInformation":{"usedMemory":"420005 KB","availableMemory":"69467 KB","systemCpuLoad":"empty","availableProcessors":4},"clientCfgs":{"id":1,"machineId":"vmId_fd7ad877-2276-4628-be98-163b2be09b8a","connectionMode":"DIRECT","numberOfClients":0,"excrgns":"[]","clientEndpoints":{"https://cosmosdb.documents.azure.com:443/":1},"connCfg":{"rntbd":"(cto:PT5S, nrto:PT5S, icto:PT0S, ieto:PT1H, mcpe:130, mrpc:30, cer:true)","gw":"(cps:1000, nrto:PT1M, icto:PT1M, p:false)","other":"(ed: true, cs: false, rv: true)"},"consistencyCfg":"(consistency: Eventual, mm: true, prgns: [])","proactiveInitCfg":"","e2ePolicyCfg":"","sessionRetryCfg":"","partitionLevelCircuitBreakerCfg":"(cb: false, type: CONSECUTIVE_EXCEPTION_COUNT_BASED, rexcntt: 10, wexcntt: 5)"}}},{"cosmosDiagnostics":{"userAgent":"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3","activityId":"def9f373-320a-43e8-97bf-16333f17a23b","requestLatencyInMs":7044,"requestStartTimeUTC":"2024-09-09T13:01:48.126148046Z","requestEndTimeUTC":"2024-09-09T13:01:55.170849322Z","responseStatisticsList":[{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14091/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133676055553723771p/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:48.127009766Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14177/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133676055553723772s/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:48.127450861Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14099/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133692711729103238s/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:48.127897676Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14161/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133676055524504443s/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:48.128306490Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14099/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133692711729103238s/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:48.134210709Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14161/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133676055524504443s/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:48.134461668Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14177/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133676055553723772s/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:48.134729739Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14091/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133676055553723771p/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:48.134976400Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14161/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133676055524504443s/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:49.145128666Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14099/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133692711729103238s/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:49.145338739Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query"
gataricd commented 1 month ago

There is also, but it was cut since it is too big:

,"requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14177/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133676055553723772s/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:49.145538602Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14091/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133676055553723771p/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:49.145735460Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14091/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133676055553723771p/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:51.156661364Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14099/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133692711729103238s/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:51.156857310Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14161/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133676055524504443s/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:51.157064408Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14177/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133676055553723772s/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:51.157265754Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14099/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133692711729103238s/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:55.170237229Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14091/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133676055553723771p/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:55.170441049Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14177/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133676055553723772s/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:55.170649639Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}},{"storeResult":{"storePhysicalAddress":"rntbd://cdb-ms-prod-westeurope1-be184.documents.azure.com:14161/apps/7b80ee00-3a09-4275-a8ef-db43edddb36c/services/3989490f-289d-48bd-ac2b-8776223b40c5/partitions/75d0da40-13b8-4899-8366-bcd1ca043372/replicas/133676055524504443s/","lsn":-1,"quorumAckedLSN":-1,"currentReplicaSetSize":0,"globalCommittedLsn":-1,"partitionKeyRangeId":"0","isValid":false,"statusCode":500,"subStatusCode":20910,"isGone":false,"isNotFound":false,"isInvalidPartition":false,"isThroughputControlRequestRateTooLarge":false,"requestCharge":0.0,"itemLSN":-1,"sessionToken":null,"backendLatencyInMs":null,"retryAfterInMs":null,"exceptionMessage":"Unexpected exception RntbdTransportClient({\"id\":1,\"isClosed\":true,\"configuration\":{\"wireLogLevel\":null,\"maxRequestsPerChannel\":30,\"maxChannelsPerEndpoint\":130,\"maxConcurrentRequestsPerEndpoint\":10000,\"timeoutDetectionOnWriteThreshold\":1,\"timeoutDetectionDisableCPUThreshold\":90.0,\"sslHandshakeTimeoutInMillis\":5000,\"shutdownTimeoutInNanos\":15000000000,\"connectTimeoutInMillis\":5000,\"idleConnectionTimeoutInNanos\":0,\"idleEndpointTimeoutInNanos\":3600000000000,\"sendHangDetectionTimeInNanos\":10000000000,\"connectionAcquisitionTimeoutInNanos\":5000000000,\"idleConnectionTimerResolutionInNanos\":100000000,\"receiveHangDetectionTimeInNanos\":65000000000,\"timeoutDetectionTimeLimitInNanos\":60000000000,\"tcpNetworkRequestTimeoutInNanos\":5000000000,\"requestTimerResolutionInNanos\":100000000,\"timeoutDetectionOnWriteTimeLimitInNanos\":6000000000,\"timeoutDetectionHighFrequencyTimeLimitInNanos\":10000000000,\"nonRespondingChannelReadDelayTimeLimitInNanos\":10000000000,\"tcpKeepIdle\":1,\"tcpKeepIntvl\":1,\"cancellationCountSinceLastReadThreshold\":3,\"timeoutDetectionHighFrequencyThreshold\":3,\"preferTcpNative\":true,\"bufferPageSize\":8192,\"minConnectionPoolSizePerEndpoint\":1,\"connectionEndpointRediscoveryEnabled\":true,\"timeoutDetectionEnabled\":true,\"channelAcquisitionContextLatencyThresholdInMillis\":1000,\"userAgent\":{\"suffix\":\"\",\"userAgent\":\"azsdk-java-cosmos/4.63.0 Linux/5.15.0-1071-azure JRE/21.0.3\"},\"maxBufferCapacity\":8388608},\"serviceEndpoints\":{\"count\":0,\"items\":[]}}) is closed received while reading from store. - Unknown server error occurred when processing this request. If the issue persists, please contact Azure Support: http://aka.ms/azure-support","exceptionResponseHeaders":"{x-ms-substatus=20910}","replicaStatusList":{},"transportRequestTimeline":null,"rntbdRequestLengthInBytes":0,"rntbdResponseLengthInBytes":0,"requestPayloadLengthInBytes":0,"responsePayloadLengthInBytes":0,"channelStatistics":null,"serviceEndpointStatistics":null},"requestResponseTimeUTC":"2024-09-09T13:01:55.170849322Z","requestStartTimeUTC":null,"requestResourceType":"Document","requestOperationType":"Query","requestSessionToken":null,"e2ePolicyCfg":null,"excludedRegions":null,"sessionTokenEvaluationResults":[],"locationToLocationSpecificHealthContext":{"v":null}}],"supplementalResponseStatisticsList":[],"addressResolutionStatistics":{},"regionsContacted":["west europe"],"retryContext":{"statusAndSubStatusCodes":[[410,21009],[410,21009],[410,21009],[410,21009],[410,21009]],"retryLatency":7042,"retryCount":5},"metadataDiagnosticsContext":{"metadataDiagnosticList":null},"serializationDiagnosticsContext":{"serializationDiagnosticsList":null},"gatewayStatisticsList":[],"samplingRateSnapshot":1.0,"bloomFilterInsertionCountSnapshot":0,"systemInformation":{"usedMemory":"420031 KB","availableMemory":"69441 KB","systemCpuLoad":"empty","availableProcessors":4},"clientCfgs":{"id":1,"machineId":"vmId_fd7ad877-2276-4628-be98-163b2be09b8a","connectionMode":"DIRECT","numberOfClients":0,"excrgns":"[]","clientEndpoints":{"https://cosmosdb.documents.azure.com:443/":1},"connCfg":{"rntbd":"(cto:PT5S, nrto:PT5S, icto:PT0S, ieto:PT1H, mcpe:130, mrpc:30, cer:true)","gw":"(cps:1000, nrto:PT1M, icto:PT1M, p:false)","other":"(ed: true, cs: false, rv: true)"},"consistencyCfg":"(consistency: Eventual, mm: true, prgns: [])","proactiveInitCfg":"","e2ePolicyCfg":"","sessionRetryCfg":"","partitionLevelCircuitBreakerCfg":"(cb: false, type: CONSECUTIVE_EXCEPTION_COUNT_BASED, rexcntt: 10, wexcntt: 5)"}}}]}
kushagraThapar commented 1 month ago

@gataricd thanks for providing these, I investigated our backend metrics and couldn't find anything out of order on these, it could be possible this was a transient issue. Is this reproducible on your end? Is it still happening?

gataricd commented 1 month ago

@kushagraThapar thanks for checking it. It is not reproducible, and it is not happening more. Do you have any advice how to recover from such a case, except restarting the application? Because if this keep happening, and we can't recover, we can't use CosmosDB.

kushagraThapar commented 1 month ago

@gataricd to be honest, I will suggest opening a support case in these scenarios and let the support and backend service team look into this more deeply the next time it happens. That way, we will be able to find the root cause for it in the real time. From the SDK point of view, this behavior is sort of expected, because the SDK kept retrying and then finally exhausted all the retries. Maybe we can have better logic in the SDK to avoid RntbdTransportClient being closing. @xinlian12 thoughts on this?

gataricd commented 1 week ago

The problem was that Tomcat tried to shutdown and somehow it closed the RntbdTransportClient, but it didn't affect anything else and it didn't shut down.