search

Azure / azure-cosmosdb-java

Java Async SDK for SQL API of Azure Cosmos DB
MIT License
54 stars 61 forks source link

com.azure.data.cosmos.GoneException when attempting to read document from CosmosDB collection (azure-cosmos:3.4.0) #309

Closed ischoch closed 4 years ago

ischoch commented 4 years ago

Describe the bug When connecting to CosmosDB via the com.microsoft.azure:azure-cosmos:3.4.0 package, I receive an com.azure.data.cosmos.GoneException when I attempt to read a document from a collection. I have verified that the collection exists using the azure portal, and the request is in fact successful with the 3.4 api when running this from my local machine. However, when I use the 3.4 api to connect from a machine with a firewall, I get a stack trace (displayed below). I have whitelisted "https://k2-pcx-sql.documents.azure.com/443" in my firewall as well as "https://k2-pcx-sql.documents.azure.com". Based on the stack trace, it cannot connect to Connection refused: cdb-ms-prod-westus2-fd16.documents.azure.com/40.78.250.1:14011. Is it a new requirement that we need our firewall to whitelist this fault domain "cdb-ms-prod-westus2-fd16.documents.azure.com"? Previously we did not whitelist it for the previous azure SDK. If so, I am concerned about maintaining this firewall whitelist in case the fault domain changes.

I am using the Direct connection policy and consistency level session

This is the stack trace:

Caused by: com.azure.data.cosmos.CosmosClientException: null, RequestStartTime: "24 Jan 2020 22:40:44.759", RequestEndTime: "24 Jan 2020 22:41:15.880", Duration: 31121 ms, NUMBER of regions attempted: 1 StoreResponseStatistics{requestResponseTime="24 Jan 2020 22:40:46.059", storeResult=storePhysicalAddress: null, lsn: -1, globalCommittedLsn: -1, partitionKeyRangeId: null, isValid: false, statusCode: 410, subStatusCode: 0, isGone: true, isNotFound: false, isInvalidPartition: false, requestCharge: 0.0, itemLSN: -1, sessionToken: null, exception: failed to establish connection to cdb-ms-prod-westus2-fd16.documents.azure.com:14011: Connection refused: cdb-ms-prod-westus2-fd16.documents.azure.com/40.78.250.1:14011, requestResourceType=Document, requestOperationType=Read} StoreResponseStatistics{requestResponseTime="24 Jan 2020 22:40:46.071", storeResult=storePhysicalAddress: null, lsn: -1, globalCommittedLsn: -1, partitionKeyRangeId: null, isValid: false, statusCode: 410, subStatusCode: 0, isGone: true, isNotFound: false, isInvalidPartition: false, requestCharge: 0.0, itemLSN: -1, sessionToken: null, exception: failed to establish connection to cdb-ms-prod-westus2-fd16.documents.azure.com:14166: Connection refused: cdb-ms-prod-westus2-fd16.documents.azure.com/40.78.250.1:14166, requestResourceType=Document, requestOperationType=Read} StoreResponseStatistics{requestResponseTime="24 Jan 2020 22:40:46.081", storeResult=storePhysicalAddress: null, lsn: -1, globalCommittedLsn: -1, partitionKeyRangeId: null, isValid: false, statusCode: 410, subStatusCode: 0, isGone: true, isNotFound: false, isInvalidPartition: false, requestCharge: 0.0, itemLSN: -1, sessionToken: null, exception: failed to establish connection to cdb-ms-prod-westus2-fd16.documents.azure.com:14490: Connection refused: cdb-ms-prod-westus2-fd16.documents.azure.com/40.78.250.1:14490, requestResourceType=Document, requestOperationType=Read} StoreResponseStatistics{requestResponseTime="24 Jan 2020 22:40:46.071", storeResult=storePhysicalAddress: null, lsn: -1, globalCommittedLsn: -1, partitionKeyRangeId: null, isValid: false, statusCode: 410, subStatusCode: 0, isGone: true, isNotFound: false, isInvalidPartition: false, requestCharge: 0.0, itemLSN: -1, sessionToken: null, exception: failed to establish connection to cdb-ms-prod-westus2-fd16.documents.azure.com:14166: Connection refused: cdb-ms-prod-westus2-fd16.documents.azure.com/40.78.250.1:14166, requestResourceType=Document, requestOperationType=Read} StoreResponseStatistics{requestResponseTime="24 Jan 2020 22:40:46.081", storeResult=storePhysicalAddress: null, lsn: -1, globalCommittedLsn: -1, partitionKeyRangeId: null, isValid: false, statusCode: 410, subStatusCode: 0, isGone: true, isNotFound: false, isInvalidPartition: false, requestCharge: 0.0, itemLSN: -1, sessionToken: null, exception: failed to establish connection to cdb-ms-prod-westus2-fd16.documents.azure.com:14490: Connection refused: cdb-ms-prod-westus2-fd16.documents.azure.com/40.78.250.1:14490, requestResourceType=Document, requestOperationType=Read} StoreResponseStatistics{requestResponseTime="24 Jan 2020 22:40:46.091", storeResult=storePhysicalAddress: null, lsn: -1, globalCommittedLsn: -1, partitionKeyRangeId: null, isValid: false, statusCode: 410, subStatusCode: 0, isGone: true, isNotFound: false, isInvalidPartition: false, requestCharge: 0.0, itemLSN: -1, sessionToken: null, exception: failed to establish connection to cdb-ms-prod-westus2-fd16.documents.azure.com:14205: Connection refused: cdb-ms-prod-westus2-fd16.documents.azure.com/40.78.250.1:14205, requestResourceType=Document, requestOperationType=Read} StoreResponseStatistics{requestResponseTime="24 Jan 2020 22:40:46.415", storeResult=storePhysicalAddress: null, lsn: -1, globalCommittedLsn: -1, partitionKeyRangeId: null, isValid: false, statusCode: 410, subStatusCode: 0, isGone: true, isNotFound: false, isInvalidPartition: false, requestCharge: 0.0, itemLSN: -1, sessionToken: null, exception: failed to establish connection to cdb-ms-prod-westus2-fd16.documents.azure.com:14166: Connection refused: cdb-ms-prod-westus2-fd16.documents.azure.com/40.78.250.1:14166, requestResourceType=Document, requestOperationType=Read} StoreResponseStatistics{requestResponseTime="24 Jan 2020 22:40:46.421", storeResult=storePhysicalAddress: null, lsn: -1, globalCommittedLsn: -1, partitionKeyRangeId: null, isValid: false, statusCode: 410, subStatusCode: 0, isGone: true, isNotFound: false, isInvalidPartition: false, requestCharge: 0.0, itemLSN: -1, sessionToken: null, exception: failed to establish connection to cdb-ms-prod-westus2-fd16.documents.azure.com:14011: Connection refused: cdb-ms-prod-westus2-fd16.documents.azure.com/40.78.250.1:14011, requestResourceType=Document, requestOperationType=Read} StoreResponseStatistics{requestResponseTime="24 Jan 2020 22:40:46.427", storeResult=storePhysicalAddress: null, lsn: -1, globalCommittedLsn: -1, partitionKeyRangeId: null, isValid: false, statusCode: 410, subStatusCode: 0, isGone: true, isNotFound: false, isInvalidPartition: false, requestCharge: 0.0, itemLSN: -1, sessionToken: null, exception: failed to establish connection to cdb-ms-prod-westus2-fd16.documents.azure.com:14490: Connection refused: cdb-ms-prod-westus2-fd16.documents.azure.com/40.78.250.1:14490, requestResourceType=Document, requestOperationType=Read} StoreResponseStatistics{requestResponseTime="24 Jan 2020 22:40:46.433", storeResult=storePhysicalAddress: null, lsn: -1, globalCommittedLsn: -1, partitionKeyRangeId: null, isValid: false, statusCode: 410, subStatusCode: 0, isGone: true, isNotFound: false, isInvalidPartition: false, requestCharge: 0.0, itemLSN: -1, sessionToken: null, exception: failed to establish connection to cdb-ms-prod-westus2-fd16.documents.azure.com:14205: Connection refused: cdb-ms-prod-westus2-fd16.documents.azure.com/40.78.250.1:14205, requestResourceType=Document, requestOperationType=Read} StoreResponseStatistics{requestResponseTime="24 Jan 2020 22:40:47.717", storeResult=storePhysicalAddress: null, lsn: -1, globalCommittedLsn: -1, partitionKeyRangeId: null, isValid: false, statusCode: 410, subStatusCode: 0, isGone: true, isNotFound: false, isInvalidPartition: false, requestCharge: 0.0, itemLSN: -1, sessionToken: null, exception: failed to establish connection to cdb-ms-prod-westus2-fd16.documents.azure.com:14166: Connection refused: cdb-ms-prod-westus2-fd16.documents.azure.com/40.78.250.1:14166, requestResourceType=Document, requestOperationType=Read} StoreResponseStatistics{requestResponseTime="24 Jan 2020 22:40:47.724", storeResult=storePhysicalAddress: null, lsn: -1, globalCommittedLsn: -1, partitionKeyRangeId: null, isValid: false, statusCode: 410, subStatusCode: 0, isGone: true, isNotFound: false, isInvalidPartition: false, requestCharge: 0.0, itemLSN: -1, sessionToken: null, exception: failed to establish connection to cdb-ms-prod-westus2-fd16.documents.azure.com:14011: Connection refused: cdb-ms-prod-westus2-fd16.documents.azure.com/40.78.250.1:14011, requestResourceType=Document, requestOperationType=Read} StoreResponseStatistics{requestResponseTime="24 Jan 2020 22:40:47.731", storeResult=storePhysicalAddress: null, lsn: -1, globalCommittedLsn: -1, partitionKeyRangeId: null, isValid: false, statusCode: 410, subStatusCode: 0, isGone: true, isNotFound: false, isInvalidPartition: false, requestCharge: 0.0, itemLSN: -1, sessionToken: null, exception: failed to establish connection to cdb-ms-prod-westus2-fd16.documents.azure.com:14205: Connection refused: cdb-ms-prod-westus2-fd16.documents.azure.com/40.78.250.1:14205, requestResourceType=Document, requestOperationType=Read} . . . [logs skipped] AddressResolutionStatistics{startTime="24 Jan 2020 22:40:45.565", endTime="24 Jan 2020 22:40:45.831", targetEndpoint='https://k2-pcx-sql.documents.azure.com:443/addresses/?$resolveFor=dbs%2F37lwAA%3D%3D%2Fcolls%2F37lwAOSS4fw%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0'} AddressResolutionStatistics{startTime="24 Jan 2020 22:40:49.741", endTime="24 Jan 2020 22:40:49.991", targetEndpoint='https://k2-pcx-sql.documents.azure.com:443/addresses/?$resolveFor=dbs%2F37lwAA%3D%3D%2Fcolls%2F37lwAOSS4fw%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0'} AddressResolutionStatistics{startTime="24 Jan 2020 22:40:54.016", endTime="24 Jan 2020 22:40:54.299", targetEndpoint='https://k2-pcx-sql.documents.azure.com:443/addresses/?$resolveFor=dbs%2F37lwAA%3D%3D%2Fcolls%2F37lwAOSS4fw%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0'} AddressResolutionStatistics{startTime="24 Jan 2020 22:40:46.110", endTime="24 Jan 2020 22:40:46.405", targetEndpoint='https://k2-pcx-sql.documents.azure.com:443/addresses/?$resolveFor=dbs%2F37lwAA%3D%3D%2Fcolls%2F37lwAOSS4fw%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0'} AddressResolutionStatistics{startTime="24 Jan 2020 22:41:02.323", endTime="24 Jan 2020 22:41:02.581", targetEndpoint='https://k2-pcx-sql.documents.azure.com:443/addresses/?$resolveFor=dbs%2F37lwAA%3D%3D%2Fcolls%2F37lwAOSS4fw%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0'} AddressResolutionStatistics{startTime="24 Jan 2020 22:40:47.435", endTime="24 Jan 2020 22:40:47.708", targetEndpoint='https://k2-pcx-sql.documents.azure.com:443/addresses/?$resolveFor=dbs%2F37lwAA%3D%3D%2Fcolls%2F37lwAOSS4fw%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0'} AddressResolutionStatistics{startTime="24 Jan 2020 22:41:15.604", endTime="24 Jan 2020 22:41:15.867", targetEndpoint='https://k2-pcx-sql.documents.azure.com:443/addresses/?$resolveFor=dbs%2F37lwAA%3D%3D%2Fcolls%2F37lwAOSS4fw%3D%2Fdocs&$filter=protocol%20eq%20rntbd&$partitionKeyRangeIds=0'}

    at com.azure.data.cosmos.BridgeInternal.createCosmosClientException(BridgeInternal.java:346) ~[azure-cosmos-3.4.0.jar:?]
    at com.azure.data.cosmos.internal.directconnectivity.GoneAndRetryWithRetryPolicy.shouldRetry(GoneAndRetryWithRetryPolicy.java:88) ~[azure-cosmos-3.4.0.jar:?]
    at com.azure.data.cosmos.internal.RetryUtils.lambda$toRetryWithAlternateFunc$4(RetryUtils.java:60) ~[azure-cosmos-3.4.0.jar:?]
    at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:88) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE]
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:165) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE]
    at reactor.core.publisher.MonoSingle$SingleSubscriber.onError(MonoSingle.java:141) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE]
    at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onError(FluxPeekFuseable.java:227) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE]
    at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:165) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE]

. . . [logs skipped] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-all-4.1.41.Final.jar:4.1.41.Final] ... 1 more Caused by: com.azure.data.cosmos.GoneException: The requested resource is no longer available at the server. at com.azure.data.cosmos.internal.directconnectivity.StoreReader.createReadReplicaResult(StoreReader.java:307) ~[azure-cosmos-3.4.0.jar:?] at com.azure.data.cosmos.internal.directconnectivity.StoreReader.lambda$readMultipleReplicasInternalAsync$12(StoreReader.java:423) ~[azure-cosmos-3.4.0.jar:?] at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:46) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.Flux.subscribe(Flux.java:8134) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onComplete(FluxConcatArray.java:207) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.MonoIgnoreElements$IgnoreElementsSubscriber.onComplete(MonoIgnoreElements.java:81) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxTakeUntil$TakeUntilPredicateSubscriber.onComplete(FluxTakeUntil.java:114) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxTakeUntil$TakeUntilPredicateSubscriber.onNext(FluxTakeUntil.java:92) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxRepeatPredicate$RepeatPredicateSubscriber.onNext(FluxRepeatPredicate.java:79) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1592) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:121) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxFlatMap$FlatMapMain.checkTerminated(FluxFlatMap.java:823) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxFlatMap$FlatMapMain.drainLoop(FluxFlatMap.java:589) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxFlatMap$FlatMapMain.innerComplete(FluxFlatMap.java:892) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxFlatMap$FlatMapInner.onComplete(FluxFlatMap.java:986) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onComplete(Operators.java:1820) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxJust$WeakScalarSubscription.request(FluxJust.java:101) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:1956) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:68) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxJust.subscribe(FluxJust.java:70) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.Flux.subscribe(Flux.java:8134) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:97) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxFlatMap$FlatMapMain.checkTerminated(FluxFlatMap.java:819) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxFlatMap$FlatMapMain.drainLoop(FluxFlatMap.java:589) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxFlatMap$FlatMapMain.drain(FluxFlatMap.java:569) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxFlatMap$FlatMapMain.onError(FluxFlatMap.java:441) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onError(FluxDoFinally.java:129) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE] at reactor.core.publisher.MonoCompletionStage.lambda$subscribe$0(MonoCompletionStage.java:83) ~[reactor-core-3.3.0.RELEASE.jar:3.3.0.RELEASE]

Environment summary SDK Version: com.microsoft.azure:azure-cosmos:3.4.0 Java JDK version: 1.8.0_121 OS Version: linux

kushagraThapar commented 4 years ago

@ischoch v3 SDK is not in this repo, rather this one: https://github.com/Azure/azure-sdk-for-java/tree/master/sdk/cosmos Can you please move this issue here: https://github.com/Azure/azure-sdk-for-java/issues and the issue will automatically be assigned to someone from our team using Microsoft bot. You can close the issue here since it is related to v3 SDK.

ischoch commented 4 years ago

Thanks, moved