search

Azure / azure-cosmosdb-java

Java Async SDK for SQL API of Azure Cosmos DB
MIT License
54 stars 61 forks source link

AsyncDocumentClient created using withPermissionFeed is unable to access Database Account #78

Closed marc-sensenich closed 5 years ago

marc-sensenich commented 5 years ago

Describe the issue When building an AsyncDocumentClient using .withPermissionFeed the application emits error logs regarding failures for reading the Database Account

To Reproduce Create an AsyncDocumentClient using a code block similar to

String permissionJSON = "{  \n" +
        "    \"id\": \"a_permission\",  \n" +
        "    \"permissionMode\": \"Read\",  \n" +
        "    \"resource\": \"dbs/volcanodb/colls/volcano1\",  \n" +
        "    \"_rid\": \"Sl8fAG8cXgBn6Ju2GqNsAA==\",  \n" +
        "    \"_ts\": 1449604760,  \n" +
        "    \"_self\": \"dbs\\/Sl8fAA==\\/users\\/Sl8fAG8cXgA=\\/permissions\\/Sl8fAG8cXgBn6Ju2GqNsAA==\\/\",  \n" +
        "    \"_etag\": \"\\\"00000e00-0000-0000-0000-566736980000\\\"\",  \n" +
        "    \"_token\": \"type=resource&ver=1&sig=ocPyc9QQFybITu1EqzX0kg==;w+WR1aWafB3+yZq5JSoBwgz78XDlU+k9Xiqvc+Q7TlAl1P4h4t721Cn5cjhZ9h3TSd2\\/MJLy+wG+YkhDL9UlGkVv05RZGy2fMaLGdeQkWc7TShkc\\/M2boPc3GXq2yiERKl5CN4AZWSOcrFhOFuuTOqF4ZdBlflmNudaakodr\\/8qTip0i+a7moz1Jkc5+9iLAsDFyqTR1sirp7kAVNFbiqPdYTjNkvZUHF3nYYmRskOg=;\"  \n" +
        "}  ";
String cosmosDBURL = "[your host]";
Permission thePermission = new Permission(permissionJSON);

AsyncDocumentClient limitedDocumentClient = new AsyncDocumentClient.Builder()
        .withConnectionPolicy(ConnectionPolicy.GetDefault())
        .withConsistencyLevel(ConsistencyLevel.Strong)
        .withServiceEndpoint(cosmosDBURL)
        .withPermissionFeed(new ArrayList<>(Arrays.asList(thePermission)))
        .build();

Expected behavior

The Database Account is read and the LocationCache is updated.

Actual behavior The Database Account can not be read and the error logs are emitted due to an error parsing the ResourceId provided, which is an empty string.

2019-01-11 15:05:57,119       [RxComputationScheduler-1] INFO  com.microsoft.azure.cosmosdb.rx.internal.RxDocumentClientImpl - Getting database account endpoint from https://testing.documents.azure.com:443/
2019-01-11 15:05:57,139       [RxComputationScheduler-1] ERROR com.microsoft.azure.cosmosdb.rx.internal.GlobalEndpointManager - Fail to reach global gateway [https://testing.documents.azure.com:443/], [Invalid resource id ]
2019-01-11 15:05:57,140       [RxComputationScheduler-1] ERROR com.microsoft.azure.cosmosdb.rx.internal.GlobalEndpointManager - startRefreshLocationTimerAsync() - Unable to refresh database account from any location. Exception: java.lang.IllegalArgumentException: Invalid resource id 
java.lang.IllegalArgumentException: Invalid resource id 
    at com.microsoft.azure.cosmosdb.internal.ResourceId.parse(ResourceId.java:75)
    at com.microsoft.azure.cosmosdb.internal.ResourceTokenAuthorizationHelper.getAuthorizationTokenUsingResourceTokens(ResourceTokenAuthorizationHelper.java:161)
    at com.microsoft.azure.cosmosdb.rx.internal.RxDocumentClientImpl.getUserAuthorizationToken(RxDocumentClientImpl.java:1057)
    at com.microsoft.azure.cosmosdb.rx.internal.RxDocumentClientImpl.populateHeaders(RxDocumentClientImpl.java:1022)
    at com.microsoft.azure.cosmosdb.rx.internal.RxDocumentClientImpl.lambda$getDatabaseAccountFromEndpoint$144(RxDocumentClientImpl.java:3042)
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46)
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
    at rx.Observable.unsafeSubscribe(Observable.java:10256)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10256)
    at rx.internal.operators.OnSubscribeSingle.call(OnSubscribeSingle.java:81)
    at rx.internal.operators.OnSubscribeSingle.call(OnSubscribeSingle.java:27)
    at rx.Single.subscribe(Single.java:1979)
    at rx.internal.operators.SingleOperatorOnErrorResumeNext.call(SingleOperatorOnErrorResumeNext.java:77)
    at rx.internal.operators.SingleOperatorOnErrorResumeNext.call(SingleOperatorOnErrorResumeNext.java:23)
    at rx.internal.operators.SingleToObservable.call(SingleToObservable.java:39)
    at rx.internal.operators.SingleToObservable.call(SingleToObservable.java:27)
    at rx.Observable.unsafeSubscribe(Observable.java:10256)
    at rx.internal.operators.OnSubscribeSingle.call(OnSubscribeSingle.java:81)
    at rx.internal.operators.OnSubscribeSingle.call(OnSubscribeSingle.java:27)
    at rx.Single.subscribe(Single.java:1979)
    at rx.internal.operators.CompletableFlatMapSingleToCompletable.call(CompletableFlatMapSingleToCompletable.java:43)
    at rx.internal.operators.CompletableFlatMapSingleToCompletable.call(CompletableFlatMapSingleToCompletable.java:28)
    at rx.Completable.unsafeSubscribe(Completable.java:2035)
    at rx.Completable.subscribe(Completable.java:2056)
    at rx.internal.operators.CompletableFlatMapSingleToCompletable$SourceSubscriber.onSuccess(CompletableFlatMapSingleToCompletable.java:73)
    at rx.internal.operators.OnSubscribeSingle$1.onCompleted(OnSubscribeSingle.java:55)
    at rx.internal.operators.OnSubscribeTimerOnce$1.call(OnSubscribeTimerOnce.java:54)
    at rx.internal.schedulers.EventLoopsScheduler$EventLoopWorker$2.call(EventLoopsScheduler.java:189)
    at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Exception in thread "main" java.lang.IllegalArgumentException: Invalid resource id 
    at com.microsoft.azure.cosmosdb.internal.ResourceId.parse(ResourceId.java:75)
    at com.microsoft.azure.cosmosdb.internal.ResourceTokenAuthorizationHelper.getAuthorizationTokenUsingResourceTokens(ResourceTokenAuthorizationHelper.java:161)
    at com.microsoft.azure.cosmosdb.rx.internal.RxDocumentClientImpl.getUserAuthorizationToken(RxDocumentClientImpl.java:1057)
    at com.microsoft.azure.cosmosdb.rx.internal.RxDocumentClientImpl.populateHeaders(RxDocumentClientImpl.java:1022)
    at com.microsoft.azure.cosmosdb.rx.internal.RxDocumentClientImpl.query(RxDocumentClientImpl.java:721)
    at com.microsoft.azure.cosmosdb.rx.internal.RxDocumentClientImpl.access$500(RxDocumentClientImpl.java:132)
    at com.microsoft.azure.cosmosdb.rx.internal.RxDocumentClientImpl$2.executeQueryAsync(RxDocumentClientImpl.java:1422)
    at com.microsoft.azure.cosmosdb.rx.internal.query.DocumentQueryExecutionContextBase.executeQueryRequestInternalAsync(DocumentQueryExecutionContextBase.java:144)
    at com.microsoft.azure.cosmosdb.rx.internal.query.DocumentQueryExecutionContextBase.executeQueryRequestAsync(DocumentQueryExecutionContextBase.java:125)
    at com.microsoft.azure.cosmosdb.rx.internal.query.DocumentQueryExecutionContextBase.executeRequestAsync(DocumentQueryExecutionContextBase.java:120)
    at com.microsoft.azure.cosmosdb.rx.internal.query.DefaultDocumentQueryExecutionContext.lambda$null$1(DefaultDocumentQueryExecutionContext.java:126)
    at com.microsoft.azure.cosmosdb.rx.internal.BackoffRetryUtility.lambda$executeRetry$2(BackoffRetryUtility.java:94)
    at rx.Single$18.call(Single.java:2511)
    at rx.Single$18.call(Single.java:2505)
    at rx.internal.operators.SingleToObservable.call(SingleToObservable.java:39)
    at rx.internal.operators.SingleToObservable.call(SingleToObservable.java:27)
    at rx.Observable.unsafeSubscribe(Observable.java:10256)
    at rx.internal.operators.OnSubscribeRedo$2.call(OnSubscribeRedo.java:273)
    at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.enqueue(TrampolineScheduler.java:73)
    at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.schedule(TrampolineScheduler.java:52)
    at rx.internal.operators.OnSubscribeRedo$5.request(OnSubscribeRedo.java:361)
    at rx.Subscriber.setProducer(Subscriber.java:211)
    at rx.internal.operators.OnSubscribeRedo.call(OnSubscribeRedo.java:353)
    at rx.internal.operators.OnSubscribeRedo.call(OnSubscribeRedo.java:47)
    at rx.Observable.unsafeSubscribe(Observable.java:10256)
    at rx.internal.operators.OnSubscribeSingle.call(OnSubscribeSingle.java:81)
    at rx.internal.operators.OnSubscribeSingle.call(OnSubscribeSingle.java:27)
    at rx.internal.operators.SingleToObservable.call(SingleToObservable.java:39)
    at rx.internal.operators.SingleToObservable.call(SingleToObservable.java:27)
    at rx.Observable.unsafeSubscribe(Observable.java:10256)
    at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
    at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
    at rx.Observable.unsafeSubscribe(Observable.java:10256)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    at rx.Observable.subscribe(Observable.java:10352)
    at rx.Observable.subscribe(Observable.java:10319)
    at rx.observables.AsyncOnSubscribe$AsyncOuterManager.subscribeBufferToObservable(AsyncOnSubscribe.java:627)
    at rx.observables.AsyncOnSubscribe$AsyncOuterManager.onNext(AsyncOnSubscribe.java:591)
    at rx.observables.AsyncOnSubscribe$AsyncOuterManager.onNext(AsyncOnSubscribe.java:356)
    at rx.observers.SerializedObserver.onNext(SerializedObserver.java:91)
    at com.microsoft.azure.cosmosdb.rx.internal.query.Paginator$1.next(Paginator.java:92)
    at com.microsoft.azure.cosmosdb.rx.internal.query.Paginator$1.next(Paginator.java:80)
    at rx.observables.AsyncOnSubscribe$AsyncOuterManager.nextIteration(AsyncOnSubscribe.java:413)
    at rx.observables.AsyncOnSubscribe$AsyncOuterManager.tryEmit(AsyncOnSubscribe.java:534)
    at rx.observables.AsyncOnSubscribe$AsyncOuterManager.request(AsyncOnSubscribe.java:455)
    at rx.Subscriber.setProducer(Subscriber.java:211)
    at rx.observables.AsyncOnSubscribe.call(AsyncOnSubscribe.java:352)
    at rx.observables.AsyncOnSubscribe.call(AsyncOnSubscribe.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10256)
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:51)
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10256)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
    at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
    at rx.internal.producers.SingleProducer.request(SingleProducer.java:65)
    at rx.Subscriber.setProducer(Subscriber.java:211)
    at rx.internal.operators.OnSubscribeMap$MapSubscriber.setProducer(OnSubscribeMap.java:102)
    at rx.internal.operators.OperatorSingle$ParentSubscriber.onCompleted(OperatorSingle.java:110)
    at rx.internal.util.ScalarSynchronousObservable$WeakSingleProducer.request(ScalarSynchronousObservable.java:285)
    at rx.Subscriber.setProducer(Subscriber.java:211)
    at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:233)
    at rx.internal.util.ScalarSynchronousObservable$3.call(ScalarSynchronousObservable.java:228)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10256)
    at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
    at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.Observable.subscribe(Observable.java:10352)
    at rx.Observable.subscribe(Observable.java:10319)
    at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:443)
    at rx.observables.BlockingObservable.single(BlockingObservable.java:340)
    at com.microsoft.azure.cosmosdb.benchmark.DocDBUtils.getDatabase(DocDBUtils.java:43)
    at com.microsoft.azure.cosmosdb.benchmark.AsyncBenchmark.<init>(AsyncBenchmark.java:109)
    at com.microsoft.azure.cosmosdb.benchmark.AsyncWriteBenchmark.<init>(AsyncWriteBenchmark.java:74)
    at com.microsoft.azure.cosmosdb.benchmark.Main.main(Main.java:55)

Environment summary SDK Version: 2.3.1 Java JDK version: 1.8 OS Version (e.g. Windows, Linux, MacOSX): Linux

Additional context I've validated this action with 2 ResourceTokens, one granted Read access to a collection and one granted All access to a collection.

simplynaveen20 commented 5 years ago

Thanks Marc for reporting the problem !! Is this blocking or silent exception with everything goes through as expected or if other things fail too?

marc-sensenich commented 5 years ago

It lets other operations complete if followed after initializing the client, see the code block below for reproduction. But will log the exception and occur the network cost of trying to refresh the location cache.

String permissionJSON = "{  \n" +
        "    \"id\": \"a_permission\",  \n" +
        "    \"permissionMode\": \"Read\",  \n" +
        "    \"resource\": \"dbs/volcanodb/colls/volcano1\",  \n" +
        "    \"_rid\": \"Sl8fAG8cXgBn6Ju2GqNsAA==\",  \n" +
        "    \"_ts\": 1449604760,  \n" +
        "    \"_self\": \"dbs\\/Sl8fAA==\\/users\\/Sl8fAG8cXgA=\\/permissions\\/Sl8fAG8cXgBn6Ju2GqNsAA==\\/\",  \n" +
        "    \"_etag\": \"\\\"00000e00-0000-0000-0000-566736980000\\\"\",  \n" +
        "    \"_token\": \"type=resource&ver=1&sig=ocPyc9QQFybITu1EqzX0kg==;w+WR1aWafB3+yZq5JSoBwgz78XDlU+k9Xiqvc+Q7TlAl1P4h4t721Cn5cjhZ9h3TSd2\\/MJLy+wG+YkhDL9UlGkVv05RZGy2fMaLGdeQkWc7TShkc\\/M2boPc3GXq2yiERKl5CN4AZWSOcrFhOFuuTOqF4ZdBlflmNudaakodr\\/8qTip0i+a7moz1Jkc5+9iLAsDFyqTR1sirp7kAVNFbiqPdYTjNkvZUHF3nYYmRskOg=;\"  \n" +
        "}  ";
String cosmosDBURL = "[your host]";
Permission thePermission = new Permission(permissionJSON);

AsyncDocumentClient limitedDocumentClient = new AsyncDocumentClient.Builder()
        .withConnectionPolicy(ConnectionPolicy.GetDefault())
        .withConsistencyLevel(ConsistencyLevel.Strong)
        .withServiceEndpoint(cosmosDBURL)
        .withPermissionFeed(new ArrayList<>(Arrays.asList(thePermission)))
        .build();

String collectionLink = "dbs/integration-testing/colls/integration-testing-collection";

limitedDocumentClient.readCollection(collectionLink, null).toBlocking().first().getResource();
simplynaveen20 commented 5 years ago

Thanks Marc !! We are get the new release by early next week with the fix , in mean time would you mind using withMasterKeyOrResourceToken instead of withPermissionFeed . Thanks again for your patience.

marc-sensenich commented 5 years ago

Thanks @simplynaveen20! Looking forward to the fix

John20001 commented 5 years ago

Hi @simplynaveen20 , I work with @marc-sensenich . I wanted to follow up to make sure your fix would address this issue as well. Yesterday we were attempting to failover locations while reading and writing documents. The azure-cosmosdb library had some helpful log messages indicating inability to reach the gateway server, but the thrown exception itself matches the one in this ticket, making it difficult to catch and identify accurately. If this is considered a different problem I'll create a new issue for it. Thanks!

My logger config file didnt have the correct format, so the client library logs dont have timestamp/etc, but you can see the messages. Apologies for that. 


RequestStartTime: 2019-01-14T22:13:28.6848000Z, Number of regions attempted: 1
, SDK: Microsoft.Azure.Documents.Common/2.1.0.0, StatusCode: Forbidden, responseHeaders={content-length=501, x-ms-last-state-change-utc=Mon, 14 Jan 2019 20:09:33.908 GMT, x-ms-current-replica-set-size=4, Server=Microsoft-HTTPAPI/2.0, x-ms-session-token=0:15#1104#1=1094, lsn=1104, x-ms-schemaversion=1.6, x-ms-transport-request-id=1437, x-ms-number-of-read-regions=1, x-ms-current-write-quorum=3, x-ms-cosmos-quorum-acked-llsn=1102, x-ms-quorum-acked-lsn=1104, x-ms-activity-id=6dcaebb0-0cf5-4570-8abb-409a10783332, Date=Mon, 14 Jan 2019 22:13:27 GMT, x-ms-xp-role=4, Strict-Transport-Security=max-age=31536000, x-ms-global-Committed-lsn=1104, x-ms-cosmos-llsn=1102, x-ms-gatewayversion=version=2.1.0.0, x-ms-serviceversion=version=2.1.0.0, Content-Type=application/json, x-ms-substatus=3}}
marking the endpoint https://bnr-marc-sensenich-testing.documents.azure.com:443/ as unavailable for write
Getting database account endpoint from https://bnr-marc-sensenich-testing.documents.azure.com:443/
Fail to reach global gateway [https://bnr-marc-sensenich-testing.documents.azure.com:443/], [Invalid resource id ]
java.lang.IllegalArgumentException: Invalid resource id 
    at com.microsoft.azure.cosmosdb.internal.ResourceId.parse(ResourceId.java:75)
    at com.microsoft.azure.cosmosdb.internal.ResourceTokenAuthorizationHelper.getAuthorizationTokenUsingResourceTokens(ResourceTokenAuthorizationHelper.java:161)
    at com.microsoft.azure.cosmosdb.rx.internal.RxDocumentClientImpl.getUserAuthorizationToken(RxDocumentClientImpl.java:1057)
    at com.microsoft.azure.cosmosdb.rx.internal.RxDocumentClientImpl.populateHeaders(RxDocumentClientImpl.java:1022)
    at com.microsoft.azure.cosmosdb.rx.internal.RxDocumentClientImpl.lambda$getDatabaseAccountFromEndpoint$144(RxDocumentClientImpl.java:3042)
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:46)
    at rx.internal.operators.OnSubscribeDefer.call(OnSubscribeDefer.java:35)
    at rx.Observable.unsafeSubscribe(Observable.java:10327)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:41)
    at rx.internal.operators.OnSubscribeDoOnEach.call(OnSubscribeDoOnEach.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10327)
    at rx.internal.operators.OnSubscribeSingle.call(OnSubscribeSingle.java:81)
    at rx.internal.operators.OnSubscribeSingle.call(OnSubscribeSingle.java:27)
    at rx.Single.subscribe(Single.java:1979)
    at rx.internal.operators.SingleOperatorOnErrorResumeNext.call(SingleOperatorOnErrorResumeNext.java:77)
    at rx.internal.operators.SingleOperatorOnErrorResumeNext.call(SingleOperatorOnErrorResumeNext.java:23)
    at rx.Single.subscribe(Single.java:1979)
    at rx.internal.operators.SingleOnSubscribeMap.call(SingleOnSubscribeMap.java:45)
    at rx.internal.operators.SingleOnSubscribeMap.call(SingleOnSubscribeMap.java:30)
    at rx.Single.subscribe(Single.java:1979)
    at rx.internal.operators.CompletableFlatMapSingleToCompletable.call(CompletableFlatMapSingleToCompletable.java:43)
    at rx.internal.operators.CompletableFlatMapSingleToCompletable.call(CompletableFlatMapSingleToCompletable.java:28)
    at rx.Completable.unsafeSubscribe(Completable.java:2035)
    at rx.Completable$5.call(Completable.java:398)
    at rx.Completable$5.call(Completable.java:379)
    at rx.Completable.unsafeSubscribe(Completable.java:2035)
    at rx.Completable$19.call(Completable.java:1365)
    at rx.Completable$19.call(Completable.java:1362)
    at rx.Completable.unsafeSubscribe(Completable.java:2035)
    at rx.Completable$5.call(Completable.java:398)
    at rx.Completable$5.call(Completable.java:379)
    at rx.Completable.unsafeSubscribe(Completable.java:2035)
    at rx.Completable.unsafeSubscribe(Completable.java:2083)
    at rx.Completable.unsafeSubscribe(Completable.java:2067)
    at rx.Completable$32.call(Completable.java:2253)
    at rx.Completable$32.call(Completable.java:2250)
    at rx.Observable.subscribe(Observable.java:10423)
    at rx.Observable.subscribe(Observable.java:10390)
    at rx.internal.operators.SingleOnSubscribeDelaySubscriptionOther.call(SingleOnSubscribeDelaySubscriptionOther.java:87)
    at rx.internal.operators.SingleOnSubscribeDelaySubscriptionOther.call(SingleOnSubscribeDelaySubscriptionOther.java:29)
    at rx.internal.operators.SingleToObservable.call(SingleToObservable.java:39)
    at rx.internal.operators.SingleToObservable.call(SingleToObservable.java:27)
    at rx.Observable.unsafeSubscribe(Observable.java:10327)
    at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
    at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
    at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
    at rx.Observable.unsafeSubscribe(Observable.java:10327)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
    at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
    at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
    at rx.internal.operators.OnSubscribeRedo$3$1.onNext(OnSubscribeRedo.java:302)
    at rx.internal.operators.OnSubscribeRedo$3$1.onNext(OnSubscribeRedo.java:284)
    at rx.internal.operators.NotificationLite.accept(NotificationLite.java:135)
    at rx.subjects.SubjectSubscriptionManager$SubjectObserver.emitNext(SubjectSubscriptionManager.java:253)
    at rx.subjects.BehaviorSubject.onNext(BehaviorSubject.java:160)
    at rx.observers.SerializedObserver.onNext(SerializedObserver.java:91)
    at rx.subjects.SerializedSubject.onNext(SerializedSubject.java:67)
    at rx.internal.operators.OnSubscribeRedo$2$1.onError(OnSubscribeRedo.java:237)
    at rx.internal.operators.SingleLiftObservableOperator$WrapSubscriberIntoSingle.onError(SingleLiftObservableOperator.java:81)
    at rx.internal.operators.OnSubscribeSingle$1.onError(OnSubscribeSingle.java:64)
    at rx.internal.operators.OnSubscribeMap$MapSubscriber.onError(OnSubscribeMap.java:88)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.reportError(OperatorMerge.java:266)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.checkTerminate(OperatorMerge.java:818)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.emitLoop(OperatorMerge.java:579)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.emit(OperatorMerge.java:568)
    at rx.internal.operators.OperatorMerge$InnerSubscriber.onError(OperatorMerge.java:855)
    at rx.internal.operators.OnSubscribeDoOnEach$DoOnEachSubscriber.onError(OnSubscribeDoOnEach.java:87)
    at rx.internal.operators.OnSubscribeMap$MapSubscriber.onError(OnSubscribeMap.java:88)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.reportError(OperatorMerge.java:266)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.checkTerminate(OperatorMerge.java:818)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.emitLoop(OperatorMerge.java:579)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.emit(OperatorMerge.java:568)
    at rx.internal.operators.OperatorMerge$InnerSubscriber.onError(OperatorMerge.java:855)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.reportError(OperatorMerge.java:266)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.checkTerminate(OperatorMerge.java:818)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.emitLoop(OperatorMerge.java:579)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.emit(OperatorMerge.java:568)
    at rx.internal.operators.OperatorMerge$InnerSubscriber.onError(OperatorMerge.java:855)
    at rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:44)
    at rx.internal.operators.OnSubscribeThrow.call(OnSubscribeThrow.java:28)
    at rx.Observable.unsafeSubscribe(Observable.java:10327)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:248)
    at rx.internal.operators.OperatorMerge$MergeSubscriber.onNext(OperatorMerge.java:148)
    at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
    at rx.internal.operators.OnSubscribeMap$MapSubscriber.onNext(OnSubscribeMap.java:77)
    at rx.internal.operators.DeferredScalarSubscriber.complete(DeferredScalarSubscriber.java:100)
    at rx.internal.operators.DeferredScalarSubscriber.onCompleted(DeferredScalarSubscriber.java:73)
    at io.reactivex.netty.protocol.http.UnicastContentSubject$AutoReleaseByteBufOperator$1.onCompleted(UnicastContentSubject.java:260)
    at rx.internal.operators.BufferUntilSubscriber.onCompleted(BufferUntilSubscriber.java:156)
    at io.reactivex.netty.protocol.http.UnicastContentSubject.onCompleted(UnicastContentSubject.java:282)
    at io.reactivex.netty.protocol.http.client.ClientRequestResponseConverter$ResponseState.sendOnComplete(ClientRequestResponseConverter.java:413)
    at io.reactivex.netty.protocol.http.client.ClientRequestResponseConverter$ResponseState.access$500(ClientRequestResponseConverter.java:350)
    at io.reactivex.netty.protocol.http.client.ClientRequestResponseConverter.channelRead(ClientRequestResponseConverter.java:168)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)
    at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284)
    at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1389)
    at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1159)
    at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1203)
    at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
    at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
    at io.reactivex.netty.metrics.BytesInspector.channelRead(BytesInspector.java:59)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.reactivex.netty.pipeline.InternalReadTimeoutHandler.channelRead(InternalReadTimeoutHandler.java:108)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:146)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
    at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.lang.Thread.run(Thread.java:748)
simplynaveen20 commented 5 years ago

Hi @John20001 / @marc-sensenich - Please try both the issues with version 2.3.1

marc-sensenich commented 5 years ago

@simplynaveen20 this code has been tested by @John20001 and I. This resolves the issues we were encountering and I am going to close the issue.

simplynaveen20 commented 5 years ago

Glad to hear that , thank you again for actively working on it , let us know if you face any other problem.