tknandu
commented
6 years ago Could you please provide more context on your scenario? And possibly a code snippet?
Open yuvrajyr opened 6 years ago
tknandu
commented
6 years ago Could you please provide more context on your scenario? And possibly a code snippet?
alrickdbritto
commented
5 years ago Hi,
i am facing a similar type of error while writing data to cosmosdb.
below is my error: 18/09/21 09:08:19 ERROR Executor: Exception in task 0.0 in stage 22.0 (TID 25) java.lang.Exception: Errors encountered in bulk import API execution. Exceptions observed: [java.lang.RuntimeException: com.microsoft.azure.documentdb.DocumentClientException: {"Errors":["Encountered exception while executing function. Exception = JSRT error 0x30003"]} ActivityId: e18ebda4-bd7d-11e8-90ec-032dc568ba7f, Request URI: /apps/9131100f-38cc-47f3-9b1a-fb55c528a04b/services/6a0847ad-e73c-4eae-98d4-f5151f4f77be/partitions/1203fb02-beee-4b92-b67b-4c0e4ac4d50e/replicas/131818494049782580p//, StatusCode: InternalServerError, SubStatusCode: ] at com.microsoft.azure.cosmosdb.spark.CosmosDBSpark$.bulkImport(CosmosDBSpark.scala:311) at com.microsoft.azure.cosmosdb.spark.CosmosDBSpark$.savePartition(CosmosDBSpark.scala:467) at com.microsoft.azure.cosmosdb.spark.CosmosDBSpark$.com$microsoft$azure$cosmosdb$spark$CosmosDBSpark$$savePartition(CosmosDBSpark.scala:408) at com.microsoft.azure.cosmosdb.spark.CosmosDBSpark$$anonfun$1.apply(CosmosDBSpark.scala:192) at com.microsoft.azure.cosmosdb.spark.CosmosDBSpark$$anonfun$1.apply(CosmosDBSpark.scala:183) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:847) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsWithIndex$1$$anonfun$apply$26.apply(RDD.scala:847) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324) at org.apache.spark.rdd.RDD.iterator(RDD.scala:288) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:109) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 18/09/21 09:08:19 INFO CoarseGrainedExecutorBackend: Got assigned task 26 18/09/21 09:08:19 INFO Executor: Running task 0.1 in stage 22.0 (TID 26) 18/09/21 09:08:19 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 1 blocks 18/09/21 09:08:19 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 18/09/21 09:08:19 ERROR BatchInserter: pki 0 failed to import mini-batch. Exception was {"Errors":["Encountered exception while executing function. Exception = JSRT error 0x30003"]} ActivityId: e22ef547-bd7d-11e8-90ec-032dc568ba7f, Request URI: /apps/9131100f-38cc-47f3-9b1a-fb55c528a04b/services/6a0847ad-e73c-4eae-98d4-f5151f4f77be/partitions/1203fb02-beee-4b92-b67b-4c0e4ac4d50e/replicas/131818494049782580p//, StatusCode: InternalServerError, SubStatusCode: . Status code was 500 com.microsoft.azure.documentdb.DocumentClientException: {"Errors":["Encountered exception while executing function. Exception = JSRT error 0x30003"]} ActivityId: e22ef547-bd7d-11e8-90ec-032dc568ba7f, Request URI: /apps/9131100f-38cc-47f3-9b1a-fb55c528a04b/services/6a0847ad-e73c-4eae-98d4-f5151f4f77be/partitions/1203fb02-beee-4b92-b67b-4c0e4ac4d50e/replicas/131818494049782580p//, StatusCode: InternalServerError, SubStatusCode: at com.microsoft.azure.documentdb.internal.ErrorUtils.maybeThrowException(ErrorUtils.java:74) at com.microsoft.azure.documentdb.internal.directconnectivity.HttpTransportClient.processResponse(HttpTransportClient.java:187) at com.microsoft.azure.documentdb.internal.directconnectivity.HttpTransportClient.invokeStore(HttpTransportClient.java:156) at com.microsoft.azure.documentdb.internal.directconnectivity.HttpTransportClient.invokeStore(HttpTransportClient.java:163) at com.microsoft.azure.documentdb.internal.directconnectivity.TransportClient.invokeResourceOperation(TransportClient.java:12) at com.microsoft.azure.documentdb.internal.directconnectivity.ConsistencyWriter.writePrivate(ConsistencyWriter.java:105) at com.microsoft.azure.documentdb.internal.directconnectivity.ConsistencyWriter.write(ConsistencyWriter.java:69) at com.microsoft.azure.documentdb.internal.directconnectivity.ReplicatedResourceClient.invoke(ReplicatedResourceClient.java:66) at com.microsoft.azure.documentdb.internal.directconnectivity.ServerStoreModel$1.apply(ServerStoreModel.java:103) at com.microsoft.azure.documentdb.internal.RetryUtility.executeStoreClientRequest(RetryUtility.java:133) at com.microsoft.azure.documentdb.internal.directconnectivity.ServerStoreModel.processMessage(ServerStoreModel.java:114) at com.microsoft.azure.documentdb.DocumentClient$4.apply(DocumentClient.java:3037) at com.microsoft.azure.documentdb.internal.RetryUtility.executeDocumentClientRequest(RetryUtility.java:71) at com.microsoft.azure.documentdb.DocumentClient.doCreate(DocumentClient.java:3042) at com.microsoft.azure.documentdb.DocumentClient.executeStoredProcedure(DocumentClient.java:1578) at com.microsoft.azure.documentdb.bulkexecutor.internal.BatchInserter$1.call(BatchInserter.java:158) at com.microsoft.azure.documentdb.bulkexecutor.internal.BatchInserter$1.call(BatchInserter.java:130) at cosmosdb_connector_shaded.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111) at cosmosdb_connector_shaded.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58) at cosmosdb_connector_shaded.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 18/09/21 09:08:19 ERROR CongestionController: pki 0 encountered failure {} releasing semaphore java.lang.RuntimeException: com.microsoft.azure.documentdb.DocumentClientException: {"Errors":["Encountered exception while executing function. Exception = JSRT error 0x30003"]} ActivityId: e22ef547-bd7d-11e8-90ec-032dc568ba7f, Request URI: /apps/9131100f-38cc-47f3-9b1a-fb55c528a04b/services/6a0847ad-e73c-4eae-98d4-f5151f4f77be/partitions/1203fb02-beee-4b92-b67b-4c0e4ac4d50e/replicas/131818494049782580p//, StatusCode: InternalServerError, SubStatusCode: at com.microsoft.azure.documentdb.bulkexecutor.internal.BatchInserter$1.call(BatchInserter.java:233) at com.microsoft.azure.documentdb.bulkexecutor.internal.BatchInserter$1.call(BatchInserter.java:130) at cosmosdb_connector_shaded.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:111) at cosmosdb_connector_shaded.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:58) at cosmosdb_connector_shaded.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:75) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: com.microsoft.azure.documentdb.DocumentClientException: {"Errors":["Encountered exception while executing function. Exception = JSRT error 0x30003"]} ActivityId: e22ef547-bd7d-11e8-90ec-032dc568ba7f, Request URI: /apps/9131100f-38cc-47f3-9b1a-fb55c528a04b/services/6a0847ad-e73c-4eae-98d4-f5151f4f77be/partitions/1203fb02-beee-4b92-b67b-4c0e4ac4d50e/replicas/131818494049782580p//, StatusCode: InternalServerError, SubStatusCode: at com.microsoft.azure.documentdb.internal.ErrorUtils.maybeThrowException(ErrorUtils.java:74) at com.microsoft.azure.documentdb.internal.directconnectivity.HttpTransportClient.processResponse(HttpTransportClient.java:187) at com.microsoft.azure.documentdb.internal.directconnectivity.HttpTransportClient.invokeStore(HttpTransportClient.java:156) at com.microsoft.azure.documentdb.internal.directconnectivity.HttpTransportClient.invokeStore(HttpTransportClient.java:163) at com.microsoft.azure.documentdb.internal.directconnectivity.TransportClient.invokeResourceOperation(TransportClient.java:12) at com.microsoft.azure.documentdb.internal.directconnectivity.ConsistencyWriter.writePrivate(ConsistencyWriter.java:105) at com.microsoft.azure.documentdb.internal.directconnectivity.ConsistencyWriter.write(ConsistencyWriter.java:69) at com.microsoft.azure.documentdb.internal.directconnectivity.ReplicatedResourceClient.invoke(ReplicatedResourceClient.java:66) at com.microsoft.azure.documentdb.internal.directconnectivity.ServerStoreModel$1.apply(ServerStoreModel.java:103) at com.microsoft.azure.documentdb.internal.RetryUtility.executeStoreClientRequest(RetryUtility.java:133) at com.microsoft.azure.documentdb.internal.directconnectivity.ServerStoreModel.processMessage(ServerStoreModel.java:114) at com.microsoft.azure.documentdb.DocumentClient$4.apply(DocumentClient.java:3037) at com.microsoft.azure.documentdb.internal.RetryUtility.executeDocumentClientRequest(RetryUtility.java:71) at com.microsoft.azure.documentdb.DocumentClient.doCreate(DocumentClient.java:3042) at com.microsoft.azure.documentdb.DocumentClient.executeStoredProcedure(DocumentClient.java:1578) at com.microsoft.azure.documentdb.bulkexecutor.internal.BatchInserter$1.call(BatchInserter.java:158) ... 7 more
Enoch-Ye
commented
5 years ago We are facing the similar issue under the stress testing and how cosmos db identify the each resource? By resource/document id?
slechta
commented
5 years ago I am also facing the same issue. Could you please describe when exactly this issue happens and what is causing it? Thank you.
2019-07-29 15:40:33 [rxnetty-nio-eventloop-4-1] WARN GoneAndRetryWithRetryPolicy:194 - Received retrywith exception, will retry, {}
--
| com.microsoft.azure.cosmosdb.internal.directconnectivity.RetryWithException: Message: {"Errors":["Conflicting request to resource has been attempted. Retry to avoid conflicts."]}
| at com.microsoft.azure.cosmosdb.internal.directconnectivity.HttpTransportClient.lambda$createErrorResponseFromHttpResponse$5(HttpTransportClient.java:927) ~[azure-cosmosdb-direct-2.4.5.jar!/:?]
| at rx.internal.operators.SingleOnSubscribeMap$MapSubscriber.onSuccess(SingleOnSubscribeMap.java:66) [rxjava-1.3.8.jar!/:1.3.8]
| at rx.internal.operators.OnSubscribeSingle$1.onCompleted(OnSubscribeSingle.java:55) [rxjava-1.3.8.jar!/:1.3.8]
| at rx.internal.operators.OnSubscribeMap$MapSubscriber.onCompleted(OnSubscribeMap.java:97) [rxjava-1.3.8.jar!/:1.3.8]
| at rx.internal.operators.DeferredScalarSubscriber.complete(DeferredScalarSubscriber.java:102) [rxjava-1.3.8.jar!/:1.3.8]
| at rx.internal.operators.DeferredScalarSubscriber.onCompleted(DeferredScalarSubscriber.java:73) [rxjava-1.3.8.jar!/:1.3.8]
| at io.reactivex.netty.protocol.http.UnicastContentSubject$AutoReleaseByteBufOperator$1.onCompleted(UnicastContentSubject.java:260) [rxnetty-0.4.20.jar!/:0.4.20]
| at rx.internal.operators.BufferUntilSubscriber.onCompleted(BufferUntilSubscriber.java:156) [rxjava-1.3.8.jar!/:1.3.8]
| at io.reactivex.netty.protocol.http.UnicastContentSubject.onCompleted(UnicastContentSubject.java:282) [rxnetty-0.4.20.jar!/:0.4.20]
| at io.reactivex.netty.protocol.http.client.ClientRequestResponseConverter$ResponseState.sendOnComplete(ClientRequestResponseConverter.java:413) [rxnetty-0.4.20.jar!/:0.4.20]
| at io.reactivex.netty.protocol.http.client.ClientRequestResponseConverter$ResponseState.access$500(ClientRequestResponseConverter.java:350) [rxnetty-0.4.20.jar!/:0.4.20]
| at io.reactivex.netty.protocol.http.client.ClientRequestResponseConverter.channelRead(ClientRequestResponseConverter.java:168) [rxnetty-0.4.20.jar!/:0.4.20]
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323) [netty-codec-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1478) [netty-handler-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) [netty-handler-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) [netty-handler-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) [netty-codec-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) [netty-codec-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) [netty-codec-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.reactivex.netty.metrics.BytesInspector.channelRead(BytesInspector.java:59) [rxnetty-0.4.20.jar!/:0.4.20]
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.reactivex.netty.pipeline.InternalReadTimeoutHandler.channelRead(InternalReadTimeoutHandler.java:108) [rxnetty-0.4.20.jar!/:0.4.20]
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:617) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:534) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906) [netty-common-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.36.Final.jar!/:4.1.36.Final]
| at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.36.Final.jar!/:4.1.36.Final]
| at java.lang.Thread.run(Thread.java:834) [?:?]
Hi i am frequently getting below error while writing to empty collection.
java.lang.RuntimeException: com.microsoft.azure.documentdb.DocumentClientException: Message: {"Errors":["Conflicting request to resource has been attempted. Retry to avoid conflicts."]} RequestStats: , SDK: Microsoft.Azure.Documents.Common/2.0.0.0, StatusCode: Retrytherequest