FeedIterator.ReadNextAsync() fails when client in Direct mode and uses TokenCredential

[jformacek]

Describe the bug Developing code that is planned to run in Azure function (.NET7 isolated). Using Cosmos SDK 3.31.2 Getting results from cross-partition query via FeedIterator. Planning to use ManagedItentity of Azure function for authentication to DB. service principal has 'Cosmos DB Built-in Data Contributor' role assigned. When developing, Visual Studio authenticates to DB with my account via InteractiveBrowserCredential(), my account has 'Cosmos DB Built-in Data Contributor' assigned, too.

• when Cosmos client created with TokenCredential and uses Gateway mode -> FeedIterator works as expected
• when Cosmos client created with ConnectionString instead of TokenCredential and uses Direct mode -> FeedIterator works as expected
• when Cosmos client created with TokenCredential and uses Direct mode -> FeedIterator returns first page of results, but 2nd call to ReadNextAsync() produces ServiceUnavailableException (503) with SubStatus 20001. Takes ~30 secs to produce the exception, while throwing and handling many exceptions internally Stack trace and diagnostic data from exception below. All tests performed on the same machine, I'm not aware of any blocking of outbound traffic, so I do not think it's networking issue. Other operations that use Cosmos client with Token credential and Direct connection seem to be working fine - at least item creation, read and partial update.

To Reproduce

• Cosmos account provisioned in westeurope region, geo replication disabled
• Single DB and single container with 4000 RU limit
• ~1M items across ~10K partitions in the container
• query to return items like this

  var query = @"SELECT 
          c.sub.code as code, 
          c.sub.division as division 
      FROM c 
      WHERE 
          c.itemType = @itemType
          and c.sub.userType=@userType 
          and not is_null(c.sub.code) 
          and not is_null(c.sub.division )
          and c.sub.division !=@unknown"
       ).WithParameter("@unknown", "unknown")
        .WithParameter("@userType","Type1")
        .WithParameter("@itemType","Account");

• code basically like this

  using var iterator = _inventory.GetItemQueryIterator<ResultData>(query);
  while (iterator.HasMoreResults)
  {
    //throws ServiceUnavailableException on 2nd call with TokenCredential and Direct mode
   var rsp = await iterator.ReadNextAsync();
  }

Expected behavior FeedIterator.ReadNextAsync() is expected to provide same result in Direct mode regardless client initialized with connection string or TokenCredential

Actual behavior In Direct mode, FeedIterator.ReadNextAsync() throws http 503 when client initialized with TokenCredential, but works as expected when client initialized with connection string

Environment summary SDK Version: 3.31.2

    <PackageReference Include="Azure.Identity" Version="1.8.0" />
    <PackageReference Include="Microsoft.Azure.Cosmos" Version="3.31.2" />

OS Version (e.g. Windows, Linux, MacOSX): Windows 11, Windows 10. Visual Studio 2022

Additional context Stack trace:

ActivityId: 37a89a4d-9555-486e-8014-d38e493babe6, Windows/10.0.22621 cosmos-netstandard-sdk/3.29.4
 ---> Microsoft.Azure.Documents.GoneException: Message: The requested resource is no longer available at the server.
ActivityId: 37a89a4d-9555-486e-8014-d38e493babe6, Request URI: /apps/78b26377-fa10-497d-9d94-bf41b72ff220/services/10423294-eb5e-460a-9516-d890c9a11c8c/partitions/8b46f69d-d1af-42cf-887a-ad8f85898097/replicas/133178469155636226s/, RequestStats: , SDK: Windows/10.0.22621 cosmos-netstandard-sdk/3.29.4
 ---> Microsoft.Azure.Documents.TransportException: A client transport error occurred: The connection failed. (Time: 2023-01-13T10:13:28.0558878Z, activity ID: 37a89a4d-9555-486e-8014-d38e493babe6, error code: ConnectionBroken [0x0012], base error: HRESULT 0x80131500, URI: rntbd://cdb-ms-prod-westeurope1-fd70.documents.azure.com:14309/apps/78b26377-fa10-497d-9d94-bf41b72ff220/services/10423294-eb5e-460a-9516-d890c9a11c8c/partitions/8b46f69d-d1af-42cf-887a-ad8f85898097/replicas/133178469155636226s/, connection: 192.168.1.135:50554 -> 13.69.66.26:14309, payload sent: True)
 ---> Microsoft.Azure.Documents.TransportException: A client transport error occurred: The remote system closed the connection. (Time: 2023-01-13T10:13:27.9799108Z, activity ID: 00000000-0000-0000-0000-000000000000, error code: ReceiveStreamClosed [0x0011], base error: HRESULT 0x80131500, URI: rntbd://cdb-ms-prod-westeurope1-fd70.documents.azure.com:14309/, connection: 192.168.1.135:50554 -> 13.69.66.26:14309, payload sent: True)
   at Microsoft.Azure.Documents.Rntbd.Connection.TraceAndThrowEndOfStream(String type, ChannelCommonArguments args)
   at Microsoft.Azure.Documents.Rntbd.Connection.ReadPayloadAsync(Byte[] payload, Int32 length, String type, ChannelCommonArguments args)
   at Microsoft.Azure.Documents.Rntbd.Connection.ReadResponseMetadataAsync(ChannelCommonArguments args)
   at Microsoft.Azure.Documents.Rntbd.Dispatcher.ReceiveLoopAsync()
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Documents.Rntbd.Dispatcher.CallAsync(ChannelCallArguments args, TransportRequestStats transportRequestStats)
   at Microsoft.Azure.Documents.Rntbd.Channel.RequestAsync(DocumentServiceRequest request, TransportAddressUri physicalAddress, ResourceOperation resourceOperation, Guid activityId, TransportRequestStats transportRequestStats)
   at Microsoft.Azure.Documents.Rntbd.LoadBalancingPartition.RequestAsync(DocumentServiceRequest request, TransportAddressUri physicalAddress, ResourceOperation resourceOperation, Guid activityId, TransportRequestStats transportRequestStats)
   at Microsoft.Azure.Documents.Rntbd.TransportClient.InvokeStoreAsync(TransportAddressUri physicalAddress, ResourceOperation resourceOperation, DocumentServiceRequest request)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Documents.Rntbd.TransportClient.InvokeStoreAsync(TransportAddressUri physicalAddress, ResourceOperation resourceOperation, DocumentServiceRequest request)
   at Microsoft.Azure.Documents.StoreReader.CompleteActivity(Task`1 task, QueryRequestPerformanceActivity activity)
   at Microsoft.Azure.Documents.StoreReader.ReadFromStoreAsync(TransportAddressUri physicalAddress, DocumentServiceRequest request)
   at Microsoft.Azure.Documents.StoreReader.ReadMultipleReplicasInternalAsync(DocumentServiceRequest entity, Boolean includePrimary, Int32 replicaCountToRead, Boolean requiresValidLsn, Boolean useSessionToken, ReadMode readMode, Boolean checkMinLSN, Boolean forceReadAll)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Documents.StoreReader.ReadMultipleReplicasInternalAsync(DocumentServiceRequest entity, Boolean includePrimary, Int32 replicaCountToRead, Boolean requiresValidLsn, Boolean useSessionToken, ReadMode readMode, Boolean checkMinLSN, Boolean forceReadAll)
   at Microsoft.Azure.Documents.StoreReader.ReadMultipleReplicaAsync(DocumentServiceRequest entity, Boolean includePrimary, Int32 replicaCountToRead, Boolean requiresValidLsn, Boolean useSessionToken, ReadMode readMode, Boolean checkMinLSN, Boolean forceReadAll)
   at Microsoft.Azure.Documents.ConsistencyReader.ReadSessionAsync(DocumentServiceRequest entity, ReadMode readMode)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetryAsync[TParam,TPolicy](Func`1 callbackMethod, Func`3 callbackMethodWithParam, Func`2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy`1 retryPolicyWithArg, Func`1 inBackoffAlternateCallbackMethod, Func`2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback)
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetryAsync[TParam,TPolicy](Func`1 callbackMethod, Func`3 callbackMethodWithParam, Func`2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy`1 retryPolicyWithArg, Func`1 inBackoffAlternateCallbackMethod, Func`2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass31_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func`1 executeAsync, Func`1 prepareRequest, IRequestRetryPolicy`2 policy, CancellationToken cancellationToken, Func`1 inBackoffAlternateCallbackMethod, Nullable`1 minBackoffForInBackoffCallback)
DocDBTrace Error: 0 : DocumentClientException with status code ServiceUnavailable, message: The request failed because the client was unable to establish connections to 3 endpoints across 1 regions. Please check for client resource starvation issues and verify connectivity between client and server. More info: https://aka.ms/cosmosdb-tsg-service-unavailable, inner exception: Microsoft.Azure.Documents.GoneException: The requested resource is no longer available at the server.
ActivityId: 37a89a4d-9555-486e-8014-d38e493babe6, Windows/10.0.22621 cosmos-netstandard-sdk/3.29.4
 ---> Microsoft.Azure.Documents.GoneException: Message: The requested resource is no longer available at the server.
ActivityId: 37a89a4d-9555-486e-8014-d38e493babe6, Request URI: /apps/78b26377-fa10-497d-9d94-bf41b72ff220/services/10423294-eb5e-460a-9516-d890c9a11c8c/partitions/8b46f69d-d1af-42cf-887a-ad8f85898097/replicas/133178469155636226s/, RequestStats: , SDK: Windows/10.0.22621 cosmos-netstandard-sdk/3.29.4
 ---> Microsoft.Azure.Documents.TransportException: A client transport error occurred: The connection failed. (Time: 2023-01-13T10:13:28.0558878Z, activity ID: 37a89a4d-9555-486e-8014-d38e493babe6, error code: ConnectionBroken [0x0012], base error: HRESULT 0x80131500, URI: rntbd://cdb-ms-prod-westeurope1-fd70.documents.azure.com:14309/apps/78b26377-fa10-497d-9d94-bf41b72ff220/services/10423294-eb5e-460a-9516-d890c9a11c8c/partitions/8b46f69d-d1af-42cf-887a-ad8f85898097/replicas/133178469155636226s/, connection: 192.168.1.135:50554 -> 13.69.66.26:14309, payload sent: True)
 ---> Microsoft.Azure.Documents.TransportException: A client transport error occurred: The remote system closed the connection. (Time: 2023-01-13T10:13:27.9799108Z, activity ID: 00000000-0000-0000-0000-000000000000, error code: ReceiveStreamClosed [0x0011], base error: HRESULT 0x80131500, URI: rntbd://cdb-ms-prod-westeurope1-fd70.documents.azure.com:14309/, connection: 192.168.1.135:50554 -> 13.69.66.26:14309, payload sent: True)
   at Microsoft.Azure.Documents.Rntbd.Connection.TraceAndThrowEndOfStream(String type, ChannelCommonArguments args)
   at Microsoft.Azure.Documents.Rntbd.Connection.ReadPayloadAsync(Byte[] payload, Int32 length, String type, ChannelCommonArguments args)
   at Microsoft.Azure.Documents.Rntbd.Connection.ReadResponseMetadataAsync(ChannelCommonArguments args)
   at Microsoft.Azure.Documents.Rntbd.Dispatcher.ReceiveLoopAsync()
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Documents.Rntbd.Dispatcher.CallAsync(ChannelCallArguments args, TransportRequestStats transportRequestStats)
   at Microsoft.Azure.Documents.Rntbd.Channel.RequestAsync(DocumentServiceRequest request, TransportAddressUri physicalAddress, ResourceOperation resourceOperation, Guid activityId, TransportRequestStats transportRequestStats)
   at Microsoft.Azure.Documents.Rntbd.LoadBalancingPartition.RequestAsync(DocumentServiceRequest request, TransportAddressUri physicalAddress, ResourceOperation resourceOperation, Guid activityId, TransportRequestStats transportRequestStats)
   at Microsoft.Azure.Documents.Rntbd.TransportClient.InvokeStoreAsync(TransportAddressUri physicalAddress, ResourceOperation resourceOperation, DocumentServiceRequest request)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Documents.Rntbd.TransportClient.InvokeStoreAsync(TransportAddressUri physicalAddress, ResourceOperation resourceOperation, DocumentServiceRequest request)
   at Microsoft.Azure.Documents.StoreReader.CompleteActivity(Task`1 task, QueryRequestPerformanceActivity activity)
   at Microsoft.Azure.Documents.StoreReader.ReadFromStoreAsync(TransportAddressUri physicalAddress, DocumentServiceRequest request)
   at Microsoft.Azure.Documents.StoreReader.ReadMultipleReplicasInternalAsync(DocumentServiceRequest entity, Boolean includePrimary, Int32 replicaCountToRead, Boolean requiresValidLsn, Boolean useSessionToken, ReadMode readMode, Boolean checkMinLSN, Boolean forceReadAll)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Documents.StoreReader.ReadMultipleReplicasInternalAsync(DocumentServiceRequest entity, Boolean includePrimary, Int32 replicaCountToRead, Boolean requiresValidLsn, Boolean useSessionToken, ReadMode readMode, Boolean checkMinLSN, Boolean forceReadAll)
   at Microsoft.Azure.Documents.StoreReader.ReadMultipleReplicaAsync(DocumentServiceRequest entity, Boolean includePrimary, Int32 replicaCountToRead, Boolean requiresValidLsn, Boolean useSessionToken, ReadMode readMode, Boolean checkMinLSN, Boolean forceReadAll)
   at Microsoft.Azure.Documents.ConsistencyReader.ReadSessionAsync(DocumentServiceRequest entity, ReadMode readMode)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetryAsync[TParam,TPolicy](Func`1 callbackMethod, Func`3 callbackMethodWithParam, Func`2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy`1 retryPolicyWithArg, Func`1 inBackoffAlternateCallbackMethod, Func`2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback)
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetryAsync[TParam,TPolicy](Func`1 callbackMethod, Func`3 callbackMethodWithParam, Func`2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy`1 retryPolicyWithArg, Func`1 inBackoffAlternateCallbackMethod, Func`2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass31_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func`1 executeAsync, Func`1 prepareRequest, IRequestRetryPolicy`2 policy, CancellationToken cancellationToken, Func`1 inBackoffAlternateCallbackMethod, Nullable`1 minBackoffForInBackoffCallback), and response headers: null

Diagnostics coming with exception is pretty huge and contains PII, so not sharing here, however it's full of reports like below

"StoreResult": {
    "ActivityId": "37a89a4d-9555-486e-8014-d38e493babe6",
    "StatusCode": "Gone",
    "SubStatusCode": "TransportGenerated410",
    "LSN": -1,
    "PartitionKeyRangeId": null,
    "GlobalCommittedLSN": -1,
    "ItemLSN": -1,
    "UsingLocalLSN": true,
    "QuorumAckedLSN": -1,
    "SessionToken": null,
    "CurrentWriteQuorum": -1,
    "CurrentReplicaSetSize": -1,
    "NumberOfReadRegions": -1,
    "IsValid": false,
    "StorePhysicalAddress": "rntbd://cdb-ms-prod-westeurope1-fd70.documents.azure.com:14007/apps/78b26377-fa10-497d-9d94-bf41b72ff220/services/10423294-eb5e-460a-9516-d890c9a11c8c/partitions/8b46f69d-d1af-42cf-887a-ad8f85898097/replicas/133178469155636227p/",
    "RequestCharge": 0,
    "RetryAfterInMs": null,
    "BELatencyInMs": null,
    "transportRequestTimeline": {
        "requestTimeline": [{
                "event": "Created",
                "startTimeUtc": "2023-01-13T10:12:58.3131744Z",
                "durationInMs": 0.0511
            }, {
                "event": "ChannelAcquisitionStarted",
                "startTimeUtc": "2023-01-13T10:12:58.3132255Z",
                "durationInMs": 238.1493
            }, {
                "event": "Pipelined",
                "startTimeUtc": "2023-01-13T10:12:
                58.5513748Z",
                "durationInMs": 9.0976
            }, {
                "event": "Transit Time",
                "startTimeUtc": "2023-01-13T10:12:58.5604724Z",
                "durationInMs": 285.7278
            }, {
                "event": "Failed",
                "startTimeUtc": "2023-01-13T10:12:58.8462002Z",
                "durationInMs": 0
            }
        ],
        "serviceEndpointStats": {
            "inflightRequests": 1,
            "openConnections": 1
        },
        "connectionStats": {
            "waitforConnectionInit": "True",
            "callsPendingReceive": 0,
            "lastSendAttempt": "2023-01-13T10:12:58.5167830Z",
            "lastSend": "2023-01-13T10:12:58.5168791Z",
            "lastReceive": "2023-01-13T10:12:58.5503610Z"
        },
        "requestSizeInBytes": 18837,
        "requestBodySizeInBytes": 627
    },
    "TransportException": "A client transport error occurred: The connection failed. (Time: 2023-01-13T10:12:58.7636435Z, activity ID: 37a89a4d-9555-486e-8014-d38e493babe6, error code: ConnectionBroken [0x0012], base error: HRESULT 0x80131500, URI: rntbd://cdb-ms-prod-westeurope1-fd70.documents.azure.com:14007/apps/78b26377-fa10-497d-9d94-bf41b72ff220/services/10423294-eb5e-460a-9516-d890c9a11c8c/partitions/8b46f69d-d1af-42cf-887a-ad8f85898097/replicas/1331784691556
    36227p/, connection: 192.168.1.135:50509 -> 13.69.66.26:14007, payload sent: True)"
}

[ealsur]

ActivityId: 37a89a4d-9555-486e-8014-d38e493babe6, Windows/10.0.22621 cosmos-netstandard-sdk/3.29.4
 ---> Microsoft.Azure.Documents.GoneException: Message: The requested resource is no longer available at the server.
ActivityId: 37a89a4d-9555-486e-8014-d38e493babe6, Request URI: /apps/78b26377-fa10-497d-9d94-bf41b72ff220/services/10423294-eb5e-460a-9516-d890c9a11c8c/partitions/8b46f69d-d1af-42cf-887a-ad8f85898097/replicas/133178469155636226s/, RequestStats: , SDK: Windows/10.0.22621 cosmos-netstandard-sdk/3.29.4
 ---> Microsoft.Azure.Documents.TransportException: A client transport error occurred: The connection failed. (Time: 2023-01-13T10:13:28.0558878Z, activity ID: 37a89a4d-9555-486e-8014-d38e493babe6, error code: ConnectionBroken [0x0012], base error: HRESULT 0x80131500, URI: rntbd://cdb-ms-prod-westeurope1-fd70.documents.azure.com:14309/apps/78b26377-fa10-497d-9d94-bf41b72ff220/services/10423294-eb5e-460a-9516-d890c9a11c8c/partitions/8b46f69d-d1af-42cf-887a-ad8f85898097/replicas/133178469155636226s/, connection: 192.168.1.135:50554 -> 13.69.66.26:14309, payload sent: True)
 ---> Microsoft.Azure.Documents.TransportException: A client transport error occurred: The remote system closed the connection. (Time: 2023-01-13T10:13:27.9799108Z, activity ID: 00000000-0000-0000-0000-000000000000, error code: ReceiveStreamClosed [0x0011], base error: HRESULT 0x80131500, URI: rntbd://cdb-ms-prod-westeurope1-fd70.documents.azure.com:14309/, connection: 192.168.1.135:50554 -> 13.69.66.26:14309, payload sent: True)

{
                "event": "Transit Time",
                "startTimeUtc": "2023-01-13T10:12:58.5604724Z",
                "durationInMs": 285.7278
            },
, {
                "event": "Failed",
                "startTimeUtc": "2023-01-13T10:12:58.8462002Z",
                "durationInMs": 0
            }

This is a connectivity problem on TCP. It is saying that the TCP connection is being broken or terminated. It is also showing that the SDK being used is 3.29.4, not 3.31.2. At this point, there is nothing related to the auth mechanism, it's the TCP request experiencing connectivity issues.

The Diagnostics the only PII they might contain is the name of your account, which you can remove/scrape if desired.

• How often is this happening? Is it consistent? Can you share a repro?
• What is the environment? Is it a Debug session locally?
• Have you checked https://docs.microsoft.com/en-us/azure/cosmos-db/troubleshoot-service-unavailable ?

[jformacek]

Hello, Thanks for reply. So... I started collecting data for answer and suddenly it started working... Same network, same machine. I spent few hours on it yesterday on completely different network, then few hours today...

If it still works tomorrow, then it must really have been some connectivity issue, even when I did not see any connections in SYN_SENT state when ReadNextAsync() was running before throwing exception.

• RE SDK version: I noticed 3.29.4 in report as well. That's why I shared package reference from project. Also now checked version of Microsoft.Azure.Cosmos.Client.dll in project publishing dir, and it's indeed 3.31.2
• RE connectivity: The same behavior on my physical laptop and on my Azure VM. When command was running, before exception was thrown, I checked netstat -ano -p tcp and did not see any connections in SYN_SENT state. Client with connection string instead of TokenCredential works fine. That's why I believe it isn't connectivity. Maybe you can advise better way how to diagnose?
• How often is this happening? Is it consistent? Can you share a repro?
  • so far always happened. How to share repro? I can create simple console app reaching for the same data the same way. I do not want to publicly reveal complete stack traces and diagnostics - would it be possible to upload to private storage?
• What is the environment? Is it a Debug session locally?
  • yes, it's debug session in VS2022 and Azure Functions Core Tools 4.0.4915
• Have you checked https://docs.microsoft.com/en-us/azure/cosmos-db/troubleshoot-service-unavailable ?
  • yes, I did. I believe ports are open

[ealsur]

So... I started collecting data for answer and suddenly it started working... Same network, same machine. I spent few hours on it yesterday on completely different network, then few hours today...

If the problem would be on the SDK code or somehow connected to how the TokenCredential information is sent, then this would not happen. If the code is exactly the same, working against the same account, and the problem is the TokenCredential information, it should consistently fail, it would not even work on the first request. If it starts to work, it seems to point to an environmental condition that is generating these errors, what could be the variable between the moment it works and the moment it does not.

RE SDK version: I noticed 3.29.4 in report as well. That's why I shared package reference from project. Also now checked version of Microsoft.Azure.Cosmos.Client.dll in project publishing dir, and it's indeed 3.31.2

I don't know how the application is running, but the version of the SDK is what the loaded DLL has. I cannot say why it is 3.29.4 because that would depend on the application and runtime itself, but for the case of those errors, 3.29.4 was loaded and not 3.31.2. 3.31.2 simply would not print out 3.29.4.

RE connectivity: The same behavior on my physical laptop and on my Azure VM. When command was running, before exception was thrown, I checked netstat -ano -p tcp and did not see any connections in SYN_SENT state. Client with connection string instead of TokenCredential works fine. That's why I believe it isn't connectivity. Maybe you can advise better way how to diagnose?

Did you check for CPU on the machines? SNAT port exhaustion? Is the Function environment reaching the limit of open connections?

would it be possible to upload to private storage?

If you don't feel comfortable sharing diagnostics here, you can email them to maquaran{at}microsoft.com

[jformacek]

Seems that it still works fine after running for few days and I cannot repro anymore - so most likely intermittent issue. Let's close now; I will reopen if I see it again. Thanks for help!

[dinoo]

@ealsur

I am seeing similar error, the call to ReadNextAsync returns the first 1000 items, but on the second attempt it fails with similar failure as described in this issue.

I am using direct connection and DefaultAzureCredentials, when changing to using a connection string, no error is reported on subsequent calls of ReadNextAsync.

CPU is below 20%, so that shouldn't be any issue.

Environment summary I am running this simple console app example from my local machine.

<PackageReference Include="Azure.Identity" Version="1.12.0" />
<PackageReference Include="Microsoft.Azure.Cosmos" Version="3.42.0" />

Visual Studio 2022 .NET 8.0 Windows 10

How to Reproduce

Autoscale: 1000 RU Items: ~5 M Logical partitions: ~60k

The data is quite evenly distributed over the partitions.

Query is performed to get all partition keys, the goal of original program, on which the sample is based, is to clean up items with a certain value over all logical partitions.

About: +/- 16k keys should get returned, but it fails on the second ReadNextAsync call.

using Azure.Identity;
using Microsoft.Azure.Cosmos;
using Newtonsoft.Json.Linq;

const string cosmosAccountName = "https://<accountname>.documents.azure.com:443/";

const string DbName = "<dbName>";

var client = new CosmosClient(cosmosAccountName, new AzureCliCredential());

var query = $"SELECT DISTINCT c.partitionkey FROM c WHERE c.prop = @id";

var container = client.GetContainer(DbName, "<collectionName>");

var queryDefinition = new QueryDefinition(query)
    .WithParameter("@id", "03fb7ba5-2686-4674-8c57-fb737efe1ae5");

var feedIter = container.GetItemQueryIterator<JObject>(queryDefinition, requestOptions: new QueryRequestOptions
{
    MaxConcurrency = 1
});

while (feedIter.HasMoreResults)
{
    var next = await feedIter.ReadNextAsync();

    if (next.Count == 0)
    {
        continue;
    }

    var keys = next.Select(j => j["partitionkey"].Value<string>());

    await DoSomeAction(keys);
}

async Task DoSomeAction(IEnumerable<string> keys)
{
    await Task.Delay(1000);
}

Logs

DocDBTrace Information: 0 : [RNTBD Connection 54a9ecfc-0084-41e0-a41f-ca3d21dae93e] RNTBD connection established 10.99.0.2:54717 -> 10.100.48.20:5861
DocDBTrace Information: 0 : [RNTBD Channel eba0cde0-09c3-4f7a-adb4-2c56b6fe24a5] Disposing RNTBD Channel 10.99.0.2:54680 -> 10.100.48.20:5861
DocDBTrace Information: 0 : [RNTBD Dispatcher eba0cde0-09c3-4f7a-adb4-2c56b6fe24a5] Disposing RNTBD Dispatcher 10.99.0.2:54680 -> 10.100.48.20:5861
DocDBTrace Information: 0 : [RNTBD Connection eba0cde0-09c3-4f7a-adb4-2c56b6fe24a5] Disposing RNTBD connection 10.99.0.2:54680 -> 10.100.48.20:5861 to server rntbd://<accountName>-westeurope.documents.azure.com:5861/. Last send attempt time: 2024-08-21T11:10:15.1957335Z. Last send time: 2024-08-21T11:10:15.1966346Z. Last receive time: 2024-08-21T11:10:15.0945360Z. First sends since last receieve: 2024-08-21T11:10:15.1966346Z. # of sends since last receive: 1
DocDBTrace Information: 0 : [RNTBD Dispatcher eba0cde0-09c3-4f7a-adb4-2c56b6fe24a5] RNTBD Dispatcher <disconnected> 10.99.0.2:54680 -> 10.100.48.20:5861 is disposed
DocDBTrace Information: 0 : Closed unhealthy channel <disconnected> 10.99.0.2:54680 -> 10.100.48.20:5861
DocDBTrace Information: 0 : [RNTBD Channel 67aa2aa3-b6ba-4a57-9472-b83ba85071fb] Disposing RNTBD Channel 10.99.0.2:54701 -> 10.100.48.20:5861
DocDBTrace Information: 0 : [RNTBD Dispatcher 67aa2aa3-b6ba-4a57-9472-b83ba85071fb] Disposing RNTBD Dispatcher 10.99.0.2:54701 -> 10.100.48.20:5861
DocDBTrace Information: 0 : [RNTBD Connection 67aa2aa3-b6ba-4a57-9472-b83ba85071fb] Disposing RNTBD connection 10.99.0.2:54701 -> 10.100.48.20:5861 to server rntbd://<accountName>-westeurope.documents.azure.com:5861/. Last send attempt time: 2024-08-21T11:10:18.2250392Z. Last send time: 2024-08-21T11:10:18.2272284Z. Last receive time: 2024-08-21T11:10:18.2238322Z. First sends since last receieve: 2024-08-21T11:10:18.2272284Z. # of sends since last receive: 1
DocDBTrace Information: 0 : [RNTBD Dispatcher 67aa2aa3-b6ba-4a57-9472-b83ba85071fb] RNTBD Dispatcher <disconnected> 10.99.0.2:54701 -> 10.100.48.20:5861 is disposed
DocDBTrace Information: 0 : Closed unhealthy channel <disconnected> 10.99.0.2:54701 -> 10.100.48.20:5861
DocDBTrace Information: 0 : [RNTBD Channel 54a9ecfc-0084-41e0-a41f-ca3d21dae93e] Awaiting RNTBD channel initialization. Request URI: rntbd://<accountName>-westeurope.documents.azure.com:5861/apps/91785045-30d0-4f73-ba5d-2b80e1b9f782/services/e03a97a6-c61f-414f-8def-1eaaa0712194/partitions/aee2db5f-ec9c-429d-921f-b7da796a5ab2/replicas/133654305069572866s/
DocDBTrace Information: 0 : [RNTBD Connection 54a9ecfc-0084-41e0-a41f-ca3d21dae93e] SSL handshake complete 10.99.0.2:54717 -> 10.100.48.20:5861

DocDBTrace Error: 0 : [RNTBD Connection 54a9ecfc-0084-41e0-a41f-ca3d21dae93e] Reached end of stream. Read 0 bytes while reading header on connection 10.99.0.2:54717 -> 10.100.48.20:5861. Last send attempt time: 2024-08-21T11:10:20.7465719Z. Last send time: 2024-08-21T11:10:20.7482861Z. Last receive time: 2024-08-21T11:10:20.7450592Z. First sends since last receieve: 2024-08-21T11:10:20.7482861Z. # of sends since last receive: 1
DocDBTrace Information: 0 : RequestAsync failed: RID: dbs/<dbName>/colls/<collectionName>, Resource Type: Document, Op: (operationType: Query, resourceType: Document), Address: rntbd://<accountName>-westeurope.documents.azure.com:5861/apps/91785045-30d0-4f73-ba5d-2b80e1b9f782/services/e03a97a6-c61f-414f-8def-1eaaa0712194/partitions/aee2db5f-ec9c-429d-921f-b7da796a5ab2/replicas/133654305069572866s/, Exception: Microsoft.Azure.Documents.TransportException: A client transport error occurred: The connection failed. (Time: 2024-08-21T11:10:20.8786162Z, activity ID: b9086588-7f56-4405-88be-9304f4ed5de3, error code: ConnectionBroken [0x0012], base error: HRESULT 0x80131500, URI: rntbd://<accountName>-westeurope.documents.azure.com:5861/apps/91785045-30d0-4f73-ba5d-2b80e1b9f782/services/e03a97a6-c61f-414f-8def-1eaaa0712194/partitions/aee2db5f-ec9c-429d-921f-b7da796a5ab2/replicas/133654305069572866s/, connection: 10.99.0.2:54717 -> 10.100.48.20:5861, payload sent: True)
 ---> Microsoft.Azure.Documents.TransportException: A client transport error occurred: The remote system closed the connection. (Time: 2024-08-21T11:10:20.8777288Z, activity ID: 00000000-0000-0000-0000-000000000000, error code: ReceiveStreamClosed [0x0011], base error: HRESULT 0x80131500, URI: rntbd://<accountName>-westeurope.documents.azure.com:5861/, connection: 10.99.0.2:54717 -> 10.100.48.20:5861, payload sent: True)
   at Microsoft.Azure.Documents.Rntbd.Connection.TraceAndThrowEndOfStream(String type, ChannelCommonArguments args)
   at Microsoft.Azure.Documents.Rntbd.Connection.ReadPayloadAsync(Byte[] payload, Int32 length, String type, ChannelCommonArguments args)
   at Microsoft.Azure.Documents.Rntbd.Connection.ReadResponseMetadataAsync(ChannelCommonArguments args)
   at Microsoft.Azure.Documents.Rntbd.Dispatcher.ReceiveLoopAsync()
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Documents.Rntbd.Dispatcher.CallAsync(ChannelCallArguments args, TransportRequestStats transportRequestStats)
   at Microsoft.Azure.Documents.Rntbd.Channel.RequestAsync(DocumentServiceRequest request, TransportAddressUri physicalAddress, ResourceOperation resourceOperation, Guid activityId, TransportRequestStats transportRequestStats)
   at Microsoft.Azure.Documents.Rntbd.LoadBalancingPartition.RequestAsync(DocumentServiceRequest request, TransportAddressUri physicalAddress, ResourceOperation resourceOperation, Guid activityId, TransportRequestStats transportRequestStats)
   at Microsoft.Azure.Documents.Rntbd.TransportClient.InvokeStoreAsync(TransportAddressUri physicalAddress, ResourceOperation resourceOperation, DocumentServiceRequest request)
DocDBTrace Information: 0 : Converting to Gone (read-only request)
DocDBTrace Information: 0 : StoreReader.ReadMultipleReplicasInternalAsync exception thrown: Exception: Microsoft.Azure.Documents.GoneException: Message: The requested resource is no longer available at the server.
ActivityId: b9086588-7f56-4405-88be-9304f4ed5de3, Request URI: /apps/91785045-30d0-4f73-ba5d-2b80e1b9f782/services/e03a97a6-c61f-414f-8def-1eaaa0712194/partitions/aee2db5f-ec9c-429d-921f-b7da796a5ab2/replicas/133654305069572866s/, RequestStats: , SDK: Windows/10.0.19045 cosmos-netstandard-sdk/3.34.4
 ---> Microsoft.Azure.Documents.TransportException: A client transport error occurred: The connection failed. (Time: 2024-08-21T11:10:20.8786162Z, activity ID: b9086588-7f56-4405-88be-9304f4ed5de3, error code: ConnectionBroken [0x0012], base error: HRESULT 0x80131500, URI: rntbd://<accountName>-westeurope.documents.azure.com:5861/apps/91785045-30d0-4f73-ba5d-2b80e1b9f782/services/e03a97a6-c61f-414f-8def-1eaaa0712194/partitions/aee2db5f-ec9c-429d-921f-b7da796a5ab2/replicas/133654305069572866s/, connection: 10.99.0.2:54717 -> 10.100.48.20:5861, payload sent: True)
 ---> Microsoft.Azure.Documents.TransportException: A client transport error occurred: The remote system closed the connection. (Time: 2024-08-21T11:10:20.8777288Z, activity ID: 00000000-0000-0000-0000-000000000000, error code: ReceiveStreamClosed [0x0011], base error: HRESULT 0x80131500, URI: rntbd://<accountName>-westeurope.documents.azure.com:5861/, connection: 10.99.0.2:54717 -> 10.100.48.20:5861, payload sent: True)
   at Microsoft.Azure.Documents.Rntbd.Connection.TraceAndThrowEndOfStream(String type, ChannelCommonArguments args)
   at Microsoft.Azure.Documents.Rntbd.Connection.ReadPayloadAsync(Byte[] payload, Int32 length, String type, ChannelCommonArguments args)
   at Microsoft.Azure.Documents.Rntbd.Connection.ReadResponseMetadataAsync(ChannelCommonArguments args)
   at Microsoft.Azure.Documents.Rntbd.Dispatcher.ReceiveLoopAsync()
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Documents.Rntbd.Dispatcher.CallAsync(ChannelCallArguments args, TransportRequestStats transportRequestStats)
   at Microsoft.Azure.Documents.Rntbd.Channel.RequestAsync(DocumentServiceRequest request, TransportAddressUri physicalAddress, ResourceOperation resourceOperation, Guid activityId, TransportRequestStats transportRequestStats)
   at Microsoft.Azure.Documents.Rntbd.LoadBalancingPartition.RequestAsync(DocumentServiceRequest request, TransportAddressUri physicalAddress, ResourceOperation resourceOperation, Guid activityId, TransportRequestStats transportRequestStats)
   at Microsoft.Azure.Documents.Rntbd.TransportClient.InvokeStoreAsync(TransportAddressUri physicalAddress, ResourceOperation resourceOperation, DocumentServiceRequest request)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Documents.Rntbd.TransportClient.InvokeStoreAsync(TransportAddressUri physicalAddress, ResourceOperation resourceOperation, DocumentServiceRequest request)
   at Microsoft.Azure.Documents.StoreReader.CompleteActivity(Task`1 task, QueryRequestPerformanceActivity activity)
   at Microsoft.Azure.Documents.StoreReader.ReadFromStoreAsync(TransportAddressUri physicalAddress, DocumentServiceRequest request)
   at Microsoft.Azure.Documents.StoreReader.ReadMultipleReplicasInternalAsync(DocumentServiceRequest entity, Boolean includePrimary, Int32 replicaCountToRead, Boolean requiresValidLsn, Boolean useSessionToken, ReadMode readMode, Boolean checkMinLSN, Boolean forceReadAll)
DocDBTrace Information: 0 : Could not get quorum number of responses. ValidResponsesReceived: 0 ResponsesExpected: 1, ResolvedAddressCount: 4, ResponsesString: 
DocDBTrace Error: 0 : Received GoneException after backoff/retry. Will fail the request. The requested resource is no longer available at the server.
ActivityId: b9086588-7f56-4405-88be-9304f4ed5de3, Windows/10.0.19045 cosmos-netstandard-sdk/3.34.4
DocDBTrace Error: 0 : DocumentClientException with status code ServiceUnavailable, message: The request failed because the client was unable to establish connections to 4 endpoints across 1 regions. Please check for client resource starvation issues and verify connectivity between client and server. More info: https://aka.ms/cosmosdb-tsg-service-unavailable, inner exception: Microsoft.Azure.Documents.GoneException: The requested resource is no longer available at the server.
ActivityId: b9086588-7f56-4405-88be-9304f4ed5de3, Windows/10.0.19045 cosmos-netstandard-sdk/3.34.4
 ---> Microsoft.Azure.Documents.GoneException: Message: The requested resource is no longer available at the server.
ActivityId: b9086588-7f56-4405-88be-9304f4ed5de3, Request URI: /apps/91785045-30d0-4f73-ba5d-2b80e1b9f782/services/e03a97a6-c61f-414f-8def-1eaaa0712194/partitions/aee2db5f-ec9c-429d-921f-b7da796a5ab2/replicas/133654305069572866s/, RequestStats: , SDK: Windows/10.0.19045 cosmos-netstandard-sdk/3.34.4
 ---> Microsoft.Azure.Documents.TransportException: A client transport error occurred: The connection failed. (Time: 2024-08-21T11:10:20.8786162Z, activity ID: b9086588-7f56-4405-88be-9304f4ed5de3, error code: ConnectionBroken [0x0012], base error: HRESULT 0x80131500, URI: rntbd://<accountName>-westeurope.documents.azure.com:5861/apps/91785045-30d0-4f73-ba5d-2b80e1b9f782/services/e03a97a6-c61f-414f-8def-1eaaa0712194/partitions/aee2db5f-ec9c-429d-921f-b7da796a5ab2/replicas/133654305069572866s/, connection: 10.99.0.2:54717 -> 10.100.48.20:5861, payload sent: True)
 ---> Microsoft.Azure.Documents.TransportException: A client transport error occurred: The remote system closed the connection. (Time: 2024-08-21T11:10:20.8777288Z, activity ID: 00000000-0000-0000-0000-000000000000, error code: ReceiveStreamClosed [0x0011], base error: HRESULT 0x80131500, URI: rntbd://<accountName>-westeurope.documents.azure.com:5861/, connection: 10.99.0.2:54717 -> 10.100.48.20:5861, payload sent: True)
   at Microsoft.Azure.Documents.Rntbd.Connection.TraceAndThrowEndOfStream(String type, ChannelCommonArguments args)
   at Microsoft.Azure.Documents.Rntbd.Connection.ReadPayloadAsync(Byte[] payload, Int32 length, String type, ChannelCommonArguments args)
   at Microsoft.Azure.Documents.Rntbd.Connection.ReadResponseMetadataAsync(ChannelCommonArguments args)
   at Microsoft.Azure.Documents.Rntbd.Dispatcher.ReceiveLoopAsync()
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Documents.Rntbd.Dispatcher.CallAsync(ChannelCallArguments args, TransportRequestStats transportRequestStats)
   at Microsoft.Azure.Documents.Rntbd.Channel.RequestAsync(DocumentServiceRequest request, TransportAddressUri physicalAddress, ResourceOperation resourceOperation, Guid activityId, TransportRequestStats transportRequestStats)
   at Microsoft.Azure.Documents.Rntbd.LoadBalancingPartition.RequestAsync(DocumentServiceRequest request, TransportAddressUri physicalAddress, ResourceOperation resourceOperation, Guid activityId, TransportRequestStats transportRequestStats)
   at Microsoft.Azure.Documents.Rntbd.TransportClient.InvokeStoreAsync(TransportAddressUri physicalAddress, ResourceOperation resourceOperation, DocumentServiceRequest request)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Documents.Rntbd.TransportClient.InvokeStoreAsync(TransportAddressUri physicalAddress, ResourceOperation resourceOperation, DocumentServiceRequest request)
   at Microsoft.Azure.Documents.StoreReader.CompleteActivity(Task`1 task, QueryRequestPerformanceActivity activity)
   at Microsoft.Azure.Documents.StoreReader.ReadFromStoreAsync(TransportAddressUri physicalAddress, DocumentServiceRequest request)
   at Microsoft.Azure.Documents.StoreReader.ReadMultipleReplicasInternalAsync(DocumentServiceRequest entity, Boolean includePrimary, Int32 replicaCountToRead, Boolean requiresValidLsn, Boolean useSessionToken, ReadMode readMode, Boolean checkMinLSN, Boolean forceReadAll)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Documents.StoreReader.ReadMultipleReplicasInternalAsync(DocumentServiceRequest entity, Boolean includePrimary, Int32 replicaCountToRead, Boolean requiresValidLsn, Boolean useSessionToken, ReadMode readMode, Boolean checkMinLSN, Boolean forceReadAll)
   at Microsoft.Azure.Documents.StoreReader.ReadMultipleReplicaAsync(DocumentServiceRequest entity, Boolean includePrimary, Int32 replicaCountToRead, Boolean requiresValidLsn, Boolean useSessionToken, ReadMode readMode, Boolean checkMinLSN, Boolean forceReadAll)
   at Microsoft.Azure.Documents.ConsistencyReader.ReadSessionAsync(DocumentServiceRequest entity, ReadMode readMode)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetryAsync[TParam,TPolicy](Func`1 callbackMethod, Func`3 callbackMethodWithParam, Func`2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy`1 retryPolicyWithArg, Func`1 inBackoffAlternateCallbackMethod, Func`2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback)
   at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetryAsync[TParam,TPolicy](Func`1 callbackMethod, Func`3 callbackMethodWithParam, Func`2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy`1 retryPolicyWithArg, Func`1 inBackoffAlternateCallbackMethod, Func`2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback)
   at Microsoft.Azure.Documents.BackoffRetryUtility`1.ExecuteRetryAsync[TParam,TPolicy](Func`1 callbackMethod, Func`3 callbackMethodWithParam, Func`2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy`1 retryPolicyWithArg, Func`1 inBackoffAlternateCallbackMethod, Func`2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action`1 preRetryCallback)
   at Microsoft.Azure.Documents.ReplicatedResourceClient.<>c__DisplayClass32_0.<<InvokeAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func`1 executeAsync, Func`1 prepareRequest, IRequestRetryPolicy`2 policy, CancellationToken cancellationToken, Func`1 inBackoffAlternateCallbackMethod, Nullable`1 minBackoffForInBackoffCallback), and response headers: null
DocDBTrace Warning: 0 : ClientRetryPolicy: ServiceUnavailable. Refresh cache and retry. Failed Location: https://<accountName>-westeurope.documents.azure.com/; ResourceAddress: dbs/<dbName>/colls/<collectionName>
DocDBTrace Information: 0 : ClientRetryPolicy: ShouldRetryOnServiceUnavailable() Not retrying. No other regions available for the request. AvailablePreferredLocations = 0.
Exception thrown: 'Microsoft.Azure.Cosmos.CosmosException' in System.Private.CoreLib.dll
An exception of type 'Microsoft.Azure.Cosmos.CosmosException' occurred in System.Private.CoreLib.dll but was not handled in user code

[dinoo]

@ealsur when I try to connect using Gateway mode, I get a bad request

DocDBTrace Error: 0 : DocumentClientException with status code BadRequest, message: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Bad Request</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Bad Request - Request Too Long</h2>
<hr><p>HTTP Error 400. The size of the request headers is too long.</p>
</BODY></HTML>

RequestUri: https://<accountName>-westeurope.documents.azure.com/dbs/<dbName>/colls/<collectionName>/docs;
RequestMethod: POST;
Header: Authorization Length: 8669;
Header: x-ms-max-item-count Length: 4;
Header: x-ms-session-token Length: 13;
Header: x-ms-continuation Length: 11952;
Header: x-ms-documentdb-partitionkeyrangeid Length: 1;
Header: x-ms-documentdb-populatequerymetrics Length: 4;
Header: x-ms-cosmos-sdk-supportedcapabilities Length: 1;
Header: x-ms-cosmos-correlated-activityid Length: 36;
Header: x-ms-cosmos-supported-serialization-formats Length: 21;
Header: x-ms-cosmos-query-optimisticdirectexecute Length: 5;
Header: x-ms-documentdb-query-enablecrosspartition Length: 4;
Header: x-ms-documentdb-query-iscontinuationexpected Length: 5;
Header: x-ms-documentdb-query-parallelizecrosspartitionquery Length: 4;
Header: x-ms-documentdb-isquery Length: 4;
Header: x-ms-activity-id Length: 36;
Header: Cache-Control Length: 8;
Header: User-Agent Length: 78;
Header: x-ms-version Length: 10;
Header: Accept Length: 16;
, inner exception: null, and response headers: {
"Date": "Wed, 21 Aug 2024 12:01:19 GMT",
"Server": "Microsoft-HTTPAPI/2.0",
"x-ms-gatewayversion": "2.0.0",
}
Exception thrown: 'Microsoft.Azure.Cosmos.CosmosException' in System.Private.CoreLib.dll
An exception of type 'Microsoft.Azure.Cosmos.CosmosException' occurred in System.Private.CoreLib.dll but was not handled in user code

[ealsur]

@dinoo your first comment is a Service Unavailable exception. Unrelated to this thread. Please look at the exception aka.ms link and investigate using that.

The second one is an HTTP 400 coming from the size of your Continuation

Header: x-ms-continuation Length: 11952;

The continuation is too large, see related issues: https://github.com/Azure/azure-cosmos-dotnet-v3/issues/3791#issuecomment-1494694611

Otherwise create a new Issue.

[dinoo]

@ealsur, I checked the documentation on aka.ms link before I wrote my comment and checked all the basics, CPU starvation, et cetera. But all that looked fine.

The weird thing was that it occurred immediately after I called ReadNextAsync on the FeedIterator for the second time. This probably means that the server side cannot handle the continuation token that is being submitted.

The second part of your comment triggered me to try setting ResponseContinuationTokenLimitInKb to 1, basically disabling the continuation token. Using that setting, the subsequent requests in both Direct and Gateway mode when authenticated using DefaultAzureCredentials work as expected.

I compared continuation token between authentication method using DefaultAzureCredentials or connection string when using Direct connection. I don't see any obvious differences in size.

The part that popped out for me in the partial log that I posted is DocDBTrace Information: 0 : StoreReader.ReadMultipleReplicasInternalAsync exception thrown: Exception: Microsoft.Azure.Documents.GoneException: Message: The requested resource is no longer available at the server.

Is the continuation token somehow faulty and the state cannot be restored on the server side, when using DefaultAzureCredentials for authentication?

[ealsur]

Gateway mode uses HTTP, HTTP has limits in the size of the headers. The same size of Headers would work in Direct mode (uses TCP).

The error is due to the service closing the connection:

A client transport error occurred: The remote system closed the connection.

This doesn't seem to be related to the client SDK as the SDK is simply returning the Continuation Token to the user and then passing what you pass to the service.

I don't work on this repository anymore but this condition, particularly if it reproes consistently, seems to qualify for contacting support as service side debugging is outside of the scope of this repository.