UpsertItemAsync fails with 403 when using Item-scoped Resource Token

[briansboyd]

We are continuously addressing and improving the SDK, if possible, make sure the problem persist in the latest SDK version.

Using the latest, v3.35.4 as of Oct 11th, 2023.

Describe the bug I have created a Resource Token that is scoped to a single item, 'some_test_id_here' with a PermissionMode.All.

I am able to read the item using ReadItemAsync(...) but I receive a 403 when trying to save an update using UpsertItemAsync.

Notice that ReplaceItemAsync succeeds.

To Reproduce Steps to reproduce the behavior. If you can include code snippets or links to repositories containing a repro of the issue that can helps us in detecting the scenario it would speed up the resolution.

var cosmosClient = new CosmosClient($"https://{cosmosName}.documents.azure.com:443/", cosmosPrimaryKey);

var userResponse = await database.UpsertUserAsync($"some_integrationtestuser_id_here");
var user = userResponse.User;

var container = cosmosClient.GetContainer("TestAppsDB", "IntegrationTests");

var onePermissionProperties = new PermissionProperties(
    id: $"item_doc_all",
    permissionMode: PermissionMode.All,
    container: container,
    resourcePartitionKey: new PartitionKey("some_key_here"),
    itemId: "some_test_id_here");

var requestOptions = new RequestOptions() { Properties = new Dictionary<string, object>() { { "license", "test_license" } } };

var permissionResponse = await user.UpsertPermissionAsync(onePermissionProperties, 30 * 60, requestOptions);

var permissionResourceToken = permissionResponse.Resource;

// Access the resource token.
var resourceToken = permissionResourceToken.Token;

var cosmosClientToken = new CosmosClient($"https://{CosmosName}.documents.azure.com:443/", resourceToken);

var containerToken = cosmosClientToken.GetContainer("TestAppsDB", "IntegrationTests");

var itemResponse = await containerToken.ReadItemAsync<TestItemDTO>("some_test_id_here", new PartitionKey("some_key_here"));

var item = itemResponse.Resource;

var utcNow = DateTime.UtcNow;
item.DateTimeTestData = utcNow;
item.LastUpdatedDateUtc = utcNow;

var response = await containerToken.UpsertItemAsync<TestItemDTO>(item, new PartitionKey(partitionKeyValue)); // throws 403

Note that:

var response = await containerToken.ReplaceItemAsync<TestItemDTO>(item, item.Id, new PartitionKey(partitionKeyValue));

succeeds.

Expected behavior A clear and concise description of what you expected to happen.

I expect both the Upsert and Replace methods to succeed.

Actual behavior Provide a description of the actual behavior observed.

The Upsert fails with a 403 but the Replace succeeds.

Environment summary SDK Version: 3.35.4 OS Version (e.g. Windows, Linux, MacOSX) Windows 11, accessing Azure from Visual Studio 2022 Version 17.7.4

Additional context Add any other context about the problem here (for example, complete stack traces or logs).

ex.ToString() from Visual Studio:

Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: Forbidden (403); Substatus: 0; ActivityId: 7edc9991-deb6-4e2a-9d93-6ebd297a8d9e; Reason: (Message: {"Errors":["Request is blocked. Please check your authorization token and Cosmos DB account firewall settings. Learn more: https:\/\/aka.ms\/cosmosdb-tsg-forbidden"]} ActivityId: 7edc9991-deb6-4e2a-9d93-6ebd297a8d9e, Request URI: /apps/bf247f2c-4db2-47b8-9427-a2c6f88b4294/services/2c66950f-2993-42b9-8d2c-fc00d0f07cbc/partitions/9541f42f-ee45-4783-a92b-5d3522ae7b3b/replicas/133279548074872600p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.22621 cosmos-netstandard-sdk/3.31.4); at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker) at Microsoft.Azure.Documents.ConsistencyWriter.WritePrivateAsync(DocumentServiceRequest request, TimeoutHelper timeout, Boolean forceRefresh) at Microsoft.Azure.Documents.BackoffRetryUtility1.ExecuteRetryAsync[TParam,TPolicy](Func1 callbackMethod, Func3 callbackMethodWithParam, Func2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy1 retryPolicyWithArg, Func1 inBackoffAlternateCallbackMethod, Func2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action1 preRetryCallback) at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException) at Microsoft.Azure.Documents.BackoffRetryUtility1.ExecuteRetryAsync[TParam,TPolicy](Func1 callbackMethod, Func3 callbackMethodWithParam, Func2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy1 retryPolicyWithArg, Func1 inBackoffAlternateCallbackMethod, Func2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action1 preRetryCallback) at Microsoft.Azure.Documents.ConsistencyWriter.WriteAsync(DocumentServiceRequest entity, TimeoutHelper timeout, Boolean forceRefresh, CancellationToken cancellationToken) at Microsoft.Azure.Documents.ReplicatedResourceClient.<>cDisplayClass31_0.<b0>d.MoveNext() --- End of stack trace from previous location --- at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func1 executeAsync, Func1 prepareRequest, IRequestRetryPolicy2 policy, CancellationToken cancellationToken, Func1 inBackoffAlternateCallbackMethod, Nullable1 minBackoffForInBackoffCallback) at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException) at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func1 executeAsync, Func1 prepareRequest, IRequestRetryPolicy2 policy, CancellationToken cancellationToken, Func1 inBackoffAlternateCallbackMethod, Nullable1 minBackoffForInBackoffCallback) at Microsoft.Azure.Documents.StoreClient.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken, IRetryPolicy retryPolicy) at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken) at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken) --- Cosmos Diagnostics ---{"Summary":{"DirectCalls":{"(403, 0)":1}},"name":"UpsertItemAsync","start datetime":"2023-10-12T02:27:41.987Z","duration in milliseconds":216.358,"data":{"Synchronization Context":"Xunit.Sdk.MaxConcurrencySyncContext"},"children":[{"name":"ItemSerialize","duration in milliseconds":0.1317},{"name":"Microsoft.Azure.Cosmos.Handlers.RequestInvokerHandler","duration in milliseconds":204.8786,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.DiagnosticsHandler","duration in milliseconds":204.8132,"data":{"System Info":{"systemHistory":[{"dateUtc":"2023-10-12T02:27:32.7430337Z","cpu":0.644,"memory":50211780.000,"threadInfo":{"isThreadStarving":"no info","availableThreads":32765,"minThreads":24,"maxThreads":32767},"numberOfOpenTcpConnection":0}]}},"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RetryHandler","duration in milliseconds":204.8006,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.RouterHandler","duration in milliseconds":201.039,"children":[{"name":"Microsoft.Azure.Cosmos.Handlers.TransportHandler","duration in milliseconds":201.0356,"children":[{"name":"Microsoft.Azure.Documents.ServerStoreModel Transport Request","duration in milliseconds":198.7685,"data":{"Client Side Request Stats":{"Id":"AggregatedClientSideRequestStatistics","ContactedReplicas":[{"Count":1,"Uri":"rntbd://cdb-ms-prod-westus3-be34.documents.azure.com:14036/apps/bf247f2c-4db2-47b8-9427-a2c6f88b4294/services/2c66950f-2993-42b9-8d2c-fc00d0f07cbc/partitions/9541f42f-ee45-4783-a92b-5d3522ae7b3b/replicas/133279548074872600p/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-westus3-be34.documents.azure.com:14004/apps/bf247f2c-4db2-47b8-9427-a2c6f88b4294/services/2c66950f-2993-42b9-8d2c-fc00d0f07cbc/partitions/9541f42f-ee45-4783-a92b-5d3522ae7b3b/replicas/133279548074872601s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-westus3-be34.documents.azure.com:14068/apps/bf247f2c-4db2-47b8-9427-a2c6f88b4294/services/2c66950f-2993-42b9-8d2c-fc00d0f07cbc/partitions/9541f42f-ee45-4783-a92b-5d3522ae7b3b/replicas/133279548074872602s/"},{"Count":1,"Uri":"rntbd://cdb-ms-prod-westus3-be34.documents.azure.com:14372/apps/bf247f2c-4db2-47b8-9427-a2c6f88b4294/services/2c66950f-2993-42b9-8d2c-fc00d0f07cbc/partitions/9541f42f-ee45-4783-a92b-5d3522ae7b3b/replicas/133352245894561103s/"}],"RegionsContacted":[],"FailedReplicas":[],"AddressResolutionStatistics":[],"StoreResponseStatistics":[{"ResponseTimeUTC":"2023-10-12T02:27:42.1932061Z","ResourceType":"Document","OperationType":"Upsert","LocationEndpoint":"https://configuration-and-more-westus3.documents.azure.com/","StoreResult":{"ActivityId":"7edc9991-deb6-4e2a-9d93-6ebd297a8d9e","StatusCode":"Forbidden","SubStatusCode":"Unknown","LSN":1704,"PartitionKeyRangeId":null,"GlobalCommittedLSN":1704,"ItemLSN":-1,"UsingLocalLSN":false,"QuorumAckedLSN":1704,"SessionToken":"-1#1704","CurrentWriteQuorum":3,"CurrentReplicaSetSize":4,"NumberOfReadRegions":0,"IsValid":true,"StorePhysicalAddress":"rntbd://cdb-ms-prod-westus3-be34.documents.azure.com:14036/apps/bf247f2c-4db2-47b8-9427-a2c6f88b4294/services/2c66950f-2993-42b9-8d2c-fc00d0f07cbc/partitions/9541f42f-ee45-4783-a92b-5d3522ae7b3b/replicas/133279548074872600p/","RequestCharge":0,"RetryAfterInMs":null,"BELatencyInMs":"0.165","ReplicaHealthStatuses":["(port: 14036 | status: Connected | lkt: 10/12/2023 2:27:42 AM)"],"transportRequestTimeline":{"requestTimeline":[{"event": "Created", "startTimeUtc": "2023-10-12T02:27:41.9994527Z", "durationInMs": 0.014},{"event": "ChannelAcquisitionStarted", "startTimeUtc": "2023-10-12T02:27:41.9994667Z", "durationInMs": 142.6968},{"event": "Pipelined", "startTimeUtc": "2023-10-12T02:27:42.1421635Z", "durationInMs": 5.6566},{"event": "Transit Time", "startTimeUtc": "2023-10-12T02:27:42.1478201Z", "durationInMs": 27.7413},{"event": "Received", "startTimeUtc": "2023-10-12T02:27:42.1755614Z", "durationInMs": 1.6789},{"event": "Completed", "startTimeUtc": "2023-10-12T02:27:42.1772403Z", "durationInMs": 0}],"serviceEndpointStats":{"inflightRequests":1,"openConnections":1},"connectionStats":{"waitforConnectionInit":"True","callsPendingReceive":0,"lastSendAttempt":"2023-10-12T02:27:42.1136352Z","lastSend":"2023-10-12T02:27:42.1136896Z","lastReceive":"2023-10-12T02:27:42.1415342Z"},"requestSizeInBytes":1062,"requestBodySizeInBytes":304,"responseMetadataSizeInBytes":203,"responseBodySizeInBytes":166},"TransportException":null}}]},"AuthProvider LifeSpan InSec":3.4356119,"Point Operation Statistics":{"Id":"PointOperationStatistics","ActivityId":"7edc9991-deb6-4e2a-9d93-6ebd297a8d9e","ResponseTimeUtc":"2023-10-12T02:27:42.1976947Z","StatusCode":403,"SubStatusCode":0,"RequestCharge":0,"RequestUri":"dbs/TestAppsDB/colls/IntegrationTestsKey","ErrorMessage":"Microsoft.Azure.Documents.ForbiddenException: Message: {\"Errors\":[\"Request is blocked. Please check your authorization token and Cosmos DB account firewall settings. Learn more: https:\/\/aka.ms\/cosmosdb-tsg-forbidden\"]}\r\nActivityId: 7edc9991-deb6-4e2a-9d93-6ebd297a8d9e, Request URI: /apps/bf247f2c-4db2-47b8-9427-a2c6f88b4294/services/2c66950f-2993-42b9-8d2c-fc00d0f07cbc/partitions/9541f42f-ee45-4783-a92b-5d3522ae7b3b/replicas/133279548074872600p/, RequestStats: Microsoft.Azure.Cosmos.Tracing.TraceData.ClientSideRequestStatisticsTraceDatum, SDK: Windows/10.0.22621 cosmos-netstandard-sdk/3.31.4\r\n at Microsoft.Azure.Documents.StoreResult.ToResponse(RequestChargeTracker requestChargeTracker)\r\n at Microsoft.Azure.Documents.ConsistencyWriter.WritePrivateAsync(DocumentServiceRequest request, TimeoutHelper timeout, Boolean forceRefresh)\r\n at Microsoft.Azure.Documents.BackoffRetryUtility1.ExecuteRetryAsync[TParam,TPolicy](Func1 callbackMethod, Func3 callbackMethodWithParam, Func2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy1 retryPolicyWithArg, Func1 inBackoffAlternateCallbackMethod, Func2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action1 preRetryCallback)\r\n at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n at Microsoft.Azure.Documents.BackoffRetryUtility1.ExecuteRetryAsync[TParam,TPolicy](Func1 callbackMethod, Func3 callbackMethodWithParam, Func2 callbackMethodWithPolicy, TParam param, IRetryPolicy retryPolicy, IRetryPolicy1 retryPolicyWithArg, Func1 inBackoffAlternateCallbackMethod, Func2 inBackoffAlternateCallbackMethodWithPolicy, TimeSpan minBackoffForInBackoffCallback, CancellationToken cancellationToken, Action1 preRetryCallback)\r\n at Microsoft.Azure.Documents.ConsistencyWriter.WriteAsync(DocumentServiceRequest entity, TimeoutHelper timeout, Boolean forceRefresh, CancellationToken cancellationToken)\r\n at Microsoft.Azure.Documents.ReplicatedResourceClient.<>cDisplayClass31_0.<b0>d.MoveNext()\r\n--- End of stack trace from previous location ---\r\n at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func1 executeAsync, Func1 prepareRequest, IRequestRetryPolicy2 policy, CancellationToken cancellationToken, Func1 inBackoffAlternateCallbackMethod, Nullable1 minBackoffForInBackoffCallback)\r\n at Microsoft.Azure.Documents.ShouldRetryResult.ThrowIfDoneTrying(ExceptionDispatchInfo capturedException)\r\n at Microsoft.Azure.Documents.RequestRetryUtility.ProcessRequestAsync[TRequest,IRetriableResponse](Func1 executeAsync, Func1 prepareRequest, IRequestRetryPolicy2 policy, CancellationToken cancellationToken, Func1 inBackoffAlternateCallbackMethod, Nullable1 minBackoffForInBackoffCallback)\r\n at Microsoft.Azure.Documents.StoreClient.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken, IRetryPolicy retryPolicy)\r\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.ProcessMessageAsync(RequestMessage request, CancellationToken cancellationToken)\r\n at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)","RequestSessionToken":null,"ResponseSessionToken":"0:-1#1704","BELatencyInMs":"0.165"}}}]}]}]}]}]}]}

[ealsur]

@briansboyd SDK is not the component validating or generating the Resource Token, this is all happening on the Service side.

Having said that, we can verify at least with the provided repro code if this happens in another account. @NaluTripician can you check if you can repro this scenario pointing to the Emulator or another account?

[briansboyd]

Thanks @ealsur. To me, the resource token looks fine: I can read and write that single item using it. I've also verifid that I do not have access to other items within its partition.

The reason why I filed this ticket is that I suspect the SDK is trying to do something else besides read/write on this single item within the Upsert method. And it's that 'extra' something that is causing the exception.

Make sense?

[ealsur]

There is nothing else the SDK is doing, your attached Diagnostics show that the 403 is coming from the Upsert operation:

StoreResponseStatistics":[
{"ResponseTimeUTC":"2023-10-12T02:27:42.1932061Z",
"ResourceType":"Document",
"OperationType":"Upsert",
"LocationEndpoint":"https://<youraccount>.documents.azure.com/",
"StoreResult":{
    "ActivityId":"7edc9991-deb6-4e2a-9d93-6ebd297a8d9e",
    "StatusCode":"Forbidden",
    "SubStatusCode":"Unknown",
    "LSN":1704,
    "PartitionKeyRangeId":null,
    "GlobalCommittedLSN":1704,
    "ItemLSN":-1,
    "UsingLocalLSN":false,
    "QuorumAckedLSN":1704...

The Forbidden is tied to the ResourceType Document, OperationType Upsert.

The Summary also shows there is no other request the SDK is doing:

"Summary":{"DirectCalls":{"(403, 0)":1}}

[briansboyd]

Ah, gotcha, the Upsert operation is supported by Cosmos DB. So, this would conceivably be a bug in Cosmos DB itself? If so, can you please provide where I would go next? Thanks.

(Interested in the tool you used to 'prettify' that output: neither my JSON or XML tool would do anything with it.)

[ealsur]

@briansboyd Let us first try and repro with your code and see what we find. @NaluTripician can you please see if you can repro?

Regarding prettifying, I just used Notepad and pressed ENTER at the right places :D

[bartelink]

Regarding prettifying, I just used Notepad and pressed ENTER at the right places :D

Can you post a proposal for the next https://notepadconf.com/ please?

[NaluTripician]

I was able to reproduce the issue. For a work arround please remove the itemId parameter when creating PermissionProperties.

var onePermissionProperties = new PermissionProperties(
    id: $"item_doc_all",
    permissionMode: PermissionMode.All,
    container: container,
    resourcePartitionKey: new PartitionKey("some_key_here"));

[briansboyd]

But then the token won't be scoped to a specific item, a key requirement of scoping a resource token to a specific document.

The main idea behind using resource tokens is to provide a greater granularity of security.

I think that a better workaround would be to not use Upsert, but to use Replace instead.

Is there a Cosmos DB repo here on GitHub, or if I want to alert them to this, should I use the Azure Portal.

Thanks for your help clarifying this bug.

[NaluTripician]

Hi, yes I think the next steps would to be to create a support ticket. When creating a ticket please make sure that you specify that it is not an SDK issue but rather seems to be related to how the service validates the token when scoped to a single item so it can be routed to the correct team.

[briansboyd]

Thanks @NaluTripician !

[briansboyd]

Hi, yes I think the next steps would to be to create a support ticket. When creating a ticket please make sure that you specify that it is not an SDK issue but rather seems to be related to how the service validates the token when scoped to a single item so it can be routed to the correct team.

Actually @NaluTripician , I see no way to report this bug. I have no need for a Support plan but this seems to be the only way to let them know that this is a bug. Since I have a workaround, I can move forward, but really think that the Cosmos DB team should know about this problem.

[ealsur]

@NaluTripician Let's try to report this ourselves to the service folks