Deleting multiple partitions at once

[NoahStahl]

Does this feature accommodate simultaneous deletion jobs? In my scenario I delete sets of related partitions, perhaps 20 or 30. I'm assuming I could perform a background delete request for each partition in sequence, but ideally I could issue a request for a batch of deletions across a set of partition keys.

More conveniently in my case, I'd want to request deletion by partition key prefix, since my target partitions share the same prefix.

[jkonecki]

I don't think deleting partitions by prefix is supported but I've successfully issued multiple delete requests in parallel.

[WouterSlob]

I have the same case where multiple partitions with the same prefix have to be deleted.

In my case we have a collection with logging data with partitionkey = "partitionkey_weekNumber". We currently use TTL to clean this data anyway, but it would be nice to be able to delete it. Either by giving the prefix, or finding all partitions with a given prefix.

[dinoo]

I have done some load tests and used ParallelForEachAsync with maxDegreeOfParallelism of 12 to delete a fair amount of partitions about +/- 100 000 in our development environment.

There seems to be limit to how many delete partition range id requests can be handled simultaneously.

I am unsure when he started to complain, but I got the following error:

ActivityId: 0c75ff1d-726d-4c11-822d-6b3b30f5de0d, Request URI: /apps/...../services/..../partitions/...../replicas/132608902547087461p/, RequestStats:
RequestStartTime: 2021-03-31T16:34:06.9683051Z, RequestEndTime: 2021-03-31T16:34:06.9683051Z,  Number of regions attempted:1
ResponseTime: 2021-03-31T16:34:06.9683051Z, StoreResult: StorePhysicalAddress: rntbd://....-westeurope1-fd24.documents.azure.com:14086/apps/......./services/......./partitions/......./replicas/132608902547087461p/, LSN: 1277559, GlobalCommittedLsn: 1277559, PartitionKeyRangeId: , IsValid: True, StatusCode: 403, SubStatusCode: 0, RequestCharge: 0, ItemLSN: -1, SessionToken: -1#1277559, UsingLocalLSN: False, TransportException: null, ResourceType: PartitionKey, OperationType: Delete
, SDK: Microsoft.Azure.Documents.Common/2.11.0, Please see CosmosDiagnostics, Windows/10.0.19042 cosmos-netstandard-sdk/3.15.4); (Forbidden)
Partitionkey <partition-key-name> deletion unsuccesfull Response status code does not indicate success: Forbidden (403); Substatus: 0; ActivityId: 76d2321e-8f37-4be8-89c0-433c19c55a33; Reason: (Message: {"Errors":["Reached maximum limit for number of Partition Key deletes\/ id range deletes that can be processed. Please retry the request after sometime."]}

Would be nice to know what the limitations are

[jkonecki]

I concur that the limitiations should be published.

[ImrePyhvel]

TL;DR

• I don't see delete-partition-by-prefix feasible as single operation.
• Querying partition keys efficiently (incl by prefix) would help a lot.

Long version

Delete by partition key prefix

I don't see this as single operation at all. This would essentially still boil down to partition discovery and N delete operations which are distributed across multiple physical partitions. Imho such feature would most likely lead to either race conditions (bad) and/or locking, scalability hit (bad). I believe that's also the reason why we don't have such feature for deleting multiple documents or other transactions across partitions.

Even if backend found some tricks to do this, it would still be confusing on API level as well, since some operations could be accepted and some perhaps not. What should the HTTP/API response code and content be like? This would internally boil down to either the "Transactional" case or "Bulk" case below.

I think its better if this kind of logic is left to client app. If anyone wants to have a comfort-layer on client-side API they can easily build it themselves and have full control over race conditions between query and deletes, feedback, exception-handling, etc.

Transactional by explicit partition keys

Current transactional batch for document ops works within single partition key which makes it a local operation in single physical partition. I think getting transaction support across multiple partitions (like in a prefix case) would not be an easy thing to do without affecting general scalability.

Correct my if I'm wrong, but I don't see this happening..

Bulk by explicit partition keys

I haven't tested this myself, but if having transaction is not a requirement, and the goal is to just optimize http traffic, then most likely bulk support should work for partition deletes just like for other container operations?

Partition deletes are just 1 request per logical partition but since the link above mentions that batches are grouped by physical partition then it may still be feasible to use when deleting heaps of partitions in one go.

Iterative by explicit partition keys

There's always the fallback option to gather target partition keys and just send N separate partition deletes as provided by this preview. This is already waaaay better than what we have today in official release.

Query partition keys by prefix

This is what is think is the real blocker for this case: it is currently painful to discover partition keys. If DISTINCT query would use index, it would solve the "delete by prefix" issue for practical purposes (that is, without transaction). As it is "Started", then hopefully we'll see it soon: https://feedback.azure.com/forums/263030-azure-cosmos-db/suggestions/36349012-distinct-should-use-index-if-possible

Please do correct me if I'm fundamentally mistaken somewhere..