search

Azure / azure-cosmos-dotnet-v2

Contains samples and utilities relating to the Azure Cosmos DB .NET SDK
MIT License
577 stars 837 forks source link

FeedOptions discarded when using aggregates (x64) #543

Open Predictor opened 6 years ago

Predictor commented 6 years ago

Result of calling query.Count() and query.AsEnumerable().Count() is different because in first case FeedOptions are discarded and thereby PartitionKey is ignored. This behavior is inconsistent. See the repro below.

        [TestMethod]
        public async Task TestAggregateRespectsFeedOptions()
        {
            var endpoint = new Uri("***");
            var key = "***";
            var client = new DocumentClient(endpoint, key);
            Database db = await client.CreateDatabaseIfNotExistsAsync(new Database { Id = "aggregateTestDb" });
            DocumentCollection collection = await client.CreateDocumentCollectionIfNotExistsAsync(db.SelfLink, new DocumentCollection { Id = "aggregateTestCollection", PartitionKey = new PartitionKeyDefinition { Paths = new Collection<string> { "/p" } } });
            var entries = Enumerable.Repeat(1, 10).Select(s => new { p = Guid.NewGuid().ToString(), id = Guid.NewGuid().ToString() }).ToArray();
            foreach (var entry in entries)
            {
                await client.CreateDocumentAsync(collection.SelfLink, entry);
            }

            var feedOptions = new FeedOptions { PartitionKey = new PartitionKey(entries.First().p), EnableCrossPartitionQuery = false};
            var query = client.CreateDocumentQuery(collection.SelfLink, feedOptions).AsQueryable();

            var aggregateCount = query.Count();
            var directCount = query.AsEnumerable().Count();
            Assert.AreEqual(directCount, aggregateCount); // Assert.AreEqual failed. Expected:<1>. Actual:<10>. 
        }
ealsur commented 6 years ago

Thanks @Predictor, I'll investigate.

ealsur commented 6 years ago

@Predictor are you using the SDK version 1.22?

I created a sample project here: https://github.com/ealsur/consoledocdbsample and it seems to work as expected with that SDK version.

Predictor commented 6 years ago

@ealsur Thanks for your reply. I tested it with SDK 1.22 and result is still the same. The only difference with your sample is that I don't use emulator but a real Cosmos DB account in Azure. The code doesn't work in emulator for me, it fails to create the collection with "unknown server error". I used emulator from https://aka.ms/cosmosdb-emulator Collection creation fails in web interface as well. Can you check your sample with a real Cosmos DB?

{"code":500,"body":"{\"code\":\"InternalServerError\",\"message\":\"Unknown server error occurred when processing this request.\\r\\nActivityId: 97f11da9-1ef4-4b5b-b3f7-0356b7a24c0b, Microsoft.Azure.Documents.Common/1.22.0.0\"}","activityId":"97f11da9-1ef4-4b5b-b3f7-0356b7a24c0b"}

ealsur commented 6 years ago

Double checked with a live account and it still works. What's the Default Consistency you are using?

ealsur commented 6 years ago

@Predictor If you take my repo and point it to your account, does it also fail?

Predictor commented 6 years ago

@ealsur Default consistency is STRONG. I took your code and it still reproduces. Here are Fiddler traces, 1st for query.Count(), 2nd for query.AsEnumerable(). As you can see, the first request doesn't contain the partition key, and the second one does.

POST https://***.documents.azure.com/dbs/XBJ-AA==/colls/XBJ-AOrV-AE=/docs HTTP/1.1
x-ms-continuation: 
x-ms-documentdb-isquery: True
x-ms-documentdb-query-enablecrosspartition: False
x-ms-max-item-count: 2147483647
x-ms-documentdb-populatequerymetrics: False
x-ms-documentdb-partitionkeyrangeid: XBJ-AOrV-AE=,0
x-ms-date: Mon, 16 Jul 2018 08:48:20 GMT
authorization: ***
Cache-Control: no-cache
x-ms-consistency-level: Strong
User-Agent: documentdb-dotnet-sdk/1.22.0 Host/64-bit MicrosoftWindowsNT/10.0.17134.0
x-ms-version: 2017-11-15
Accept: application/json
Content-Type: application/query+json
Host: ***.documents.azure.com
Content-Length: 60
Expect: 100-continue

{"query":"SELECT VALUE [{\"item\": Count(1)}]\r\nFROM root"}
HTTP/1.1 200 Ok
Cache-Control: no-store, no-cache
Pragma: no-cache
Content-Type: application/json
Server: Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000
x-ms-last-state-change-utc: Thu, 12 Jul 2018 06:10:12.438 GMT
x-ms-resource-quota: documentSize=10240;documentsSize=10485760;documentsCount=-1;collectionSize=10485760;
x-ms-resource-usage: documentSize=0;documentsSize=85;documentsCount=80;collectionSize=106;
lsn: 87
x-ms-item-count: 1
x-ms-schemaversion: 1.6
x-ms-alt-content-path: dbs/aggregateTestDb/colls/aggregateTestCollection
x-ms-xp-role: 2
x-ms-global-Committed-lsn: 87
x-ms-number-of-read-regions: 0
x-ms-transport-request-id: 345
x-ms-request-charge: 9.48
x-ms-serviceversion: version=2.0.0.0
x-ms-activity-id: 08cc63e7-45b1-4d28-a0ad-d280f14e6e04
x-ms-session-token: 0:87
x-ms-gatewayversion: version=2.0.0.0
Date: Mon, 16 Jul 2018 08:48:20 GMT
Content-Length: 62

{"_rid":"XBJ-AOrV-AE=","Documents":[[{"item":80}]],"_count":1}
GET https://***.documents.azure.com/dbs/XBJ-AA==/colls/XBJ-AOrV-AE=/docs HTTP/1.1
x-ms-continuation: 
x-ms-documentdb-isquery: True
x-ms-documentdb-query-enablecrosspartition: False
x-ms-documentdb-query-iscontinuationexpected: False
x-ms-documentdb-partitionkey: ["6d2a8931-02aa-4142-a897-9a00283aeb2b"]
x-ms-date: Mon, 16 Jul 2018 08:48:25 GMT
authorization: ***
Cache-Control: no-cache
x-ms-consistency-level: Strong
User-Agent: documentdb-dotnet-sdk/1.22.0 Host/64-bit MicrosoftWindowsNT/10.0.17134.0
x-ms-version: 2017-11-15
Accept: application/json
Host: ***.documents.azure.com

HTTP/1.1 200 Ok
Cache-Control: no-store, no-cache
Pragma: no-cache
Content-Type: application/json
Content-Location: https://***.documents.azure.com/dbs/XBJ-AA==/colls/XBJ-AOrV-AE=/docs
Server: Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000
x-ms-last-state-change-utc: Thu, 12 Jul 2018 06:10:12.438 GMT
x-ms-resource-quota: documentSize=10240;documentsSize=10485760;documentsCount=-1;collectionSize=10485760;
x-ms-resource-usage: documentSize=0;documentsSize=85;documentsCount=80;collectionSize=106;
lsn: 87
x-ms-item-count: 1
x-ms-schemaversion: 1.6
x-ms-alt-content-path: dbs/aggregateTestDb/colls/aggregateTestCollection
x-ms-xp-role: 2
x-ms-global-Committed-lsn: 87
x-ms-number-of-read-regions: 0
x-ms-transport-request-id: 346
x-ms-request-charge: 5.9
x-ms-serviceversion: version=2.0.0.0
x-ms-activity-id: f303972a-dfac-445b-b867-34067562049f
x-ms-session-token: 0:87
x-ms-gatewayversion: version=2.0.0.0
Date: Mon, 16 Jul 2018 08:48:24 GMT
Content-Length: 349

{"_rid":"XBJ-AOrV-AE=","Documents":[{"p":"6d2a8931-02aa-4142-a897-9a00283aeb2b","id":"6f1bac5b-4f73-4157-b65d-b316dfb54220","_rid":"XBJ-AOrV-AFHAAAAAAAAAA==","_self":"dbs\/XBJ-AA==\/colls\/XBJ-AOrV-AE=\/docs\/XBJ-AOrV-AFHAAAAAAAAAA==\/","_etag":"\"390059a7-0000-0000-0000-5b4c5bcc0000\"","_attachments":"attachments\/","_ts":1531730892}],"_count":1}
ealsur commented 6 years ago

@Predictor In both your requests I see the Partition specification. In the POST, you have:

x-ms-documentdb-partitionkeyrangeid: XBJ-AOrV-AE=,0

And you have a result of 1 document.

In the GET, you have:

x-ms-documentdb-partitionkey: ["6d2a8931-02aa-4142-a897-9a00283aeb2b"]

And you also have a result of 1 document.

I recreated your same scenario, a live STRONG Account, and ran the code, I got a result of 1 in both requests, the direct and aggregate counts.

I really don't understand how you are getting 10 in the aggregate result. I updated my code sample with my real credentials, the ones I'm using.

I also double checked my Fiddler traces and:

  1. POST

    POST https://maquarangithub-southcentralus.documents.azure.com/dbs/mP8PAA==/colls/mP8PAM31WgA=/docs HTTP/1.1 x-ms-continuation: x-ms-documentdb-isquery: True x-ms-documentdb-query-enablecrosspartition: False x-ms-documentdb-query-iscontinuationexpected: False x-ms-documentdb-populatequerymetrics: False x-ms-documentdb-partitionkey: ["b4687596-1e50-4e9c-8dcf-6b48f29c2c28"] x-ms-date: Mon, 16 Jul 2018 11:15:43 GMT authorization: type%3dmaster%26ver%3d1.0%26sig%3dX9VPNGfkTHUPrgl%2fyiCq5%2b%2bETfPtpivF6DuFfX%2bl%2bYk%3d Cache-Control: no-cache x-ms-consistency-level: Strong User-Agent: documentdb-dotnet-sdk/1.22.0 Host/32-bit MicrosoftWindowsNT/6.2.9200.0 x-ms-version: 2017-11-15 Accept: application/json Content-Type: application/query+json Host: maquarangithub-southcentralus.documents.azure.com Content-Length: 44 Expect: 100-continue

    {"query":"SELECT VALUE Count(1) FROM root "}

Response: {"_rid":"mP8PAM31WgA=","Documents":[1],"_count":1}

  1. GET

    GET https://maquarangithub-southcentralus.documents.azure.com/dbs/mP8PAA==/colls/mP8PAM31WgA=/docs HTTP/1.1 x-ms-continuation: x-ms-documentdb-isquery: True x-ms-documentdb-query-enablecrosspartition: False x-ms-documentdb-query-iscontinuationexpected: False x-ms-documentdb-partitionkey: ["b4687596-1e50-4e9c-8dcf-6b48f29c2c28"] x-ms-date: Mon, 16 Jul 2018 11:15:43 GMT authorization: type%3dmaster%26ver%3d1.0%26sig%3dspVH3qyi0imxa7S12n%2bs9gF97XRqA1uzBwOgURVnmsk%3d Cache-Control: no-cache x-ms-consistency-level: Strong User-Agent: documentdb-dotnet-sdk/1.22.0 Host/32-bit MicrosoftWindowsNT/6.2.9200.0 x-ms-version: 2017-11-15 Accept: application/json Host: maquarangithub-southcentralus.documents.azure.com

Response: {"_rid":"mP8PAM31WgA=","Documents":[{"p":"b4687596-1e50-4e9c-8dcf-6b48f29c2c28","id":"9feec363-a552-4df7-a82e-9c8c464b0f43","_rid":"mP8PAM31WgAzAAAAAAAAAA==","_self":"dbs\/mP8PAA==\/colls\/mP8PAM31WgA=\/docs\/mP8PAM31WgAzAAAAAAAAAA==\/","_etag":"\"00006730-0000-0000-0000-5b4c7e5a0000\"","_attachments":"attachments\/","_ts":1531739738}],"_count":1}

Predictor commented 6 years ago

@ealsur Result of the POST request is not a document, it's a value of the query {"query":"SELECT **VALUE Count(1)** FROM root "}, it's COUNT query and result == 80 (it's not 10 because the collection already contains some garbage from previous test runs). {"_rid":"XBJ-AOrV-AE=","Documents":[[{"item":80}]],"_count":1} The issue is pretty obvious, there's no x-ms-documentdb-partitionkey header in my POST request, but I don't have any idea why the same client dll used in the same way produces a different HTTP request in your case. Is there a way to debug how the client constructs HTTP request?

ealsur commented 6 years ago

After talking offline, this indeed happens on x64 but not on Any CPU or x86.

vskh commented 5 years ago

Pinging this because it still happens with SDK 2.7. Additionally, from my testing it is not related to the platform of binary but instead it seems to be related to target framework. E.g.:

This is weird and annoying, took 2 days to debug that it's not actually my code but SDK. Also, suddenly enabling cross-partition queries (when I think they are disabled because I explicitly set EnableCrossPartitionQuery to false) can be expensive! Can this be looked into, please?

stephenschwan commented 4 years ago

I just ran into this when we updated our nuget (Microsoft.Azure.DocumentDb.Core) from 2.8.1 to 2.9.0. It was fixed in prior versions and it is now broken with 2.9.0.

dosegal commented 4 years ago

any update on this issue? This still happens in the latest stable release (2.10.1)

ChrisProlls commented 4 years ago

For us this issue appears after migrating .NET Core from 2.2 to 3.0 ! Without changing the Cosmos SDK (2.6.0)