search

microsoft / spring-data-cosmosdb

Access data with Azure Cosmos DB
MIT License
93 stars 68 forks source link

Pagination not sorted with strange behaviour: token is null #488

Closed lucprosa closed 4 years ago

lucprosa commented 4 years ago

Hi, I'm using spring-data-cosmosdb 2.2.2 with azure-cosmos 3.6.0 and Scala. I'm using CosmosTemplate to create custom DocumentQueries with pagination.

Now I'm facing with this strange error when doing pagination.

My POST request returned 5839 items within 6 pages (pagesize=1000).

For each request I'm getting the requestContinuation to use in the next page.

for page in nextPages:
    newUrl = urlAPI + '&page=' + str(page) + '&requestContinuation=' + str(encodeToken)
    print(newUrl)
    r = requests.post(newUrl, data=body, headers=header)
    results.extend(r.json()['content'])
    requestContinuation = r.json()['pageable']['requestContinuation']
    encodeToken = quote(str(requestContinuation))
    print('Pageable: ', r.json()['pageable'])

My output shows each request with url and pageable information.

The strange thing is about the "exec 3" that I'm getting a "null" token, but the next executions (exec 4 and 5) have token not null. Other crazy thing is that "pageSize" is not 1000 for "exec 3" but "692".

Is there a sorting issue here? (I'm not sorting my request, only paginate it).

exec 3
https://baseurl/api/documents/recent?&page=2&requestContinuation=%7B%22token%22%3A%22-RID%3A~I9lLALNXyUPqkgAAAAAAAA%3D%3D%23RT%3A2%23TRC%3A2000%23ISV%3A2%23IEO%3A65551%23QCF%3A1%22%2C%22range%22%3A%22%7B%5C%22min%5C%22%3A%5C%22%5C%22%2C%5C%22max%5C%22%3A%5C%2205C1DFFFFFFFFC%5C%22%2C%5C%22isMinInclusive%5C%22%3Atrue%2C%5C%22isMaxInclusive%5C%22%3Afalse%7D%22%7D
Pageable:  {'sort': {'sorted': False, 'unsorted': True, 'empty': True}, 'offset': 0, 'requestContinuation': '{"token":null,"range":"{\\"min\\":\\"05C1DFFFFFFFFC\\",\\"max\\":\\"FF\\",\\"isMinInclusive\\":true,\\"isMaxInclusive\\":false}"}', 'pageNumber': 2, 'pageSize': 692, 'unpaged': False, 'paged': True}

Full Output:

--------------------------------------
Total Elements:  5839
Total Pages:  6
Page Size:  1000
--------------------------------------

exec 1
https://baseurl/api/documents/recent?
Pageable:  {'sort': {'sorted': False, 'unsorted': True, 'empty': True}, 'offset': 0, 'requestContinuation': '{"token":"-RID:~I9lLALNXyUN6PAAAAAAAAA==#RT:1#TRC:1000#ISV:2#IEO:65551#QCF:1","range":"{\\"min\\":\\"\\",\\"max\\":\\"05C1DFFFFFFFFC\\",\\"isMinInclusive\\":true,\\"isMaxInclusive\\":false}"}', 'pageNumber': 0, 'pageSize': 1000, 'unpaged': False, 'paged': True}

exec 2
https://baseurl/api/documents/recent?&page=1&requestContinuation=%7B%22token%22%3A%22-RID%3A~I9lLALNXyUN6PAAAAAAAAA%3D%3D%23RT%3A1%23TRC%3A1000%23ISV%3A2%23IEO%3A65551%23QCF%3A1%22%2C%22range%22%3A%22%7B%5C%22min%5C%22%3A%5C%22%5C%22%2C%5C%22max%5C%22%3A%5C%2205C1DFFFFFFFFC%5C%22%2C%5C%22isMinInclusive%5C%22%3Atrue%2C%5C%22isMaxInclusive%5C%22%3Afalse%7D%22%7D
Pageable:  {'sort': {'sorted': False, 'unsorted': True, 'empty': True}, 'offset': 0, 'requestContinuation': '{"token":"-RID:~I9lLALNXyUPqkgAAAAAAAA==#RT:2#TRC:2000#ISV:2#IEO:65551#QCF:1","range":"{\\"min\\":\\"\\",\\"max\\":\\"05C1DFFFFFFFFC\\",\\"isMinInclusive\\":true,\\"isMaxInclusive\\":false}"}', 'pageNumber': 1, 'pageSize': 1000, 'unpaged': False, 'paged': True}

exec 3
https://baseurl/api/documents/recent?&page=2&requestContinuation=%7B%22token%22%3A%22-RID%3A~I9lLALNXyUPqkgAAAAAAAA%3D%3D%23RT%3A2%23TRC%3A2000%23ISV%3A2%23IEO%3A65551%23QCF%3A1%22%2C%22range%22%3A%22%7B%5C%22min%5C%22%3A%5C%22%5C%22%2C%5C%22max%5C%22%3A%5C%2205C1DFFFFFFFFC%5C%22%2C%5C%22isMinInclusive%5C%22%3Atrue%2C%5C%22isMaxInclusive%5C%22%3Afalse%7D%22%7D
Pageable:  {'sort': {'sorted': False, 'unsorted': True, 'empty': True}, 'offset': 0, 'requestContinuation': '{"token":null,"range":"{\\"min\\":\\"05C1DFFFFFFFFC\\",\\"max\\":\\"FF\\",\\"isMinInclusive\\":true,\\"isMaxInclusive\\":false}"}', 'pageNumber': 2, 'pageSize': 692, 'unpaged': False, 'paged': True}

exec 4
https://baseurl/api/documents/recent?&page=3&requestContinuation=%7B%22token%22%3Anull%2C%22range%22%3A%22%7B%5C%22min%5C%22%3A%5C%2205C1DFFFFFFFFC%5C%22%2C%5C%22max%5C%22%3A%5C%22FF%5C%22%2C%5C%22isMinInclusive%5C%22%3Atrue%2C%5C%22isMaxInclusive%5C%22%3Afalse%7D%22%7D
Pageable:  {'sort': {'sorted': False, 'unsorted': True, 'empty': True}, 'offset': 0, 'requestContinuation': '{"token":"-RID:~I9lLALNXyUNkLgAAAAAACA==#RT:1#TRC:1000#ISV:2#IEO:65551#QCF:1","range":"{\\"min\\":\\"05C1DFFFFFFFFC\\",\\"max\\":\\"FF\\",\\"isMinInclusive\\":true,\\"isMaxInclusive\\":false}"}', 'pageNumber': 3, 'pageSize': 1000, 'unpaged': False, 'paged': True} 

exec 5
https://baseurl/api/documents/recent?&page=4&requestContinuation=%7B%22token%22%3A%22-RID%3A~I9lLALNXyUNkLgAAAAAACA%3D%3D%23RT%3A1%23TRC%3A1000%23ISV%3A2%23IEO%3A65551%23QCF%3A1%22%2C%22range%22%3A%22%7B%5C%22min%5C%22%3A%5C%2205C1DFFFFFFFFC%5C%22%2C%5C%22max%5C%22%3A%5C%22FF%5C%22%2C%5C%22isMinInclusive%5C%22%3Atrue%2C%5C%22isMaxInclusive%5C%22%3Afalse%7D%22%7D
Pageable:  {'sort': {'sorted': False, 'unsorted': True, 'empty': True}, 'offset': 0, 'requestContinuation': '{"token":"-RID:~I9lLALNXyUO3YQAAAAAACA==#RT:2#TRC:2000#ISV:2#IEO:65551#QCF:1","range":"{\\"min\\":\\"05C1DFFFFFFFFC\\",\\"max\\":\\"FF\\",\\"isMinInclusive\\":true,\\"isMaxInclusive\\":false}"}', 'pageNumber': 4, 'pageSize': 1000, 'unpaged': False, 'paged': True}

exec 6 
https://baseurl/api/documents/recent?&page=5&requestContinuation=%7B%22token%22%3A%22-RID%3A~I9lLALNXyUO3YQAAAAAACA%3D%3D%23RT%3A2%23TRC%3A2000%23ISV%3A2%23IEO%3A65551%23QCF%3A1%22%2C%22range%22%3A%22%7B%5C%22min%5C%22%3A%5C%2205C1DFFFFFFFFC%5C%22%2C%5C%22max%5C%22%3A%5C%22FF%5C%22%2C%5C%22isMinInclusive%5C%22%3Atrue%2C%5C%22isMaxInclusive%5C%22%3Afalse%7D%22%7D
Pageable:  {'sort': {'sorted': False, 'unsorted': True, 'empty': True}, 'offset': 0, 'requestContinuation': '{"token":"-RID:~I9lLALNXyUNc8QAAAAAACA==#RT:3#TRC:3000#ISV:2#IEO:65551#QCF:1","range":"{\\"min\\":\\"05C1DFFFFFFFFC\\",\\"max\\":\\"FF\\",\\"isMinInclusive\\":true,\\"isMaxInclusive\\":false}"}', 'pageNumber': 5, 'pageSize': 1000, 'unpaged': False, 'paged': True}

Total Documents:  5692
kushagraThapar commented 4 years ago

@lucprosa There are multiple things to clarify here:

  1. To correctly paginate, please follow this example: https://docs.microsoft.com/en-us/azure/java/spring-framework/how-to-guides-spring-data-cosmosdb#pagination-and-sorting
  2. Regarding page size -> cosmosdb backend can return any number of docs between 0 and page size you request for based on the total size of the payload. There is a hard limit on cosmosdb backend of 4 MB. That is the reason you saw just 629 documents being returned on exec-3

Let me know if this helps.

lucprosa commented 4 years ago

@kushagraThapar I changed my code to use sorting and this behaviour has gone.

Now I'm just setting a default order by using the Cosmos ID column.

I wasn't using sorting before because I was concerned about performance issues.

Now I can get the right number of total items and the tokens are not empty.

--------------------------------------
Total Elements:  5839
Total Pages:  6
Page Size:  1000
--------------------------------------
https://baseurl/api/documents/recent?
Pageable:  {'sort': {'sorted': True, 'unsorted': False, 'empty': False}, 'offset': 0, 'requestContinuation': '{"compositeToken":"{\\"token\\":\\"+RID:~I9lLALNXyUM5QwAAAAAAAA==#RT:1#TRC:1000#RTD:18B3pgkDJbSB2a+3B0uvBTZnZWRmZ2M1LmZkOGUuNWJkNC5iMTgzLjQ5ZDM1NjVlN2RjMQA=#ISV:2#IEO:65551#QCF:1\\",\\"range\\":\\"{\\\\\\"min\\\\\\":\\\\\\"\\\\\\",\\\\\\"max\\\\\\":\\\\\\"05C1DFFFFFFFFC\\\\\\",\\\\\\"isMinInclusive\\\\\\":true,\\\\\\"isMaxInclusive\\\\\\":false}\\"}","orderByItems":[{"item":"2a8b63a2-23cd-4bcb-aef8-9568a6718ad9","map":{"item":"2a8b63a2-23cd-4bcb-aef8-9568a6718ad9"}}],"rid":"I9lLALNXyUMqSQAAAAAAAA==","inclusive":true}', 'pageNumber': 0, 'pageSize': 1000, 'paged': True, 'unpaged': False}

https://baseurl/api/documents/recent?&page=1&requestContinuation=%7B%22compositeToken%22%3A%22%7B%5C%22token%5C%22%3A%5C%22%2BRID%3A~I9lLALNXyUM5QwAAAAAAAA%3D%3D%23RT%3A1%23TRC%3A1000%23RTD%3A18B3pgkDJbSB2a%2B3B0uvBTZnZWRmZ2M1LmZkOGUuNWJkNC5iMTgzLjQ5ZDM1NjVlN2RjMQA%3D%23ISV%3A2%23IEO%3A65551%23QCF%3A1%5C%22%2C%5C%22range%5C%22%3A%5C%22%7B%5C%5C%5C%22min%5C%5C%5C%22%3A%5C%5C%5C%22%5C%5C%5C%22%2C%5C%5C%5C%22max%5C%5C%5C%22%3A%5C%5C%5C%2205C1DFFFFFFFFC%5C%5C%5C%22%2C%5C%5C%5C%22isMinInclusive%5C%5C%5C%22%3Atrue%2C%5C%5C%5C%22isMaxInclusive%5C%5C%5C%22%3Afalse%7D%5C%22%7D%22%2C%22orderByItems%22%3A%5B%7B%22item%22%3A%222a8b63a2-23cd-4bcb-aef8-9568a6718ad9%22%2C%22map%22%3A%7B%22item%22%3A%222a8b63a2-23cd-4bcb-aef8-9568a6718ad9%22%7D%7D%5D%2C%22rid%22%3A%22I9lLALNXyUMqSQAAAAAAAA%3D%3D%22%2C%22inclusive%22%3Atrue%7D
Pageable:  {'sort': {'sorted': True, 'unsorted': False, 'empty': False}, 'offset': 0, 'requestContinuation': '{"compositeToken":"{\\"token\\":\\"+RID:~I9lLALNXyUMrWQAAAAAACA==#RT:1#TRC:1000#RTD:18B3pgkDJbSB2a+3B0uvBThjOTo1MTk1LjM3MWQuNWY1Oi5iMWc2LmI5NmMzMjpnMzExZwA=#ISV:2#IEO:65551#QCF:1\\",\\"range\\":\\"{\\\\\\"min\\\\\\":\\\\\\"05C1DFFFFFFFFC\\\\\\",\\\\\\"max\\\\\\":\\\\\\"FF\\\\\\",\\\\\\"isMinInclusive\\\\\\":true,\\\\\\"isMaxInclusive\\\\\\":false}\\"}","orderByItems":[{"item":"56ee8931-1441-4e30-b764-811fa6c297e3","map":{"item":"56ee8931-1441-4e30-b764-811fa6c297e3"}}],"rid":"I9lLALNXyUNGCgEAAAAACA==","inclusive":true}', 'pageNumber': 1, 'pageSize': 1000, 'paged': True, 'unpaged': False}

https://baseurl/api/documents/recent?&page=2&requestContinuation=%7B%22compositeToken%22%3A%22%7B%5C%22token%5C%22%3A%5C%22%2BRID%3A~I9lLALNXyUMrWQAAAAAACA%3D%3D%23RT%3A1%23TRC%3A1000%23RTD%3A18B3pgkDJbSB2a%2B3B0uvBThjOTo1MTk1LjM3MWQuNWY1Oi5iMWc2LmI5NmMzMjpnMzExZwA%3D%23ISV%3A2%23IEO%3A65551%23QCF%3A1%5C%22%2C%5C%22range%5C%22%3A%5C%22%7B%5C%5C%5C%22min%5C%5C%5C%22%3A%5C%5C%5C%2205C1DFFFFFFFFC%5C%5C%5C%22%2C%5C%5C%5C%22max%5C%5C%5C%22%3A%5C%5C%5C%22FF%5C%5C%5C%22%2C%5C%5C%5C%22isMinInclusive%5C%5C%5C%22%3Atrue%2C%5C%5C%5C%22isMaxInclusive%5C%5C%5C%22%3Afalse%7D%5C%22%7D%22%2C%22orderByItems%22%3A%5B%7B%22item%22%3A%2256ee8931-1441-4e30-b764-811fa6c297e3%22%2C%22map%22%3A%7B%22item%22%3A%2256ee8931-1441-4e30-b764-811fa6c297e3%22%7D%7D%5D%2C%22rid%22%3A%22I9lLALNXyUNGCgEAAAAACA%3D%3D%22%2C%22inclusive%22%3Atrue%7D
Pageable:  {'sort': {'sorted': True, 'unsorted': False, 'empty': False}, 'offset': 0, 'requestContinuation': '{"compositeToken":"{\\"token\\":\\"+RID:~I9lLALNXyUNvIQAAAAAACA==#RT:1#TRC:1000#RTD:18B3pgkDJbSB2a+3B0uvBWI3ODkzODNmLjVlOjMuNTM4Mi46Y2Q5Ljk1ZDdjOTJiOGczMQA=#ISV:2#IEO:65551#QCF:1\\",\\"range\\":\\"{\\\\\\"min\\\\\\":\\\\\\"05C1DFFFFFFFFC\\\\\\",\\\\\\"max\\\\\\":\\\\\\"FF\\\\\\",\\\\\\"isMinInclusive\\\\\\":true,\\\\\\"isMaxInclusive\\\\\\":false}\\"}","orderByItems":[{"item":"832f3bf1-3964-439f-a297-507706076310","map":{"item":"832f3bf1-3964-439f-a297-507706076310"}}],"rid":"I9lLALNXyUPzWAAAAAAACA==","inclusive":true}', 'pageNumber': 2, 'pageSize': 1000, 'paged': True, 'unpaged': False}

https://baseurl/api/documents/recent?&page=3&requestContinuation=%7B%22compositeToken%22%3A%22%7B%5C%22token%5C%22%3A%5C%22%2BRID%3A~I9lLALNXyUNvIQAAAAAACA%3D%3D%23RT%3A1%23TRC%3A1000%23RTD%3A18B3pgkDJbSB2a%2B3B0uvBWI3ODkzODNmLjVlOjMuNTM4Mi46Y2Q5Ljk1ZDdjOTJiOGczMQA%3D%23ISV%3A2%23IEO%3A65551%23QCF%3A1%5C%22%2C%5C%22range%5C%22%3A%5C%22%7B%5C%5C%5C%22min%5C%5C%5C%22%3A%5C%5C%5C%2205C1DFFFFFFFFC%5C%5C%5C%22%2C%5C%5C%5C%22max%5C%5C%5C%22%3A%5C%5C%5C%22FF%5C%5C%5C%22%2C%5C%5C%5C%22isMinInclusive%5C%5C%5C%22%3Atrue%2C%5C%5C%5C%22isMaxInclusive%5C%5C%5C%22%3Afalse%7D%5C%22%7D%22%2C%22orderByItems%22%3A%5B%7B%22item%22%3A%22832f3bf1-3964-439f-a297-507706076310%22%2C%22map%22%3A%7B%22item%22%3A%22832f3bf1-3964-439f-a297-507706076310%22%7D%7D%5D%2C%22rid%22%3A%22I9lLALNXyUPzWAAAAAAACA%3D%3D%22%2C%22inclusive%22%3Atrue%7D
Pageable:  {'sort': {'sorted': True, 'unsorted': False, 'empty': False}, 'offset': 0, 'requestContinuation': '{"compositeToken":"{\\"token\\":\\"+RID:~I9lLALNXyUNqIwAAAAAAAA==#RT:1#TRC:1000#RTD:18B3pgkDJbSB2a+3B0uvBWVnZmU1NDczLmY0YzQuNTkyOS46YjczLjplNjdmOTQ4MmNiZAA=#ISV:2#IEO:65551#QCF:1\\",\\"range\\":\\"{\\\\\\"min\\\\\\":\\\\\\"\\\\\\",\\\\\\"max\\\\\\":\\\\\\"05C1DFFFFFFFFC\\\\\\",\\\\\\"isMinInclusive\\\\\\":true,\\\\\\"isMaxInclusive\\\\\\":false}\\"}","orderByItems":[{"item":"ae7da80d-cb3b-4e39-b412-fca6bc3b06fa","map":{"item":"ae7da80d-cb3b-4e39-b412-fca6bc3b06fa"}}],"rid":"I9lLALNXyUNyYwAAAAAAAA==","inclusive":true}', 'pageNumber': 3, 'pageSize': 1000, 'paged': True, 'unpaged': False}

https://baseurl/api/documents/recent?&page=4&requestContinuation=%7B%22compositeToken%22%3A%22%7B%5C%22token%5C%22%3A%5C%22%2BRID%3A~I9lLALNXyUNqIwAAAAAAAA%3D%3D%23RT%3A1%23TRC%3A1000%23RTD%3A18B3pgkDJbSB2a%2B3B0uvBWVnZmU1NDczLmY0YzQuNTkyOS46YjczLjplNjdmOTQ4MmNiZAA%3D%23ISV%3A2%23IEO%3A65551%23QCF%3A1%5C%22%2C%5C%22range%5C%22%3A%5C%22%7B%5C%5C%5C%22min%5C%5C%5C%22%3A%5C%5C%5C%22%5C%5C%5C%22%2C%5C%5C%5C%22max%5C%5C%5C%22%3A%5C%5C%5C%2205C1DFFFFFFFFC%5C%5C%5C%22%2C%5C%5C%5C%22isMinInclusive%5C%5C%5C%22%3Atrue%2C%5C%5C%5C%22isMaxInclusive%5C%5C%5C%22%3Afalse%7D%5C%22%7D%22%2C%22orderByItems%22%3A%5B%7B%22item%22%3A%22ae7da80d-cb3b-4e39-b412-fca6bc3b06fa%22%2C%22map%22%3A%7B%22item%22%3A%22ae7da80d-cb3b-4e39-b412-fca6bc3b06fa%22%7D%7D%5D%2C%22rid%22%3A%22I9lLALNXyUNyYwAAAAAAAA%3D%3D%22%2C%22inclusive%22%3Atrue%7D
Pageable:  {'sort': {'sorted': True, 'unsorted': False, 'empty': False}, 'offset': 0, 'requestContinuation': '{"compositeToken":"{\\"token\\":null,\\"range\\":\\"{\\\\\\"min\\\\\\":\\\\\\"05C1DFFFFFFFFC\\\\\\",\\\\\\"max\\\\\\":\\\\\\"FF\\\\\\",\\\\\\"isMinInclusive\\\\\\":true,\\\\\\"isMaxInclusive\\\\\\":false}\\"}","orderByItems":[{"item":"db6cff5d-c86c-46c7-b7ea-c4a22a36aa5c","map":{"item":"db6cff5d-c86c-46c7-b7ea-c4a22a36aa5c"}}],"rid":"I9lLALNXyUMvWgAAAAAACA==","inclusive":true}', 'pageNumber': 4, 'pageSize': 1000, 'paged': True, 'unpaged': False}

https://baseurl/api/documents/recent?&page=5&requestContinuation=%7B%22compositeToken%22%3A%22%7B%5C%22token%5C%22%3Anull%2C%5C%22range%5C%22%3A%5C%22%7B%5C%5C%5C%22min%5C%5C%5C%22%3A%5C%5C%5C%2205C1DFFFFFFFFC%5C%5C%5C%22%2C%5C%5C%5C%22max%5C%5C%5C%22%3A%5C%5C%5C%22FF%5C%5C%5C%22%2C%5C%5C%5C%22isMinInclusive%5C%5C%5C%22%3Atrue%2C%5C%5C%5C%22isMaxInclusive%5C%5C%5C%22%3Afalse%7D%5C%22%7D%22%2C%22orderByItems%22%3A%5B%7B%22item%22%3A%22db6cff5d-c86c-46c7-b7ea-c4a22a36aa5c%22%2C%22map%22%3A%7B%22item%22%3A%22db6cff5d-c86c-46c7-b7ea-c4a22a36aa5c%22%7D%7D%5D%2C%22rid%22%3A%22I9lLALNXyUMvWgAAAAAACA%3D%3D%22%2C%22inclusive%22%3Atrue%7D
Pageable:  {'sort': {'sorted': True, 'unsorted': False, 'empty': False}, 'offset': 0, 'requestContinuation': None, 'pageNumber': 5, 'pageSize': 839, 'paged': True, 'unpaged': False}
Time:  13.9415364
Total Documents:  5839
@Repository
class DocumentRepository(@Autowired private val template: CosmosTemplate){

  def findRecentDocuments(pageRequest: CosmosPageRequest, criteria: Criteria, documentClass: Class[Document], collectionName: String): Page[Document] = {
    val query = new DocumentQuery(criteria)
    val sortedColumn = "id"
    val defaultSort = Sort.by(sortedColumn).ascending()
    template.paginationQuery(query.`with`(pageRequest).`with`(defaultSort), documentClass, collectionName)
  }
}