search

microsoft / spring-data-cosmosdb

Access data with Azure Cosmos DB
MIT License
94 stars 64 forks source link

Using requestContinuation to make pagination #484

Closed lucprosa closed 4 years ago

lucprosa commented 4 years ago

Hi, I'm using spring-data-cosmosdb 2.2.2 with azure-cosmos 3.6.0 and Scala.

I'm using CosmosTemplate to create custom DocumentQueries using sorting and pagination.

My problem is to navigate through the pages in my request using requestContinuation. Every request (POST) is returning a requestContinuation like that:

"requestContinuation": "{\"compositeToken\":\"{\\\"token\\\":null,\\\"range\\\":\\\"{\\\\\\\"min\\\\\\\":\\\\\\\"05C1DFFFFFFFFC\\\\\\\",\\\\\\\"max\\\\\\\":\\\\\\\"FF\\\\\\\",\\\\\\\"isMinInclusive\\\\\\\":true,\\\\\\\"isMaxInclusive\\\\\\\":false}\\\"}\",\"orderByItems\":[{\"item\":\"8b90e589-09d8-42e7-a3e2-d26cf2c38a63\",\"map\":{\"item\":\"8b90e589-09d8-42e7-a3e2-d26cf2c38a63\"}}],\"rid\":\"I9lLALNXyUNECgEAAAAACA==\",\"inclusive\":true}"

The first thing I saw is the inner "token" is always null.

I tried to copy the whole string and put it on my request as parameter but it didn't work: POST https://baseurl/api/documents/recent?page=1&requestContinuation=...

The requestContinuation never changes and the documents returned in the page are always the same. I'm using Postman and Insomnia.

I don't find good examples to use requestContinuation. Am I missing something? Maybe encoding the token with base64?

I don't know if the problem is just to pass the requestContinuation or if there's something wrong in my code.

In resume I'm just using paginationQuery method to execute a documentQuery with some criterias and a "pageable" object.

My Controller:

@RestController
@RequestMapping(path = Array("/api"))
class DocumentController(@Autowired private val service: DocumentService) extends BaseController {

  @PostMapping(path = Array("/documents/recent"), produces = Array("application/json"))
  def findRecentDocuments(pageable: Pageable,
                          @RequestBody params: DocumentParams,
                          @RequestHeader(name="apiKey") token: String
                   ): Page[Document] = {

    service.findRecentDocuments(pageable, Option(params))
  }

My service:

class DocumentService(@Autowired private val documentRepository: DocumentRepository, @Autowired private val template: CosmosTemplate) extends BaseService {
  def findRecentDocuments(pageable: Pageable, params: Option[DocumentParams]): Page[Document] = {     

      val documentClass = classOf[Document]
      val collectionName = template.getCollectionName(documentClass)
      val defaultSort = Sort.by(Direction.ASC, "id")
      val sort = pageable.getSortOr(defaultSort)
      val criteria = QueryUtils.getCriteria(params, documentClass)

      getCriteriaInfo(criteria)
      documentRepository.findRecentDocuments(pageable, criteria, sort, documentClass, collectionName)
  }

My repo:

class DocumentRepository(@Autowired private val template: CosmosTemplate) {

  def findRecentDocuments(pageable: Pageable, criteria: Criteria, sort: Sort, documentClass: Class[Document], collectionName: String): Page[Document] = {
    val query = new DocumentQuery(criteria)

    template.paginationQuery(query.`with`(sort).`with`(pageable), documentClass, collectionName)
  }
}

Thanks for any help.

kushagraThapar commented 4 years ago

@lucprosa Did you try the recommended way of pagination provided in the readme section of spring-data-cosmosdb: https://github.com/microsoft/spring-data-cosmosdb/blob/master/README.md Section: Supports Spring Data pageable and sort.

Also documented here: https://docs.microsoft.com/en-us/azure/java/spring-framework/how-to-guides-spring-data-cosmosdb#pagination-and-sorting

lucprosa commented 4 years ago

@kushagraThapar Yes, I saw that I was using a "Pageable" object to control de pagination and not the "CosmosPageRequest" that uses requestContinuation token and is recommended by documentation. Thanks for your response.

But I already changed my code here and it still give me a error when I try to pass a requestContinuation to get the next page of request.

Now I getting a "INVALID JSON" error: "Failed to query items; nested exception is CosmosClientException{error={\"message\":\"INVALID JSON in continuation token {\\\\"token\\\\":\\\\"+RID:~I9lLALNXyUM5EwAAAAAAAA== for Parallel~Context\"}, resourceAddress='null', statusCode=400, message=INVALID JSON in continuation token {\\"token\\":\\"+RID:~I9lLALNXyUM5EwAAAAAAAA== for Parallel~Context, causeInfo=null, responseHeaders={}, requestHeaders=null}",

I created a Python script to make the requests through my API. I get the requestPagination of current request to use in the next request to get the next page.

r = requests.post(urlAPI + '&page=' + str(page) + '&requestContinuation=' + str(previousRequestContinuation), data=body, headers=header)
kushagraThapar commented 4 years ago

@lucprosa can you please provide updated code snippet ? How are you getting the continuation token and using it for next pageable ? If you are getting it from response, you should not get invalid json .

lucprosa commented 4 years ago

Sure, follow my code below.

But I guess I found the gap here. It was the encoding of the parameter.

Controller:

@RestController
@RequestMapping(path = Array("/api"))
class DocumentController(@Autowired private val service: DocumentService) extends BaseController {

  @PostMapping(path = Array("/documents/recent"), produces = Array("application/json"))
  def findRecentDocuments(@RequestParam(required = false, defaultValue = "0") page: Int,
                          @RequestParam(required = false, defaultValue = "100") size: Int,
                          @RequestParam(required = false) requestContinuation: String,
                          @RequestBody params: DocumentParams,
                          @RequestHeader(name="apiKey") token: String
                   ): Page[Document] = {

    val pageable: PageParams = PageParams(page, size, requestContinuation)

    service.findRecentDocuments(pageable, Option(params))
  }

Service:

@Service
class DocumentService(@Autowired private val documentRepository: DocumentRepository, @Autowired private val template: CosmosTemplate) extends BaseService {

  def findRecentDocuments(pageable: PageParams, params: Option[DocumentParams]): Page[Document] = {
    val documentClass = classOf[Document]
    val collectionName = template.getCollectionName(documentClass)
    val criteria = QueryUtils.getCriteria(params, documentClass)

    println("page: " + pageable.getPage)
    println("size: " + pageable.getSize)
    println("requestContinuation: "+ pageable.getRequestContinuation)

    val pageRequest: CosmosPageRequest = new CosmosPageRequest(pageable.page, pageable.size, pageable.requestContinuation)

    getCriteriaInfo(criteria)
    documentRepository.findRecentDocuments(pageRequest, criteria, documentClass, collectionName)
}

Repository:

@Repository
class DocumentRepository(@Autowired private val template: CosmosTemplate){

  def findRecentDocuments(pageRequest: CosmosPageRequest, criteria: Criteria, documentClass: Class[Document], collectionName: String): Page[Document] = {
    val query = new DocumentQuery(criteria)
    template.paginationQuery(query.`with`(pageRequest), documentClass, collectionName)
  }

When I send a request I received a pageable information like this:

  "pageable": {
    "sort": {
      "unsorted": true,
      "sorted": false,
      "empty": true
    },
    "offset": 0,
    "requestContinuation": "{\"token\":\"-RID:~I9lLALNXyUO2GwAAAAAAAA==#RT:2#TRC:101#ISV:2#IEO:65551#QCF:1\",\"range\":\"{\\\"min\\\":\\\"\\\",\\\"max\\\":\\\"05C1DFFFFFFFFC\\\",\\\"isMinInclusive\\\":true,\\\"isMaxInclusive\\\":false}\"}",
    "pageNumber": 0,
    "pageSize": 100,
    "paged": true,
    "unpaged": false
  },
 "totalPages": 59,
  "totalElements": 5839,
  "last": false,
  "sort": {
    "unsorted": true,
    "sorted": false,
    "empty": true
  },
  "numberOfElements": 100,
  "first": true,
  "size": 100,
  "number": 0,
  "empty": false
lucprosa commented 4 years ago

Sure.

Controller:

@RestController
@RequestMapping(path = Array("/api"))
class DocumentController(@Autowired private val service: DocumentService) extends BaseController {

  @PostMapping(path = Array("/documents/recent"), produces = Array("application/json"))
  def findRecentDocuments(@RequestParam(required = false, defaultValue = "0") page: Int,
                          @RequestParam(required = false, defaultValue = "100") size: Int,
                          @RequestParam(required = false) requestContinuation: String,
                          @RequestBody params: DocumentParams,
                          @RequestHeader(name="apiKey") token: String
                   ): Page[Document] = {

    val pageable: PageParams = PageParams(page, size, requestContinuation)

    service.findRecentDocuments(pageable, Option(params))
  }

Service:

@Service
class DocumentService(@Autowired private val documentRepository: DocumentRepository, @Autowired private val template: CosmosTemplate) extends BaseService {

  def findRecentDocuments(pageable: PageParams, params: Option[DocumentParams]): Page[Document] = {
    val documentClass = classOf[Document]
    val collectionName = template.getCollectionName(documentClass)
    val criteria = QueryUtils.getCriteria(params, documentClass)

    println("page: " + pageable.getPage)
    println("size: " + pageable.getSize)
    println("requestContinuation: "+ pageable.getRequestContinuation)

    val pageRequest: CosmosPageRequest = new CosmosPageRequest(pageable.page, pageable.size, pageable.requestContinuation)

    getCriteriaInfo(criteria)
    documentRepository.findRecentDocuments(pageRequest, criteria, documentClass, collectionName)
}

Repository:

@Repository
class DocumentRepository(@Autowired private val template: CosmosTemplate){

  def findRecentDocuments(pageRequest: CosmosPageRequest, criteria: Criteria, documentClass: Class[Document], collectionName: String): Page[Document] = {
    val query = new DocumentQuery(criteria)
    template.paginationQuery(query.`with`(pageRequest), documentClass, collectionName)
  }

When I send a request I received a pageable information like this:

  "pageable": {
    "sort": {
      "unsorted": true,
      "sorted": false,
      "empty": true
    },
    "offset": 0,
    "requestContinuation": "{\"token\":\"-RID:~I9lLALNXyUO2GwAAAAAAAA==#RT:2#TRC:101#ISV:2#IEO:65551#QCF:1\",\"range\":\"{\\\"min\\\":\\\"\\\",\\\"max\\\":\\\"05C1DFFFFFFFFC\\\",\\\"isMinInclusive\\\":true,\\\"isMaxInclusive\\\":false}\"}",
    "pageNumber": 0,
    "pageSize": 100,
    "paged": true,
    "unpaged": false
  },
 "totalPages": 59,
  "totalElements": 5839,
  "last": false,
  "sort": {
    "unsorted": true,
    "sorted": false,
    "empty": true
  },
  "numberOfElements": 100,
  "first": true,
  "size": 100,
  "number": 0,
  "empty": false

I guess I found the problem here. It's all about encoding the parameter before send to request.

from requests.utils import quote
encodeToken = quote(str(requestContinuation))

I don't get the JSON error now.

I'll still make some tests with pagination here but I think this issue can be closed by now.

Thanks

kushagraThapar commented 4 years ago

@lucprosa Thanks for providing the code, it looks good now. Glad it got resolved, please feel free to close the issue.

lucprosa commented 4 years ago

Sure. Thanks @kushagraThapar

11shobhit commented 4 years ago

i am facing similar issue. can some one please help on this

"paginationToken": "{\"compositeToken\":\"{\\"token\\":\\"+RID:~XGtkAKZ5mcBlBwAAAAAAAA==#RT:2#TRC:6#RTD:03v6tHWhTzQwL/nVyeVsBMHX21Px/+9L5A==#ISV:2#IEO:65551#FPC:AWAHAAAAAAAAaQcAAAAAAAA=\\",\\"range\\":\\"{\\\\"min\\\\":\\\\"\\\\",\\\\"max\\\\":\\\\"FF\\\\",\\\\"isMinInclusive\\\\":true,\\\\"isMaxInclusive\\\\":false}\\"}\",\"orderByItems\":[{\"item\":1.5910333972E9,\"map\":{\"item\":1.5910333972E9}}],\"rid\":\"XGtkAKZ5mcBjBwAAAAAAAA==\",\"inclusive\":true}", "messages": [ { "id": "ec6613da-06cb-4bbb-be8b-66116a35c031", "clientId": "test", "to": "springdata@jun1.com",

CODE SNIPPET: String continuationToken=((CosmosPageRequest) ((CosmosPageImpl) pagedResponse).getPageable()).getRequestContinuation(); map.put(continuationToken, response); return map

kushagraThapar commented 4 years ago

@11shobhit - try encoding the continuationToken -> like done in the above resolution comment: https://github.com/microsoft/spring-data-cosmosdb/issues/484#issuecomment-583622882

11shobhit commented 4 years ago

thanks @kushagraThapar it works