Pagination

[LadaRudnitckaia]

Design and implement pagination for GET

A reasonable requirement for kbtbr is to handle paginated server responses correctly, such that users to not silently receive only a partial response. This should actually be implemented before the first release, at best.

To that end, please

1. Generate test data for GET methods (i.e. a number of assets such that pagination actually happens)
2. Add the option for pagination to the get of the Kobo class

TBD

• whether to use the Paginator from crul or write the code ourselves.
• if we write the code ourselves, via adding a loop, we should also take care of the connection problems that might arise for different iterations of the loop, i.e. making the process robust against failure.

[MKyhos]

First finding: in the current version of {crul}, only offset-based pagination is implemented in the Paginator class:

#' Supported now:
#'
#' - `limit_offset`: the most common way (in my experience), so is the default.
#' This method involves setting how many records and what record to start at
#' for each request. We send these query parameters for you.
#' - `page_perpage`: set the page to fetch and (optionally) how many records
#' to get per page
#'
#' Supported later, hopefully:
#'
#' - `link_headers`: link headers are URLS for the next/previous/last
#' request given in the response header from the server. This is relatively
#' uncommon, though is recommended by JSONAPI and is implemented by a
#' well known API (GitHub).
#' - `cursor`: this works by a single string given back in each response, to
#' be passed in the subsequent response, and so on until no more records
#' remain. This is common in Solr

(see [here])https://github.com/ropensci/crul/blob/master/R/paginator.R#L27-L43)

As Kobotoolbox is using a link-header style pagination, we hence have to implement it by ourselves. However, the design of Paginator might be useful/inspiring :)

cc @dimitri-mar @LadaRudnitckaia