Pagination(Couchdb): anyway to do it with repository using "bookmark"

[julianobrasil]

I've noticed that net.sf.jkniv.whinstone.Repository has a lot of ready-to-use methods. I haven't found a way to run a paginated query though.

CoachDb, according to the docs, would need to receive a bookmark within the json request in order to do a performant query. The result of any query has the following shape (and I'm interested in getting access to the bookmark part):

{
  Collection<T> docs;
  String warning;
  Object execution_stats;
  String bookmark;
}

As far as I could understand, jkniv Repository gives me the docs part of the object above. I'd have to get access to the bookmark so I could send it again in the next query, to get the next page, in a performant way, like this:

  "selector": {
        ....
   }    
   "bookmark": :bookmark,
   "limit": :limit

Is this already implemented? Or is there a way I can get the raw result from a query using Repository?

[julianobrasil]

One more info: I saw I can use the QueryFactory.of(String name, int offset, int max, Object... args) to achieve the pagination, and I have no clue yet about what jkniv does internaly to perform the query. I gess it just use skip along with limit, but according to the docs, "While skip exists, it is not intended to be used for paging. The reason is that the bookmark feature is more efficient.".

Doing some google research, I found this article showing that the use of startKey is a lot faster than just skipping result documents. I have a feeling (nothing more than that) that startKey and bookmark are related inside couchdb.

Obs: This is more like a java question than a jkniv one, but when I was trying to use the of method with pagination, I got an Eclipse error.

Have I gotten wrong the way I should be using it?

[alisson-gomesc]

The native pagination with whinstone-couchdb is not implemented yet.

You right, the current version 0.6.0.M44 just give you docs part. An enhancement will be done for return the others: warning, execution_stats, bookmark.

The query startKey is efficiently when find with a range of keys, likes View Pagination or View Paging Alternate Method.

About QueryFactory.of there are 3 method to use with pagination:

Queryable of(String name, int offset, int max, Object... args)
Queryable of(String name, Object params, int offset, int max)
Queryable of(String name, int offset, int max)

You need remove the parameter ObjectReference.class, until now none accept a parameter as return type class together with paginate. I will check the ambiguous case.

If you are using View you can make use of the parameters key, keys, startkey, endkey, skip, limit, just put these parameters in your Queryable object to paginate the result.

I will go prioritize the pagination requirement to the next version and explain how do works.

[julianobrasil]

The query startKey is efficiently when find with a range of keys

Got it!

[julianobrasil]

When you're working on it.... it would be nice if the result set of a paginated query also brought the total number of elements that would be returned if the query wasn't paginated at all. It's pretty common at the frontend to have a pagination control that shows something like "Elements X to Y of totalElements" on each data slice view (or similar information).

Taking Spring data framework as an example, it has a Pageable type to help you send a paginated request to the underlying database (that feature is similar to jkniv's of pagination parameters), and it returns a Page<T> response that, among other attributes, returns the total number of elements. The Page response object is a nice feature.

[alisson-gomesc]

It's nice the Pageable interface. Queryable jkniv interface was designed more simple to supports pagination... when the query is executed jkniv fetch a count query in database to know the total of records. But the couchdb until version 2.0.0 no support bookmark feature, there hasn't a simple way to know the total of records... I will upgrade to the 2.1.0 version where bookmark is supported.

So, from queryable interface.

    /** 
     * Get total of rows that query can retrieve, filled after query is
     * executed. Useful to make pagination.
     * 
     * @return return the number of rows from query result.
     * <ul>
     *  <li>{@code n} - the real rows total</li> 
     *  <li>{@code -1} - the query isn't executed yet</li> 
     *  <li>{@code -2} - the query was executed successfully but the number of rows is unavailable</li> 
     *  <li>{@code -3} - the query was executed with error and the number of rows is unavailable</li> 
     * </ul>
     */
    long getTotal();

[alisson-gomesc]

So, CouchDb until last version 2.3.0 haven't way to find out the total of records with mango without fetch all records, mango queries with 2.1.0 or high just give us

bookmark:"g1AAAAAyeJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYozGoMkOGASEKEsAErgDR0"

With bookmark the next queries to fetch a new page of results is efficiently, so that's way I will implements a path to recovery the object response from CouchDB.

Response JSON Object:

• docs (object) – Array of documents matching the search. In each matching document, the fields specified in the fields part of the request body are listed, along with their values.
• warning (string) – Execution warnings
• execution_stats (object) – Execution statistics
• bookmark (string) – An opaque string used for paging. See the bookmark field in the request (above) for usage details.

This way I understood that the best way to paginate with CouchDB is use Infinite Scroll technique