The recommended way to paginate view results in CouchDB, and therefore the stack, is through cursors.
This is because skip performs badly on large data collections, as it requires to scan the full B-Tree.
For instance, a query on 2500 documents, with a limit of 100 docs per request, was taking more than 25s overall, with sometimes more than 1s per paginated request.
In fetchReferencedFiles, the sort was forced on datetime, which, behind the hoods, triggers the use of the referenced-by-sorted-by-datetime view (see here and here)
This view uses [doctype, id, datetime] keys, which is useful to get all the references sorted on the datetime through partial match queries on the key, e.g. key=[io.cozy.photos.albums, "album-id", {}]. However, it is not possible to use partial-match queries with cursor-based pagination as it requires the use of a startkey_docid field, which makes only sense when there are multiple results for a single key, not multiple ones.
So, this PR allows to use an id sort, which queries the referenced-by view, with [doctype, id] keys. Therefore, one can query a particular album with key=[io.cozy.photos.albums, "album-id"] and paginate the results with cursors through startkey_docid.
The recommended way to paginate view results in CouchDB, and therefore the stack, is through cursors. This is because
skip
performs badly on large data collections, as it requires to scan the full B-Tree.For instance, a query on 2500 documents, with a limit of 100 docs per request, was taking more than 25s overall, with sometimes more than 1s per paginated request.
In
fetchReferencedFiles
, the sort was forced ondatetime
, which, behind the hoods, triggers the use of thereferenced-by-sorted-by-datetime
view (see here and here) This view uses[doctype, id, datetime]
keys, which is useful to get all the references sorted on the datetime through partial match queries on the key, e.g.key=[io.cozy.photos.albums, "album-id", {}]
. However, it is not possible to use partial-match queries with cursor-based pagination as it requires the use of astartkey_docid
field, which makes only sense when there are multiple results for a single key, not multiple ones.So, this PR allows to use an
id
sort, which queries thereferenced-by
view, with[doctype, id]
keys. Therefore, one can query a particular album withkey=[io.cozy.photos.albums, "album-id"]
and paginate the results with cursors throughstartkey_docid
.