search

LiveRamp / reslang

A language for describing resource-oriented APIs & turning them into Swagger or resource diagrams. Oriented around the concepts we want to expose in the APIs.
Apache License 2.0
23 stars 7 forks source link

cursor based pagination, configurable return size, defaults #106

Closed liveandrew closed 4 years ago

liveandrew commented 4 years ago

current standards are offset / limit for pagination. this doesn't work well for fast moving data

proposal by matt conors is for after / limit where after indicates the last returned entry from a sorted list

part of this will enhance to allow limit to be adjusted in multigets, and also for a default option for both pagination style and limit

liveandrew commented 4 years ago

proposal:

MULTIGET pagination=after,limit=20

options for pagination are offset, after, none

forestgagnon commented 4 years ago

from a standards point of view I think after should be typed as an opaque string, and we should allow for implementors to return some nextPageAfter field containing the cursor for the next page in their MULTIGET response bodies. It makes it easy for clients to fetch the next page without traversing the list and figuring out how to make a cursor (is it the resource ID? something else? etc), and keeping it opaque allows the implementor to non-breakingly modify the cursor in the future if the ordering criteria evolve over time.

Also, the client could use the absence of nextPageAfter in the response to discover when there are no more pages, instead of having to keep track of a count, which may change while paging through.

I've implemented opaque cursors before and they work well.

liveandrew commented 4 years ago

why can't we just use the last returned element id? the implication is that it is sorted, no? edit: i re-read and i get your point about lack of nextPageAfter showing end of file

liveandrew commented 4 years ago

how about you guys put the changes in the standards doc, send it out to #dev-api and #dev-architects, get consensus and then i'll implement the approach people agree on

liveandrew commented 4 years ago

added support for this, i got bored. we now support 3 types of pagination: "offset" (default), "after" and "none". you can also configure the limit default (number actually returned), and the max-limit (the maximum you can ask for)

it works like this:

resource Directory { id: uuid query name: string query /operations MULTIGET pagination=after limit=20 GET POST }

default for pagination is "offset", default for limit is 10, default for max-limit is 100

i'm not publicizing this until i get the go ahead from you guys on the standards.

for the "after" approach, it worked by you setting the "after" query param with the results of the last query's X-Next-After token taken from the header. we are treating it as opaque