Using cursor to build paginated API

orgrimarr commented 3 years ago

Needs

I'm trying to build an API that needs pagination. I can't use the LIMIT/OFFSET as recommended because i'm manipulating real-time data : https://www.arangodb.com/docs/stable/aql/operations-limit.html One solution is to use a cursor based pagination (https://dev.to/rahul_ramfort/understanding-offset-vs-cursor-based-pagination-1582)

I know that arangojs is using the cursor api of arango. (https://www.arangodb.com/docs/stable/http/aql-query-cursor.html)

Example

I want to implement this workflow :

API call to my service GET /myapi -> db.query(query) -> take the first response -> Answer the API call with data and the cursor key (1234)
Asking the next set of item GET /myapi?cursor=1234 -> using arangoJS to fetch the data using the cursor -> respond to the api call with data and the next cursor key (12345)
Asking the next set of item GET /myapi?cursor=12345 -> using arangoJS to fetch the data using the cursor -> respond to the api call with data and the next cursor key (123456)
...

This example is not actual code, but it explains what I am trying to do.

const db = new Database()

if(firstRequest){
    const query = `FOR t IN test RETURN t`
    const bindvars = {}
    const options = {
        batchSize: 10
    }
    const cursor = await db.query(query, bindvars, options)
    const result = await cursor.batches.next()
    const cursorID = cursor.hasNext ? cursor.aCursorID : null
    return {
        result,
        cursorID
    }
}
else {
    // Use cursor to fetch the next data
    const cursorID = getCursorID()
    const cursor = await db.route('/cursor')
    const result = await cursor.put(cursorID)
    return {
        result: result.result,
        cursorID: result.hasMore ? result.id : null
    }
}

Questions

Can i use the query cursor response from arangoJS to do that ? Can i obtain the cursor id from the first response ?

I dont know how to use the query function to ask for the next batch.

Do i need to use the API directly ? db.route("/cursor").put('26011191') ?

Is there a better way to implement this ?

Regards

pluma commented 2 years ago

I would advise against relying on the cursor API for this as cursors are generally meant to be rather short-lived and pagination is usually not. Keep in mind when writing queries you can also sort and filter within the query, so for example if your documents have a timestamp you could say "give me the first 10 results from before that timestamp" (and then the next 10 and so on via limit and offset).

You're right that arangojs currently doesn't provide a way to manipulate cursors directly. You could try interacting with the HTTP API directly if you really want this.

pluma commented 2 years ago

I'm closing this question due to inactivity. Please check the community slack if you need further assistance: https://slack.arangodb.com

arangodb / arangojs