Pagination support in REST APIs

[ejaz-ahmed]

Currently default json/xml response rendered by grails does not include pagination information like totals, current max, offset and etc. Getting this information requires overriding default render method and index action of RestfulController. This should be part of grails core. Spring data rest can be taken as reference for that.

[ejaz-ahmed]

I have a solution for this and will provide PR.

[scheiblr]

This sounds great. @ejaz-ahmed Does your solution also include the sorting feature as described in the Spring documentation?

[ejaz-ahmed]

@schmittr it does not support sorting but I think it can be worked out. Let me share my snippet here as I'll have to setup grails locall to test it properly. Will do it in few days.

[ejaz-ahmed]

You need to register a renderer to accept pagination request

package org.gmobile.renderers

import grails.converters.JSON
import grails.rest.render.AbstractRenderer
import grails.rest.render.ContainerRenderer
import grails.rest.render.RenderContext
import grails.util.GrailsWebUtil
import org.grails.web.json.JSONWriter
import grails.web.mime.MimeType
import org.gmobile.converters.ApiJSON

class ApiRendererJson<T> extends AbstractRenderer<T> {

    String label

    public ApiRendererJson(Class<T> targetClass) {
        super(targetClass, MimeType.JSON);
    }

    @Override
    void render(T object, RenderContext context) {
        context.setContentType(GrailsWebUtil.getContentType(MimeType.JSON.name, GrailsWebUtil.DEFAULT_ENCODING))
        ApiJSON converter
        def detail = context.arguments?.detail ?: "compact"
        def out = context.writer
        JSONWriter writer = new JSONWriter(out)

        JSON.use(detail) {
            converter = object as ApiJSON
        }

        writer.object()
        writer.key(getLabel())
        converter.renderPartial(writer)

        if(context.arguments?.paging) {
            writer.key("paging")
            converter = context.arguments.paging as ApiJSON
            converter.renderPartial(writer)
        }
        writer.endObject()

        out.flush()
        out.close()
    }

    String getLabel() {
        if(label) {
            label
        }
        else if(this instanceof ContainerRenderer) {
            "entities"
        }
        else {
            "entity"
        }
    }
}

Then in index action use it like:

def index(Integer max) {
        params.max = Math.min(max ?: 10, 100)
        def detail = params.detail ?: "complete"
        respond Phone.list(params), [detail:detail, paging:[totalCount: Phone.count(),
                                                            currentMax: params.max,
                                                            curentOffset: params.offset ?: 0]]
    }

It has no sorting right now but we have overridden the render method and we can sort achieve this feature too but I am not sure how nor did I gave it a try. Will look into it some time later.

[ejaz-ahmed]

I've taken much reference of this stuff from this talk

[ejaz-ahmed]

Spring Data REST has many more like security and query stuff. I opened series of these issues after getting inspired by it.

[ejaz-ahmed]

Sorting can easily be done by overriding index action of RestfulController and passing sort query as it is to GORM's list method. I'll try to handle this too in my PR.

[ejaz-ahmed]

I've updated my fork with pagination support for JSON only. It is not ready for PR right now and I want someone to pre-audit my changes. The response rendered by get request for collection is below:

{"entity":[{"id":1,"price":63.3,"title":"Grails in  Action"},{"id":2,"price":53.3,"title":"Groovy in  Action"}],"paging":{"totalCount":2,"currentMax":10,"curentOffset":0}}

Since pagination is rendered as JSON object, there is need to contain default response of grails. I've added a dummy label named entity for this purpose. My changes require index action to be written this way:

def index(Integer max) {
        params.max = Math.min(max ?: 10, 100)
        respond Book.list(params), [paging:[totalCount: Book.count(),
                                                           currentMax: params.max,
                                                            curentOffset: params.offset ?: 0]]
    }

Which means a change in RestfulController.groovy too which I'll accomplish later. I've changed only two files and someone can pre-audit and suggest code cleanup as I've messed up few things. Here are these

• JSON.java
• DefaultJsonRenderer.groovy

Code is scattered among these two files and logically it should have been in JSON.java. The reason for this scattered code is RenderContext instance which is required by my logic. Can someone suggest some modification?

To render pagination, it is now necessary to contain default grails response in JSON object. For that I've added a property called label in DefaultJsonRender.groovy. If that property has no value, default response will add label "entities" for collection and "entity" for single object. But this logic is not working as the check I am performing checks for "ContainerRenderer" instance like below:

private String getLabel() {
        if(label) {
            label
        }
        else if(this instanceof ContainerRenderer) {
            "entities"
        }
        else {
            "entity"
        }
    }

I don't know why is it failing and else part is always executed. Is this possible to set label for each domain object this way?

[ejaz-ahmed]

Created this plugin which adds support for pagination. Sorting support is already there if we use Restful Controller. I've extended this controller in my AwesomeRestfulController which adds pagination support and inherits sorting.

[orubel]

Pagination can easily be supported through the api itself by passing the params for pagination and (if detected), using those to limit return set

If you are looking to reduce this for EVERY method, abstract the communication layer from business logic.

[ejaz-ahmed]

@orubel This sort of support is already there in grails. The only issue is it does not let client know of totals and remaining in the list. Are you talking about this? If so, please clarify it further.

[orubel]

Thats something you have to provide because it requires 2 queries. Not everyone wants to make 2 queries for an api call

Even nested, it's 2 calls. So its not so much a framework implementation but more a business logic implementation

[jeffscottbrown]

The way that the paged result list is implemented in g-d-m, the second query is only sent to the database if the total count is actually referenced. The total count is lazily initialized. https://github.com/grails/grails-data-mapping/blob/ec7cddd6e5fdf6525b4a045a6e0ec7f2517f209d/grails-datastore-gorm/src/main/groovy/grails/gorm/PagedResultList.java#L48

[orubel]

Hmmm. Never thought of that. Wow. Thanks. Going to have to integrate.

[graemerocher]

Implemented in JSON views 1.1