Expose Encode & Decode Utility Function to Get Cursor

[willjleong]

I've been using this library along with gqlgen to implement graphql cursor pagination for the graphql relay connection standard. I've currently duplicated the priavte encode and decode utility functions in my own util package to provide gorm-cursor-paginator compatible cursors each edge that is resolved through dataloaden (go dataloader implementation) (I use gorm-cursor paginator to select only ID and CreatedAt from the database, return an array of IDs and dataloader resolves so I need to be able to use the encode function outside of the direct pagination). This change would be making Encode and Decode public methods.

Here's and example of the graphql query. As you can see the standard allows the client to get the cursor for each item in the paginated struct array. Exposing the Encode will allow the paginator object to handle the Encoding of these cursors if I resolve the objects through dataloader.

{
  user {
    id
    name
    friends(first: 10, after: "opaqueCursor") {
      edges {
        cursor
        node {
          id
          name
        }
      }
      pageInfo {
        hasNextPage
      }
    }
  }
}

[pilagod]

What your primary need is to provide cursor for each item (not only head and tail) in a batch of items, is it?

I'm not familiar with GraphQL and dataloader pattern. Could you please provide some simple examples for how to integrate paginator with these tools ? I see some examples from github.com/graph-gophers/dataloader, and wonder if it is something like:

func main() {
        // init loader
    loader := dataloader.NewBatchedLoader(
                batchFunc, 
                dataloader.WithCache(&dataloader.NoCache{}),
        )

        // given after cursor
        afterCursor := "after_cursor"

        // load data from loader    
    result, _ := loader.Load(
                context.TODO(), 
                dataloader.StringKey(afterCursor),
        )()
    fmt.Printf("result: %s\n", result)
}

func batchFunc(_ context.Context, keys dataloader.Keys) []*dataloader.Result {
        var results []*dataloader.Result
        var models []Model

        // init query
        stmt := db.Select("id, createdAt")

        // init paginator
        p := paginator.New()
        p.SetAfterCursor(keys[0])

        // do paging
        p.Paginate(stmt, &models)

        // append model to loader result
        for _, model := range models {
                results = append(results, &dataloader.Result{model, nil})
        }
        return results
}

[willjleong]

Yeah pretty much what you pointed to above. The relay graphql standard has a field in the list of edges that can resolve a cursor for each record (called a node in graphql). I ended up having to copy and reuse code that is in your utils folder that I have to hardcode and make sure my cursor composite is the same as what I used for cursor pagination. With graphql, you typically allow resolvers to batch fetch the collection by pk ids so I'm making a query like this with your pagination, and then in an edge resolver, I need to get the cursor for each result.

Cursor Paginate the connection but only return ID and CreatedAt

func ListUserIDsByOrganizationIDCursor(db *gorm.DB, pq pagination.PagingCursorQuery, q map[string]interface{}) ([]int64, *paginator.Cursor, int, error) {
    var results []int64
    var resultsNoPtr []models.User
    lcdb := db.Table("users").Select("users.id, users.created_at")

    // Look for organization_id to join if needed
    if val, ok := q["organization_id"]; ok {
        lcdb = lcdb.
            Joins("inner join user_organizations on user_organizations.user_id = users.id AND user_organizations.organization_id = ?", val)
        delete(q, "organization_id")
    }

    // Todo: support other filters
    //query.Ands(lcdb, q)

    count := 0
    // Count total
    lcdb.Count(&count)

    page := pagination.GetModelCursorPaginator(pq)
    res := page.Paginate(lcdb, &resultsNoPtr)

    if res.Error != nil {
        log.Debug().Err(res.Error)
        return nil, nil, count, res.Error
    }

    cursor := page.GetNextCursor()

    for i := 0; i < len(resultsNoPtr); i++ {
        results = append(results, resultsNoPtr[i].ID)
    }

    return results, &cursor, count, nil

}

The returned int64 ID slice and cursor information gets mapped into the userConnection struct and resolved at the end of the request. The Edges in the connection which will resolve users across any dataloader loading of users across requests (if a the connection is used in combination with querying a user in the connection there will not be an extra query to the DB preventing an n+1 problem with graphql when not using dataloader).

func (r *membersConnectionResolver) Edges(ctx context.Context, obj *models.MembersConnection) ([]*models.MembersEdge, error) {
    var results []*models.MembersEdge
    members, errs := dataloader.CtxLoaders(ctx).UserByID.LoadAll(obj.Ids)
    for _, err := range errs {
        if err != nil {
            return nil, gqlerror.Errorf("Failed to Load Organization Members")
        }
    }

    for i := 0; i < len(members); i++ {
        cursor := pagination.EncodeBase64([]string{pagination.Convert(members[i].CreatedAt), pagination.Convert(members[i].ID)})
        results = append(results, &models.MembersEdge{
            Cursor: cursor,
            Node:   members[i],
        })
    }
    return results, nil
}

[pilagod]

Much clearer, thanks for your detailed explanation !

I think export encode and decode methods would be a good idea in regard to integrating paginator into different situations.

For now encode is depending on keys in paginator set by SetKeys. In order to publish it, signature of encode should be modified:

func Encode(v reflect.Value, keys []string) string {
    fields := make([]string, len(keys))
    for index, key := range keys {
        // ...
    }
    return encodeBase64(fields)
}

The second parameter keys indicates which properties to encode on struct v. After publishing, usage in your case will look like:

func (r *membersConnectionResolver) Edges(ctx context.Context, obj *models.MembersConnection) ([]*models.MembersEdge, error) {
    // ...
    for i := 0; i < len(members); i++ {
        cursor := paginator.Encode(members[i], []string{"CreatedAt", "ID"}) 
        results = append(results, &models.MembersEdge{
            Cursor: cursor,
            Node:   members[i],
        })
    }
    return results, nil
}

Do you think this new signature for Encode is ok : ) ? (Decode will not change since it is not depending on any properties in paginator)

[willjleong]

Yup, this seems reasonable and would make it easier to utilize. Thank you!

[pilagod]

@willjleong I have released a new patch v1.1.1 to cover this issue : )