Pagination (cursor-based) for GraphQL and Mongoose

[nicky-lenaers]

Feature Request

Nest is one of the most robust NodeJS frameworks out there. However, I think it would be nice to incorporate some form of out-of-the-box pagination using GraphQL and Mongoose (or any TypeORM for that matter). In this feature request I'd like to present my 2 cents for the idea of a pagination feature in Nest. I found myself using pagination in the context of GraphQL and Mongoose very often and I think others might be so too. Hence I think Nest could adopt the concept of pagination into its framework.

Is your feature request related to a problem? Please describe.

It is not.

Describe the solution you'd like

I while back I started the creation of a package I call MoGr, which combines Mongoose and GraphQL to allow for automated query projection, population and pagination.

You can check out the MoGr package on GitHub here.

In order to allow for pagination, I've set up the following steps using the MoGr package:

Note

The following code is for demonstration purposes and might not be complete as-is. The code is used to show the gist of the feature request in a more concrete fashion.

1. Pagination Service

First, a MoGr registry is created to store information coming from Mongoose and GraphQL. It uses the InjectConnection param decorator to retrieve the MongoDB connection.

pagination.service.ts

import { Injectable } from '@nestjs/common';
import { InjectConnection } from '@nestjs/mongoose';
import { Connection } from 'mongoose';
import { Registry } from '@nicky-lenaers/mogr';

@Injectable()
export class PaginationService {
  private reg: Registry;

  get registry(): Registry {
    return this.reg;
  }

  constructor(@InjectConnection() private readonly connection: Connection) {
    this.reg = new Registry(this.connection);
  }
}

2. Pagination Module

A module is created to hold and export the PaginationService:

pagination.module.ts

import { DynamicModule, Global, Module } from '@nestjs/common';
import { PaginationService } from './pagination.service';

@Global()
@Module({})
export class PaginationModule {
  static forRoot(): DynamicModule {
    return {
      module: PaginationModule,
      providers: [PaginationService],
      exports: [PaginationService]
    };
  }
}

3. Models for Edge, PageInfo and Page

Based on cursor-based pagination guidelines, the models for Edge, PageInfo and Page are as follows:

edge.interface.ts

export interface Edge<T> {
  cursor: string;
  node: T;
}

page-info.interface.ts

export interface PageInfo {
  startCursor: string;
  endCursor: string;
  hasPrevPage: boolean;
  hasNextPage: boolean;
}

page.interface.ts

import { PageInfo } from './page-info.interface';
import { Edge } from './edge.interface';

export interface Page<T> {
  pageInfo: PageInfo;
  totalCount: number;
  edges: Edge<T>[];
}

4. GraphQL Object Types for Edge, PageInfo and Page

Using the defined models for Edge, PageInfo and Page, we can now construct GraphQL Object Types. Notice that Page uses the generic type functionality from type-graphql:

page-info.type.ts

import { PageInfo as IPageInfo } from './page-info.interface';
import { Field, ObjectType } from 'type-graphql';

@ObjectType()
export class PageInfo implements IPageInfo {
  @Field({ nullable: true })
  public startCursor: string;

  @Field({ nullable: true })
  public endCursor: string;

  @Field({ nullable: true })
  public hasPrevPage: boolean;

  @Field({ nullable: true })
  public hasNextPage: boolean;
}

page.type.ts

import { Edge, Page as IPage } from './interfaces';
import { ClassType, Field, Int, ObjectType } from 'type-graphql';
import { PageInfo } from './page-info';

export function Page<T>(ItemType: ClassType<T>) {
  @ObjectType(`Paginated${ItemType.name}Edge`)
  abstract class EdgeClass implements Edge<T> {
    @Field()
    public cursor: string;

    @Field(() => ItemType)
    public node: T;
  }

  @ObjectType({ isAbstract: true })
  abstract class PageClass implements IPage<T> {
    @Field(() => PageInfo, { nullable: true })
    public pageInfo: PageInfo;

    @Field(() => Int)
    public totalCount: number;

    @Field(() => [EdgeClass])
    public edges: EdgeClass[];
  }

  return PageClass;
}

5. GraphQL Input Types for Filters and Query Options

Now we need a GraphQL Input Type in order to supply pagination filters and query options:

enums.ts

export enum OrderByDirection {
  ASC,
  DESC
}

order-by.interface.ts

import { OrderByDirection } from './enums';

export interface OrderBy<F> {
  field: F;
  direction: keyof typeof OrderByDirection;
}

query-options.interface.ts

import { OrderBy } from './order-by.interface';

export interface QueryOptions<T, O extends OrderBy<T>> {
  first?: number;
  last?: number;
  before?: string;
  after?: string;
  orderBy?: O[];
}

query-options.type.ts

import { OrderByDirection } from 'enums';
import { OrderBy } from './order-by.interface';
import { QueryOptions as IQueryOptions } from './query-options.interface';
import {
  ClassType,
  Field,
  InputType,
  Int,
  registerEnumType
} from 'type-graphql';
import { ReturnTypeFuncValue } from 'type-graphql/dist/decorators/types';

registerEnumType(OrderByDirection, {
  name: 'OrderByDirection'
});

export function QueryOptions<T, S extends ReturnTypeFuncValue>(
  ItemType: ClassType<T>,
  SortableItemType: S
) {
  @InputType({ isAbstract: true })
  abstract class OrderByClass
    implements OrderBy<keyof typeof SortableItemType> {
    @Field(() => SortableItemType)
    public field: keyof typeof SortableItemType;

    @Field(() => OrderByDirection)
    public direction: keyof typeof OrderByDirection;
  }

  @InputType(`Paginated${ItemType.name}QueryOptions`)
  abstract class PaginatedQueryOptionsClass
    implements IQueryOptions<keyof typeof SortableItemType, OrderByClass> {
    @Field(() => Int, { nullable: true })
    public first: number;

    @Field(() => Int, { nullable: true })
    public last: number;

    @Field({ nullable: true })
    public before: string;

    @Field({ nullable: true })
    public after: string;

    @Field(() => [OrderByClass], { nullable: 'itemsAndList' })
    public orderBy: OrderByClass[];
  }

  return PaginatedQueryOptionsClass;
}

filter-string.input.ts

import { PageFieldFilter } from './interfaces';
import { FieldFilter } from '@nicky-lenaers/mogr';
import { Field, InputType } from 'type-graphql';
import { FilterStringContainsInput } from './filter-string-contains.input';

@InputType()
export class FilterStringInput implements FieldFilter, PageFieldFilter {
  @Field(() => FilterStringContainsInput, { nullable: true })
  public contains: FilterStringContainsInput;

  @Field({ nullable: true })
  public eq: string;

  @Field({ nullable: true })
  public ne: string;

  @Field(() => [String], { nullable: 'itemsAndList' })
  public in: string[];
}

filter-string-contains.input.ts

import { Field, InputType } from 'type-graphql';

@InputType()
export class FilterStringContainsInput {
  @Field()
  public value: string;

  @Field({ nullable: true })
  public options: string;
}

6. Usage - Arguments

To utilize the above types, we take the example of Recipes:

enums.ts

export enum OrderByDirection {
  ASC,
  DESC
}

export enum RecipesSortableField {
  id,
  name,
  ingredients
}

recipes.args.ts

import {
  OrderByDirection,
  RecipesSortableField
} from './enums';
import { PageArgs } from '@nicky-lenaers/mogr';
import { ArgsType, Field } from 'type-graphql';
import { RecipesFiltersInput } from './recipes-filters.input';
import { RecipesQueryOptions } from './recipes-query-options.input';

export type RecipesPageArgs = PageArgs<
  keyof typeof RecipesSortableField,
  keyof typeof OrderByDirection,
  RecipesFiltersInput
>;

@ArgsType()
export class RecipesArgs implements RecipesPageArgs {
  @Field(() => [RecipesFiltersInput], { nullable: 'itemsAndList' })
  public filters: RecipesFiltersInput[];

  @Field(() => RecipesQueryOptions, { nullable: true })
  public queryOptions: RecipesQueryOptions;
}

page-field-filter.interface.ts

export interface PageFieldFilter {
  contains?: {
    value: string;
    options?: string;
  };
  eq?: string;
  ne?: string;
  in?: string[];
}

recipes-filters.input.ts

import { RecipesSortableField } from './enums';
import { PageFieldFilter } from './page-field-filter.interface';
import { PageArgsFilter } from '@nicky-lenaers/mogr';
import { Field, InputType } from 'type-graphql';
import { FilterStringInput } from '../../pagination/dto/filter-string.input';

export type RecipesPageSortableFields = {
  [K in keyof typeof RecipesSortableField]?: PageFieldFilter
};

@InputType()
export class RecipesFiltersInput implements PageArgsFilter, RecipesPageSortableFields {
  @Field(() => FilterStringInput, { nullable: true })
  public id: FilterStringInput;

  @Field(() => FilterStringInput, { nullable: true })
  public name: FilterStringInput;

  @Field(() => FilterStringInput, { nullable: true })
  public ingredients: FilterStringInput;
}

6. Usage - Page

To utilize the above types, we keep using the example of Recipes:

recipies-page.type.ts

import { Recipes } from './models';

@ObjectType()
export class RecipesPage extends Page(Recipes) {}

Then, on the RecipesResolver, we can use the pagination as follows:

recipes.resolver.ts

@Resolver()
export class RecipesResolver {

  constructor(
    private readonly recipesService: RecipesService
  ) {}

  @Query(() => RecipesPage)
  async getRecipes(
    @Args() recipesPageArgs: RecipesPageArgs
  ) {
    return this.recipesService.getRecipesPage(recipesPageArgs);
  }
}

The RecipesService looks as follows (using the MoGr package):

import { queryPage } from '@nicky-lenaers/mogr';

@Injectable()
export class RecipesService {

  constructor(
    @InjectModel('Recipe') private readonly recipeModel: Model<Recipe & Document>
  ) {}

  getRecipesPage(recipePageArgs: RecipesPageArgs) {
    return queryPage(
      this.recipeModel.find(),
      recipePageArgs
    )
  }
}

Now, any type can be paginated using the generic Page type and the generic filters and query options.

NOTE

It is also possible to automate query projection and population using this approach, but the feature is already very long, so I'll leave that to another time.

Teachability, Documentation, Adoption, Migration Strategy

This would need some documentation afterwards. This issue is just for proposing the idea of pagination.

What is the motivation / use case for changing the behavior?

Pagination is needed in many mondern web application for e.g. infinite scrolling, huge data-tables and much more. It would be a nice addition to the Nest package.

[malkaviano]

That seems more like a module/side/lib than core.

I hope you don't go that way, it will make the framework team bugs double at least, not to mention the difficulty to deploy new framework versions.

My 2cents.

[nicky-lenaers]

@malkaviano Thank you for your comment. There's a couple of things I'd like to address.

That seems more like a module/side/lib than core.

You're absolutely right about this. It should be a module, for the same reason @nestjs/mongoose and @nestjs/graphql are modules: it's an opt-in. When configured as a module, it would play nicely between a TypeORM / Mongoose and GraphQL.

I hope you don't go that way, it will make the framework team bugs double at least, not to mention the difficulty to deploy new framework versions.

I think it is a major stretch to assume a doubling of framework team bugs here. Just because it is a non-trivial feature doesn't mean it will produce bugs. It should be discussed and designed up front in order to have an integration plan for the module so to prevent design flaws, not to mention a CI/CD pipeline as a guard for production environments. There is already a detailed guideline for Relay Cursor Pagination by Facebook available that can be used to refine and adjust the design. In short, I do not agree with this remark.

I hope this feature could still be considered and discussed, as many web applications nowadays need some form of pagination / infinite loading features for performance reasons.

[malkaviano]

@nicky-lenaers Yeah about the second part of my post, I did it before learning more about Nestjs, looks like you already went that way so it's unfair to block this feature exactly.

It's a good feature indeed.

Just for clarification, I did not mean the feature couldn't be part of Nestjs or developed by the same team, I just meant it shouldn't be in the core package. Maybe that point needed more clarification.

[nicky-lenaers]

@malkaviano Thanks for your clarification :)

[WillSquire]

Works great, unless I try to use RecipesPage in another module's model. Then I get, Cannot read property 'name' of undefined for Paginated${TClass.name}Edge.

Any ideas?

[nicky-lenaers]

@WillSquire So that means that TClass is undefined. So it could be an error in:

@ObjectType()
export class RecipesPage extends Page(Recipes) {}

Are you sure you imported the Page function?

[WillSquire]

@nicky-lenaers Yes, I'm sure. It works in some places, but using it another module's model brings up the error. Banged my head for quite a while before removing it.

[marcus-sa]

@nicky-lenaers Yes, I'm sure. It works in some places, but using it another module's model brings up the error. Banged my head for quite a while before removing it.

Probably a circular dependency issue.

[WillSquire]

@marcus-sa Yeah, I thought the same thing but it's a type rather than an instance so I wasn't sure if it was resolvable in the same way as the nest docs mention?

[kamilmysliwiec]

We don't plan to release this feature as part of the ecosystem packages (no bandwidth to handle that). However, if anyone would ever release such a package, please, let others know here!

[nicky-lenaers]

@kamilmysliwiec Thanks for clarifying! Keep up great work at Nest 👍

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.