Pagination idea: type-specific refetching plugins

stubailo commented 8 years ago

In some sense, handling pagination in a smart way is just a special case of "when you ask for data, treat certain fields in a special way because we know that the arguments are meaningful." So if we're looking at a Relay spec paginated list:

// first fetch
{
  user {
    id
    name
    friends(first: 10) {
      edges {
        cursor
        node {
          id
          name
        }
      }
      pageInfo {
        hasNextPage
      }
    }
  }
}

// new query
... same stuff
friends(first: 20)

// second fetch, doesn't fetch all 20 but uses the existing information
... same stuff
friends(first: 10, after: "lastCursor")

Notice how easy it is for us as humans to imagine what data needs to be fetched to satisfy the new query.

Here's a set of hypotheses:

The initial fetch doesn't need any up-front information about types or pagination - it can look at the query result to know what to do, as long as we inject __typename fields where necessary.
The re-fetch can determine the new query by the information in the store, which now has type annotations and the arguments from the new query
The transformation from (2) can be written as a pure function that can be injected into the apollo client and associated with certain type names

Basically, you could write a function and tell the apollo client:

"When refetching any field which refers to an object with a type matching this regular expression, give me the query you were going to fetch, and the contents of the store, and I'll give you a new query that says how to fetch the missing data."

So, for Relay pagination, you'd say:

client.registerPaginationPlugin(/.+Connection/, ({ state, id, selectionSet }) => {
  ... do work ...

  return {
    // as much of the result as can be found
    result,

    // if empty, then the cache is sufficient and result contains the data.
    // otherwise, an array of queries that need to be fetched
    missingSelectionSets, 
  }
});

Ideally this will allow plugging in to different pagination systems, as long as the paginated fields have predictable type names, for example *PaginatedList or Paginated*.

If we can do this, it will achieve some really nice effects:

You don't necessarily need to use the Relay pagination spec, which can be hard to translate to some REST APIs
The store can avoid being concerned with pagination if the API above is indeed sufficient

This is just a very ambitious idea, and I'm eager to determine if this very simple model can actually handle all cases of pagination and Relay connections in particular. More analysis to come soon.

@jbaxleyiii @helfer curious what you think about this.

jbaxleyiii commented 8 years ago

@stubailo I'm a big fan of the flexibility of this method. We will be using what we have of apollo-client in an app this week that has pagination. I'll report back what usage on a non relay spec server is like.

cc @johnthepink

stubailo commented 8 years ago

What do you mean by this week? :P that sounds pretty soon!

helfer commented 8 years ago

I think these kinds of plugins are a great idea! The thing to keep in mind is that we might have to be more careful with garbage collection in the case where newer queries rely on the data from no longer active queries still being in the store. Somehow we'd have to know what data the plugins are extracting from the store. It might not be necessary for pagination, but you could imagine other plugins that know for example that the result of { base_amount, accrued_interest(after_years: 5) } is enough to calculate { base_amount, accrued_interest(after_years: 6) }. If the first query is no longer active and the garbage collection doesn't know that the second query is now using that data, we'd be fetching extra data. Probably not a huge deal in this case, but something to keep in mind. Of course this is a completely fictional example, but stuff like this should remain a possibility.

stubailo commented 8 years ago

BTW this issue blocks on #42

stubailo commented 8 years ago

Assigning to Martijn to get his feedback!

stubailo commented 8 years ago

Rather than having regex on type, we could do something similar to ESLint rules, which specify which AST nodes they are interested in: https://github.com/apollostack/eslint-plugin-graphql/blob/545fcecc8476a13c8d12291cd7fc8924a366178a/src/index.js#L25

deoqc commented 8 years ago

Any updates in pagination? I'm using a custom solution, without a very good result.

Really hoping to have an Apollo solution to this.

Pagination (see bellow) is as high-order element that initialize/fetch data on mount, delete on unmount, and expose 2 props to its children:

data with the query result
addFetch to call when needing to fetch more data (I use it in react native ListView in onEndReached).

Problems:

Duplicated data: I actually have a reducer pagination in my redux store and aggregate the results of pagination fetching. This is only temporary since I destroy on unmount and trust the apollo client query to memoize results.
Performance: I'm feeling its pain and believe its because of the duplicate the data fetching redux logic (first apollo query, then I use the result and aggregate in pagination.[config.name] branch in store. I'm using immutablejs in the store.

Here is the complete gist, and bellow the main parts and some explanation.

const query = `
  query doSomeQuery(..., $first: Int!, after: String) { # --> must accept these two parameters (after needs to allow null)
    viewer {
      nest {
         myPaginated($first: Int!, after: String) {
           edges { # --> data array must be in edges
             node {
               ...
             }
           }
           pageInfo {  #  --> must have a pageInfo with at least these two fields
             hasNextPage
             endCursor
           }
         }
       }
    }
  }
`

Now I can abstract the pagination to a custom connector:

const variables = (state, ownProps) => ({
  // custom variables: will fixed after first call
});

const config = {
  name: 'myUniquePaginationName',
  query,
  path: ['viewer', 'nest', 'myPagination'], // need to now where the pagination part is so I aggregate data
  initCount: 10, // $first parameter in initial fetching
  addCount: 10, // $first parameter in adicional fetching
  variables,
};

const ComponentWithPagination = Pagination(config)(Component);

stubailo commented 8 years ago

We're going to start working on pagination full-time soon, I'll take a deeper look at this then!

jbaxleyiii commented 8 years ago

@stubailo is there any progress / recommendations for pagination?

stubailo commented 8 years ago

Ah, sorry, not yet - we needed to do batching first because it was necessary for Galaxy. I'm not sure that the query diffing approach for pagination is the right one, I think I actually prefer something more like addFetch. @jbaxleyiii do you guys have a big paginated view in your app? It would be good to take a look at the needs there and see how it would work with different designs.

jbaxleyiii commented 8 years ago

@stubailo we do! can we schedule a call?

abhiaiyer91 commented 8 years ago

Let's write a spec for this? Should we rely on the view frameworks to power this experience for us? If so, let's write specs for the view layers we support now

dbx834 commented 8 years ago

Hello,

After having recently decided to jump on the Apollo ship, I've been teaching myself Apollo. Please see http://sandbox.kāla.com/data-emulation.

Was looking the best way to go about pagination and stumbled across this. I've tried out something similar to what @deoqc has done. Do you think is it a good idea to make some sort of a wrapper for this? So that pagination (and other repetitive and often used functions like sort, filter, etc) can be quickly implemented for any data.

How are the Apollo developers going to implement pagination?

Also, what is the status for stuff like filtering, sorting and searching? Will Apollo have some support for that, or will we have to write our own stuff?

See https://github.com/graphql/graphql-relay-js/issues/20 in this context.

Thanks!

dbx834 commented 8 years ago

Hello again,

Here's another solution that works fairly well with RESTful APIs,

I have been able to do all sorts of things - sort, filter, sort on particular columns, text-search using this method.

Server,

import { createApolloServer } from 'meteor/apollo';
import { HTTP } from 'meteor/http';
import _ from 'lodash';
import randomstring from 'randomstring';
import numeral from 'numeral';
import cache from 'memory-cache';

// ----------------------------------------------------------------------- API Adapter

/**
* @summary Adapter, connects to the API
*/
class Adapter {

  /**
  * @summary constructor, define default stuff
  */
  constructor() {
    this.configuration = { // Declare default values here or in the resolver
      currentPage: '1', // The first query will always show the first page
    };
  }

  /**
  * @summary callApi, calls the API
  * @param {string} url, the URL
  * @returns {object}, the response
  */
  callApi(url) {
    try {
      const apiResponse = HTTP.get(url);
      const returnObject = {
        count: null,
        data: [],
      };
      _.each(apiResponse.data.results, function (row) {
        returnObject.data.push({
          id: randomstring.generate(),
          name: row.name,
          diameter: row.diameter,
          rotationPeriod: row.rotation_period,
          orbitalPeriod: row.orbital_period,
          gravity: row.gravity.replace('standard', '').trim(),
          population: (row.population === 'unknown' ? row.population : numeral(parseInt(row.population, 10)).format('0a')),
          climate: row.climate,
          terrain: row.terrain,
          surfaceWater: row.surface_water,
        });
      });
      returnObject.count = apiResponse.data.count;
      return returnObject;
    } catch (error) {
      console.log(error);
      return null;
    }
  }

  /**
  * @summary configure the API
  * @param {object} args, arguments
  * @returns {object} this.configuration, returns the configuration
  */
  configure(args) {
    this.configuration.currentPage = args.currentPage;

    // Just an example. Anything can be returned to the client in the conf object
    let metaPlanetsTotalRecords = cache.get('metaPlanetsTotalRecords');
    if (!metaPlanetsTotalRecords) {
      metaPlanetsTotalRecords = this.callApi('http://swapi.co/api/planets/?page=1').count; // Get counts anyhow, this is used client-side to determine how many pages there will be
      cache.put('metaPlanetsTotalRecords', metaPlanetsTotalRecords, (60 * 60 * 1000) /* Keep this in memory of one hour */);
    }
    this.configuration.totalRecords = metaPlanetsTotalRecords;

    return this.configuration;
  }

  /**
  * @summary fetch from remote
  * @returns {object} data, returns the data from remote API call, or returns null in case of error
  */
  fetch() {
    return this.callApi(`http://swapi.co/api/planets/?page=${this.configuration.currentPage}`).data;
  }
}

const API = new Adapter();

// ----------------------------------------------------------------------- Schema

const schema = [`
type Planet {
  id: String
  name: String
  diameter: String
  gravity: String
  climate: String
  terrain: String
  rotationPeriod: String
  population: String
  orbitalPeriod: String
  surfaceWater: String
}
type MetaPlanets {
  planets: [Planet]
  totalRecords: String
  currentPage: String
}
type Query {
  planets: [Planet]
  metaPlanets(currentPage: String): MetaPlanets
}
schema {
  query: Query
}
`];

// ----------------------------------------------------------------------- Resolvers

const resolvers = {
  Query: {
    planets() {
      return API.fetch();
    },
    metaPlanets(root, { currentPage = '1' } = {}) { // Declare default values here or in the Adapter
      return API.configure({ currentPage });
    },
  },
  MetaPlanets: {
    planets() {
      return API.fetch();
    },
  },
};

createApolloServer({
  graphiql: true,
  pretty: true,
  schema,
  resolvers,
});

And, in client,

import React from 'react';
import { connect } from 'react-apollo';
import { createContainer } from 'meteor/react-meteor-data';
import gql from 'graphql-tag';

import { Table } from 'meteor/sandbox:lib-duplicate';

// ----------------------------------------------------------------------- Component JSS Stylesheet

... // Removed for brevity

// ----------------------------------------------------------------------- Table definitions

const columns = [{
  title: 'Name',
  key: 'name',
  render: (record) => {
    return (
      <span>{record.name}</span>
    );
  },
}, ... // Removed for brevity];

// ----------------------------------------------------------------------- Component

/**
 * @summary DataEmulationPaginate
 * @returns {object} Renders a DOM Object
 */
class DataEmulationPaginate extends React.Component {

  /**
   * @summary constructor function
   * @param {object} props, component properties
   * @returns {object} null
   */
  constructor(props) {
    super(props);

    // Link functions
    this.onPageChange = this.onPageChange.bind(this);
  }

  /**
   * @summary onPageChange function, what to do on page change?
   * @returns {object} null
   */
  onPageChange(page) {
    console.log(`On page: ${page}`);
  }

  /**
   * @summary render function
   * @returns {object} Returns DOM Object
   */
  render() {

    const classes = this.props.sheet.classes;
    console.log(this.props);
    console.log(this.props.data.metaPlanets);

    return (
      <section>
        <h1 className="m-b-1">Paginate</h1>
        <Table columns={columns} dataSource={this.props.data.loading === false ? this.props.data.metaPlanets.planets : []} loading={this.props.data.loading} pagination={{ total: 61, onChange: this.onPageChange }} />

      </section>
    );
  }
}

// ----------------------------------------------------------------------- GraphQL Adapter

const Adapter = connect({
  mapQueriesToProps() {
    return {
      data: {
        query: gql`
          query getMetaPlanets ($currentPage: String) {
            metaPlanets(currentPage: $currentPage) {
              currentPage
              totalRecords
              planets {
                id
                name
                diameter
                rotationPeriod
                orbitalPeriod
                gravity
                population
                climate
                terrain
                surfaceWater
              }
            }
          }
        `,
        variables: {
          currentPage: '2', // Connect this to state or props, or whatever
        },
      },
    };
  },
})(DataEmulationPaginate);

// ----------------------------------------------------------------------- Container Component

const Container = createContainer(() => {
  return {};
}, Adapter);

export default useSheet(Container, stylesheet);

dbx834 commented 8 years ago

Here's a working example, Apollo-Paginate

sandervanhooft commented 8 years ago

Here's a working example, Apollo-Paginate

@dbx834 Seems to be offline.

dbx834 commented 8 years ago

Please try again. Just checked, it works for me.

sandervanhooft commented 8 years ago

Yes, seems to be up again, thanks! :) On di 2 aug. 2016 at 15:19, Pranav notifications@github.com wrote:

Please try again. Just checked, it works for me.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/apollostack/apollo-client/issues/26#issuecomment-236901370, or mute the thread https://github.com/notifications/unsubscribe-auth/AG7dpxzitWHoslo1TLY7ZLPtf8WAGnNlks5qb0PngaJpZM4H3ukx .

stubailo commented 8 years ago

We just merged a new pagination approach: https://github.com/apollostack/apollo-client/pull/472

Docs coming soon: https://github.com/apollostack/docs/pull/157

apollographql / apollo-client

Pagination idea: type-specific refetching plugins #26