Overcoming Prismic's 20 document limit

[Jrousell]

Hi,

I've got search working with content sourced from Prismic by using the query and normalizer below .... and, it's awesome fast and works a treat - which is great.

However, because Prismic has a 20 document limit on their API the index is only being populated with the first 20 items. So, I was wondering whether it might be possible to utilise the cursors Prismic returns to fetch all content. Any guidance would be really welcomed.

Elsewhere in our app we're using these cursors for pagination, based on the example shown here. And I was wondering if a similar approach might be possible in the normalizer?

I may be way off here and going about this the wrong way, so thought I'd check if this has been solved elsewhere.

Many thanks

Query fragment from plugin config:

query: `
          {
            prismic {
              allStorys {
                pageInfo {
                  startCursor
                  endCursor
                  hasNextPage
                }
                edges {
                  node {
                    __typename
                    _meta {
                      id
                      uid
                      tags
                    }
                    title
                    seo_description
                  }
                }
              }
            }
          }
`

Normalizer extract

normalizer: ({ data }) => {

          let storyIndex = data.prismic.allStorys.edges.map(edge => {
            return {
              id: edge.node._meta.id,
              uid: edge.node._meta.uid,
              title: edge.node.title ? edge.node.title[0].text : "",
              description: edge.node.seo_description ? edge.node.seo_description[0].text : "",
              type: edge.node.__typename
            };
          });

          // other collections removed for brevity

          let nodes = [...storyIndex]

          return nodes;
        }

[angeloashmore]

Makes sense, I didn't think of how gatsby-source-graphql users would need to paginate data.

I think it's possible we could allow passing in a function or string for the query option. The string value would work as it does now. The function value would provide a graphql function that lets you query as much as necessary and return the full array of values to index.

It could look something like this:

{
  resolve: 'gatsby-plugin-local-search',
    options: {
      query: async (graphql) => {
        const nodes = []

        let startCursor
        let hasNextPage = true

        // Perform some kind of loop that appends your query results to the
        // `nodes` array above.
        while (hasNextPage) {
          const queryResult = graphql(`
            # Your query
          `, { startCursor }) // Note that you can pass GraphQL variables here.

          nodes = [...nodes, queryResults.data.nodes]

          startCursor = queryResult.data.endCursor
          hasNextPage = queryResult.data.hasNextPage
        }

        return nodes
      }
    }
}

[angeloashmore]

Would you be able to open an issue on gatsby-plugin-local-search? This repo is for the FlexSearch hook. Thanks!