Overcoming Prismic's 20 document limit

[Jrousell]

Hi,

Thanks for this great plugin. I've got a basic implementation working with content sourced from Prismic and it's blazingly fast...

However, because Prismic has a 20 document limit on their API the index is only being populated with the first 20 items. So, I was wondering whether it might be possible to utilise the cursors Prismic returns to fetch all content.

If, as @angeloashmore says, the query option accepted a function we could have some kind of loop that fetches all the documents using cursors (original post here).

Any guidance would be really welcomed.

Many thanks

[Jrousell]

So, I got this kinda working based on the approach outlined by @angeloashmore, but right now I'm bypassing the setup in gatsby-config and doing it all in the plugin. In our case the graphql needs to query multiple types and so I'm working out how to accomplish that, after which I'll try to move the setup back into the config.

  let result = []
  let startCursor = ''
  let hasNextPage = true

  while (hasNextPage) {
    const queryResults = await graphql(
      `
        query allContent($after: String) {
          prismic {
            allCtStorys(after: $after) {
              pageInfo {
                hasNextPage
                hasPreviousPage
                startCursor
                endCursor
              }
              edges {
                node {
                  __typename
                  _meta {
                    id
                    uid
                    tags
                  }
                  title
                  seo_description
                }
              }
            }
          }
        }
      `,
      { after: startCursor },
    )

    result = [...result, ...queryResults.data.prismic.allCtStorys.edges];
    startCursor = queryResults.data.prismic.allCtStorys.pageInfo.endCursor
    hasNextPage = queryResults.data.prismic.allCtStorys.pageInfo.hasNextPage
  }

  const documents = await Promise.resolve(normalizer(result))

[Jrousell]

So, got this working by adding a types property to the config and a graphql fragment with a placeholder for interpolation, then by looping over defined types and using the code above to keep fetching data until no more documents are available, I was able to get all the nodes from all the types, which I then pass back into the normalizer.

However, it feels like I've strayed too far from the original intention of this plugin, so closing for now until I've got a way of making it more consistent with the general purpose nature of this plugin.

Plugin config

{
      resolve: "gatsby-plugin-local-search",
      options: {
        name: "blogs",
        engine: "flexsearch",
        engineOptions: "speed",
        types: ["allStorys", "allBlogs", "allStudys"],
        query: `query allContent($after: String) {
          prismic {
            {type}(after: $after) { # note the placeholder {type}
              pageInfo {
                hasNextPage
                endCursor
              }
              edges {
                node {
                  __typename
                  _meta {
                    id
                    uid
                    tags
                  }
                  title
                  seo_description
                }
              }
            }
          }
        }
        `,
        ref: "id",
        index: ["title", "description", "tags"],
        store: ["title", "description", "tags", "type"],
        normalizer: edges => {
          return edges.map(edge => {
            return {
              id: edge.node._meta.id,
              uid: edge.node._meta.uid,
              title: edge.node.title ? edge.node.title[0].text : "",
              description: edge.node.seo_description
                ? edge.node.seo_description[0].text
                : "",
              type: edge.node.__typename
            };
          });
        }
      }
    },

Plugin change

  let result = []

  for (let c = 0; c < types.length; c++) {
    let startCursor = ''
    let hasNextPage = true
    let type = types[c]
    let q = query.replace('{type}', type)
    while (hasNextPage) {
      const queryResults = await graphql(q, { after: startCursor })
      result = [...result, ...queryResults.data.prismic[type].edges]
      startCursor = queryResults.data.prismic[type].pageInfo.endCursor
      hasNextPage = queryResults.data.prismic[type].pageInfo.hasNextPage
    }
  }
  const documents = await Promise.resolve(normalizer(result))