search

algolia / gatsby-plugin-algolia

A plugin to push to Algolia based on graphQl queries
https://yarn.pm/gatsby-plugin-algolia
Apache License 2.0
178 stars 45 forks source link

Use the `contentDigest` for Node's to determine which nodes to index. #161

Closed KyleAMathews closed 2 years ago

KyleAMathews commented 2 years ago

I was just looking through your README and noticed your https://github.com/algolia/gatsby-plugin-algolia#partial-updates — the simplest way to do this would be to compare the contentDigest from previous versions of nodes to the current as that's what Gatsby internally does to figure out which nodes have changed or not so it's always going to be correct.

You could automatically add contentDigest to queries and give users automatic partial updates for free!

I see a lot of sites using this plugin so thanks and keep up the good work!

Haroenv commented 2 years ago

As the user writes the query themselves, would you be thinking of somehow changing the query to add the contentDigest, or to recommend using that as the matchFields by default in the readme/code?

Thanks for the suggestion @KyleAMathews!

KyleAMathews commented 2 years ago

Yeah, you should rewrite the query to make sure it's always returning the contentDigest. And then cache the node's id/contentDigest so that you can compare it with the next build. Then you can automatically only send data for nodes that are either a) new or b) changed. This would make indexing really fast most of the time as most builds only have ~1-10 nodes that have changed. I'm just now looking at a site on Gatsby Cloud that takes ~100s on every build to index to Algolia 😱

Haroenv commented 2 years ago

Is there some way in Gatsby to transform the query? Trying to parse the query would be much too much work. If you have any idea how it could be implemented, the plugin is fairly simple, so I can review a PR you make @KyleAMathews :)

KyleAMathews commented 2 years ago

graphql-js has the utilities to parse & modify the query — https://graphql.org/graphql-js/language

You can see it in action here: https://astexplorer.net/#/gist/6741b9816f2a912253551d349096621d/5c732d8915bdc59ed75ecb6fd1838ea713a1478c

Happy to pair with you sometime on the implementation! I'd love to help y'all speed this up sooner than later as it's a slower part of many people's builds.

KyleAMathews commented 2 years ago

Here's a little gist showing the basics of rewriting a query https://gist.github.com/KyleAMathews/a7342ebad8f8f0ea6938f61649358248

KyleAMathews commented 2 years ago

Closed by https://github.com/algolia/gatsby-plugin-algolia/pull/162 — thanks @Haroenv !