Revisit Gatsby Incremental Builds on Netlify

[jeancochrane]

Background

Gatsby has a new feature called Incremental Builds where it can regenerate your app by only updating things that have changed. Netlify in turn supports this feature through its own new feature, Netlify Builds, that allows you to configure plugins for your builds.

Proposal

I'd like to test out Incremental Builds on Netlify by following Netlify's documentation. I'm going to do this by updating https://github.com/datamade/lisc-cnda/ and https://github.com/datamade/lisc-cnda-map, two projects that have had some trouble in the past with builds taking too long and erroring out.

Deliverables

• Pull requests to LISC CNDA repos using Incremental Builds
• Some documentation on how to configure Incremental Builds on Netlify

Timeline

I expect this to take one R&D day.

[jeancochrane]

So incremental builds are definitely faster, which is cool to see. Here's an example of a minute-long Gatsby build that gets cut down to 23 seconds with an incremental build.

The bad news is that incremental builds don't reliably work on deploy previews, which is kind of a deal killer. I tried a couple of different types of commits in https://github.com/datamade/lisc-cnda/pull/90 and each time the Netlify build reported No Gatsby cache found. Building fresh.

The worse news is that even if we could get incremental builds to reliably work on deploy previews, it seems like incremental builds only work for content (AKA data) changes, not code changes. From the docs:

This will run the Gatsby build process, but only build pages that have data changes since your last build. If there are any changes to code (JS, CSS) the bundling process returns a new webpack compilation hash which causes all pages to be rebuilt.

This means incremental builds would really only be helpful for a small proportion of commits, unless we're working with a CMS. For our current development patterns, I don't see this being a big time saver.

The good news, however, is that something about build speed and reliability has improved enormously in the latest versions of Gatsby and Netlify. After updating to the latest version of gatsby, gatsby-image, and gatsby-transformer-sharp, our LISC CNDA builds went from ~3 minutes to ~1.5 minutes and all built reliably. So I think the takeaway here is that we should make sure to upgrade both LISC CNDA apps, and use the latest version of Gatsby and its plugins in new apps.

[kaganjd]

👋🏻 found this issue linked to https://github.com/jlengstorf/netlify-plugin-gatsby-cache/issues/22, so wanted to follow up and let you know that caching for Deploy Previews has been fixed/implemented! Check out the Community post for more details: https://community.netlify.com/t/build-cache-is-now-updated-in-deploy-preview-builds/26344

[jeancochrane]

Thanks for letting us know @kaganjd! @hancush This definitely makes Incremental Builds more viable for DataMade's CI workflow but I think the core problem remains that it only works for content updates and not code updates. That might change if #89 ever comes in but for now it's hard for me to see how it would be useful.

[smcalilly]

@hancush @derekeder do you know if we can close this, and if not, what are the next steps we need to take?

[hancush]

I think we can close this, since we are likely going to move away from Gatsby.