natemoo-re
commented
1 year ago Ideally, this is something that could be solved generically on the Vite / Rollup level so that every framework could benefit from this. I'm really not sure if that's on the table, though, since the ultimate goal is to bypass Vite / Rollup as much as possible. If this was easy to solve incremental builds in a generic way, it would have been done already.
My current sketch for an API is very straightforward from the user's perspective:
// astro.config.mjs
import { defineConfig } from 'astro/config'
export default defineConfig({
build: { incremental: true },
experimental: { incremental: true } // until this is stable
})Unfortunately that's where the simplicity ends. To implement this, we'll likely need to:
- Generate a serializable module graph that contains every possible build input. This will need to track all module relationships (hence the "graph" part).
- Given the module graph, generate a stable checksum for each file. This will allow us to determine which parts of the graph have changed.
- On every build, we'll need to generate a new module graph and compare it to the old one. Use the checksum to determine which files need to be invalidated (changes bubble all the way up to an entry point). If any subtree of the module graph has changed shape, that also invalidates relevant portions of the module graph.
- Pass any invalidated modules as inputs to Vite. Thankfully Astro already controls every Vite input!
- As Vite generates new outputs for invalid parts of the graph, we can restore valid parts of the graph from our cache (likely in
node_modules/.cache/astroornode_modules/.astro).- Ideal scenario: the entire module graph is valid, so the entire output is restored.
- Worst case scenario: the entire module graph is invalid, so the entire output needs to be generated from scratch. This is currently what we do on every build.
- Any invalidated modules should be removed from the existing cache.
- Now that out output has been merged into a repaired state, execute it to prerender our
.htmlfiles. (We can't skip directly to restoring the.htmlfiles because the.jsoutput can depend on external data that we don't know about.) - Populate the cache with our output for next time.
- Remove the chunks needed for prerendering from our
distfolder
EyePulp
fparedlo
heyitsdoodler
ImBIOS
Summary
Incremental Build support in Astro aims to significantly speed up repeated runs of
astro build. Initial builds will populate some kind of cache, which will allow subsequent builds to bypass Rollup for unchanged trees of the module graph.Background & Motivation
The original incremental build support proposal is one of our oldest and most highly upvoted issues. Unfortunately, users have not shared many concrete examples in that thread. However, from the extensive performance profiling we've done, we know that Rollup is the main bottleneck in Astro's build process.
Now that our rendering, Markdown, and MDX performance has been optimized about as far as we can take it, now is the time to explore new options. The best option we have is to move as much of our build process out of Rollup as possible, likely with some sort of build cache. That is the goal of introducing incremental builds.
Why now? The reason is straightforward—caching is notoriously difficult to get right. We did not want to take on the additional complexity until our API surface was stable and our internal build process was easier to reason about. Thanks to significant effort from @bluwy and @ematipico in the past few months, we're now in a good place to tackle this.
Goals
astro buildperformanceastro buildas possiblestatic,hybrid, andserveroutputsNon-Goals