Preloading gzip archives

morris commented 3 years ago

Hi everyone,

I'm unsure if this has been discussed before, but I couldn't find any similar issue or thread. I'd like to share an idea that could possibly eliminate the need for JavaScipt bundling.

I believe the overhead of JavaScript bundling is a severe, real problem in our industry and not a necessity we as authors should just accept as a given. While CSS minification and concatenation is comparably simple, the complexity of JavaScript bundling is non-trivial and has spawned a whole ecosystem of tooling, and comes with hard problems like code splitting.

A solution may be preloading with HTTP/2 (under the assumption that the latter is broadly available). However, preloading with HTTP/2 seems to be notably inferior to concatenation/bundling. Here are some interesting articles and benchmarks:

In particular, when transferring many small files, it's impossible to achieve a compression ratio comparable to sending a single, compressed, concatenated bundle. Thinking about compression lead me to think about a different approach:

Consider a complex web application built from a thousand individual assets, which is not uncommon nowadays considering NPM and other ecosystems. Without bundling, the application's HTML would likely reference some subset of its assets through <link rel="preload|modulepreload"> as early as possible. From the observations made above this is sub-optimal compared to bundling.

What if we could create gzip archives from (groups of) our assets, including directory structure, and provide these archives to browsers as preloadable resources? Browsers could preload them (with significantly greater compression ratios than individual files) and resolve future requests similarly to the way <link rel="preload"> works, using the directory structure from the archives as a mapping to URLs.

An example implementation could look something like this:

<!DOCTYPE html>
<html>
  <head>
    <link rel="archivepreload" href="/preload/vendor.gz" />
  </head>
  <body>
    <script type="module">
      import * as foo from "/scripts/vendor/foo.js";
      import * as bar from "/scripts/vendor/bar.js";
      // ...
    </script>
  </body>
</html>

... where /preload/vendor.gz contains

/scripts/vendor/foo.js
/scripts/vendor/bar.js
... and all their imported modules in a directory structure matching the assets' original URLs.

It's just an example to illustrate the idea; there is likely a better syntax/construct for this!

Things like pathing, MIME types, race conditions, caching, security, etc. likely need lots of thought here. I still believe this approach could provide real-world value:

The process and complexities of bundling could be reduced to gzipping.
This could actually be even more performant than bundling, as current bundling almost always introduces some sort of runtime which would be obsolete with archive preloading.
It's not necessarily restricted to scripts and could be used for any type of asset.
It gives a lot of freedom when designing the archives in an optimal way as opposed to e.g. code splitting.
It can be introduced in a backwards-compatible way (required that authors still provide access to the individual assets).
gzip is well understood and broadly available in browsers already.

I want to stress that JavaScript bundling is not a small problem. It has made it's way into virtually any modern tool chain, and subsequently into job descriptions, and has a very high total cost across our industry.

It might be time to think about a standard solution. I believe the outlined approach could eliminate an entire class of problems and save many hours of work for authors in the future.

If there's enough interest, I'd be very happy to flesh this out some more. Looking forward to a feedback and discussion, and thanks for reading!

domenic commented 3 years ago

Are you familiar with web bundles? See https://github.com/WICG/webpackage for more information. They tackle this problem including many of the things you mention "need a lot of thought", e.g. they work with any HTTP response (not just scripts). That repository is probably the best place to discuss these subjects.

I'm not sure what the latest is on using these to fulfill JavaScript bundling-type use cases; my colleague @jyasskin may be able to point you to something there. In particular the spec that I thought would have this outlined, https://wicg.github.io/webpackage/loading.html#subresource-substitution , seems to only focus on signed exchange prefetching, not pure (unsigned) web bundle prefetching.

morris commented 3 years ago

@domenic amazing, thank you, never heard of it. And sorry if I picked the wrong place to share this, maybe some more research was in order ;)

jyasskin commented 3 years ago

There are outlines in https://github.com/WICG/webpackage/blob/master/explainers/subresource-loading.md and https://www.notion.so/Same-Origin-Web-Bundles-1226117dca7c4aaf8e2d3a2df7171dea but no specification yet.

domenic commented 3 years ago

Closing, as that seems like the right repository for this discussion, but thanks @jyasskin for the pointer, and I'm glad we could route you to the right place, @morris!

whatwg / html

Preloading gzip archives #6147