Cache-busting issues with static assets

[robertknight]

Note: This issue was originally filed against the h project, before we extracted the h-assets package and started using it across different projects.

Current issues with static assets and cache busting

• Static asset URLs use query strings as cache busters. These are used when generating the asset but ignored when serving it. Consequently, it is possible that client may request 'foo.js' version 'v1' via a request for '/assets/foo.js?v1' but actually get a newer version 'v2'. This is a problem for Sentry if it tries to process an exception report for Hypothesis release v.X after Hypothesis v.X+1 has been released.
• Sourcemap URLs do not include cache-busting tokens, so Sentry may end up trying to process an exception for '/assets/foo.js?v2' by requesting '/assets/foo.js.map' and actually end up using a cached sourcemap (where the sourcemap has been cached either by Sentry itself, or CloudFlare) for an earlier release.
• Static assets are served with a short cache expiration of 4 hours, despite the fact that assets are immutable (provided they are cache-busted). This is a small inefficiency that may lead clients to download resources they already have.
• Not all resources are served with cache-busting tokens in the URL. eg. SVG images are currently served without. This may lead to users seeing old versions of assets. Fixed

Proposed changes

• Fix manifest.json generation in client and server to include all assets (see https://github.com/hypothesis/h/pull/4106#issuecomment-262760621)
• Fix image asset serving to include cache-busting tokens (see https://github.com/hypothesis/h/pull/4106#issuecomment-262760621)
• Modify manifest generation and asset serving to use path rather than query-string cache busting tokens. This will require either a custom static view callable in the Python code to map the cache-busted URL to the path on disk or a tween to map the request's subpath to a version without the cache busting token. See https://github.com/Pylons/pyramid/issues/2675#issuecomment-230544321
• Rewrite sourcemap URLs in JS bundles to use cache-busted URLs
• Set the cache_max_age parameter in the static_view to a large value so that clients do not re-download assets that they already have.

[robertknight]

Status update for May 2023: The code for serving static assets was extracted into the h-assets package. That still uses query strings as cache busters, but it does validate them and return a 404 if the cache buster query string is invalid.

This resolves the first problem where Sentry could end up receiving a bug report for version X and end up fetching the static asset for a later release Y.

However this problem is still open:

Sourcemap URLs do not include cache-busting tokens

The challenge here is that the sourcemap URL is part of the JS / CSS bundle generated by Rollup or PostCSS respectively.