Cache offline an entire tree (or arbitrary routes) on user interaction

[queenvictoria]

I have offline caching working fairly well by following this post.

Now what I need to do is precache a set of routes (in this case all the descendents) without the visitor needing to visit every one individually (in some cases 100s of routes and media files).

I haven't been able to find or understand how to interactively set an offline cache. Can someone please point me in the right direction?

[queenvictoria]

I've implemented this using a custom worker (worker/index) that accepts messages sent to the worker from the front end.

Front end sends an action name (CACHE_ADD or CACHE_REMOVE) and a path:

  const wb = new Workbox('/sw.js')
  wb.register()

  return await wb.messageSW({ action, args })

Worker:

self.addEventListener('message', (event: ExtendableMessageEvent) => {
  // fetch and cache path or remove cache path
  // all done manually to match next-pwa's caching patterns
}

I also had to add a fetch handler in the service worker specifically for images using Next's <Image /> as even though the images were precached and seemed to have good cache-control they were unreliable when offline. The fetch handler just checked the cache and returned what it found.

[jankocian]

I'm about to tackle the very same issue.

I'd like to have multiple PWAs on the same site - and as the pages include a bunch of multimedia content, buildtime precache list does not seem to be a good option.

Would you mind sharing bits of your code? I'm interested especially in recreating next-pwa's patterns...

Thanks! :)

[queenvictoria]

Hi @jankocian

Please see the worker functions below. It calls a library that knows about the caching that is also utilised by other parts of the app.

Not shown here are the front end patterns for caching entire unvisited trees. That is dependent on your application structure.

For example I have a button on each row of a content listing. Clicking that button fetchs a data tree from the server, iterates all the paths and assets, and calling addToCache on each with the path. addToCache calls fetch() on each path and, if sucessful, caches the response.

I also make use of inCache in the front end. For example on each route on the above content listing page I call inCache and add a mark to indicate whether it is cached or not.

All of the front facing logic is handled by a different library that knows how to talk to the service worker, the caches, and knows about the data model of the app. It can therefore do things like calculate how much of a tree is cached and how much space on disk that occupies.

Hope that helps.

Two worker listeners:

1. message for receiving commands from the front end (cache add and delete)
2. fetch for intercepting requests for data from the front end before it goes to the server.

// worker/index.ts

/// <reference lib="webworker" />
export default null
// https://github.com/shadowwalker/next-pwa/blob/master/examples/custom-ts-worker/worker/index.ts

import { CMD_ADD, CMD_DEL } from '../lib/constants'
import { addToCache, removeFromCache } from '../lib/cacheHandler'

const SW_VERSION = '1.0.0'

declare let self: ServiceWorkerGlobalScope

/**
 * Subscribe to all messages. Respond to our commands.
 */
self.addEventListener('message', (event: ExtendableMessageEvent) => {
  const { action, args } = event?.data
  if ( !action ) {
    console.log(`++++ This message is not for us`, event)
    return
  }

  // If it's a cache action send to the cacheHandler.
  if ( args?.path && action === CMD_ADD ) {
    return addToCache(args.path)
      .then(res => event?.ports[0].postMessage(res))
      .catch(err => event?.ports[0].postMessage(err))
  }
  else if ( args?.path && action === CMD_DEL ) {
    return removeFromCache(args.path)
      .then(res => event?.ports[0].postMessage(res))
      .catch(err => event?.ports[0].postMessage(err))
  }
  else if ( action === 'GET_VERSION' ) {
    event?.ports[0].postMessage(SW_VERSION)
  }
})

/**
 * Default fetch listener. Does this override the next-pwa one?
 * We've implemented this one as, even though we seem to be caching images in a
 * very similar way, next-pwa doesn't seem to be returning them. So let's do it
 * ourselves.
 */
self.addEventListener('fetch', event => {
  if ( ! event ) return

  const url = event && new URL(event.request.url) || null
  if (url?.pathname === '/_next/image' ) {
    // https://developer.mozilla.org/en-US/docs/Web/API/FetchEvent/respondWith#examples
    event.respondWith(
      (async () => {
        // Try to get the response from a cache.
        const cachedResponse = await caches.match(url.href);

        // Return it if we found one.
        if (cachedResponse) return cachedResponse

        // If we didn't find a match in the cache, use the network.
        return fetch(event.request)
      })()
    )
  }
})

The library called from the worker:

1. addToCache - what it says though it won't override the existing cache
2. removeFromCache - what it says
3. openCache - selects which cache to use based on the path. This allows us to either mimic or vary the next-pwa pattern of caches (which are not always consistent).
4. inCache - check if a path is already cached.

// lib/cacheHandler.ts

/**
 * Utilities to be used by the service worker only.
 *
 * This should not need to know about trees or the structure of the data.
 * Might need to know about the build ID to correctly store next-data routes.
 * Should we correct the next-data routes (IE some runtime next_data routes
 * are stored in `others` rather than `next-data`).
 */

/**
 * Add a single path to the cache if it doesn't already exist.
 * Doesn't seem to respect the workbox configuration for limits (which might be
 * ok).
 *
 * Headers: next-pwa sets with Cache-Control: s-maxage=31536000, stale-while-revalidate
 * Unless we specify we get Cache-Control: no-store, must-revalidate
 *
 * @param path {String} - A path to consider.
 */
export async function addToCache(path: string) {
  if ( await inCache(path) ) return

  // Does this need location.host?
  const response = await fetch(path)
  if (!response.ok) {
    throw new TypeError('Bad response status')
  }

  const _response = response.clone()

  // @TODO Update the headers
  // This is not working.
  // _response.headers.set("Cache-Control", "s-maxage=31536000, stale-while-revalidate")
  // CacheableResponse?
  // https://developer.chrome.com/docs/workbox/reference/workbox-cacheable-response/

  const cache = await openCache(path)
  return await cache.put(path, _response)
}

export async function inCache(path: string) {
  const cache = await openCache(path)
  const cachedResponse = await cache.match(path)

  return cachedResponse?.ok ? true : false
}

/**
 * @FIX Arbitrate which cache it should be in and the format the data should
 * take.
 * @TODO Don't hardwire the arbitration. Either examine the caches and decide
 * automagically or figure out how the path can do it itself. These are defined
 * by regexs in workbox. Can we pull them out?
 *
 * @param path
 * @returns
 */
export async function openCache(path?: string) {
  let key:string = 'others'

  // @FIX Switch reusing the regexs in workbox
  if ( path?.includes("/_next/image?url") ) key = 'next-image'
  else if ( path?.includes("/_next/data/") ) key = 'next-data'

  return await caches.open(key)
}

/**
 * Remove a single path to the cache if it exists.
 *
 * @param path {String} - A path to consider.
 */
export async function removeFromCache(path: string) {
  if (!await inCache(path)) return
  const cache = await openCache(path)

  return cache.delete(path)
}

[queenvictoria]

As someone was asking about images and other static assets I have another method that might be interesting to the reader. Also I include my cache.js below which handles the CSS/JS caching. It isn't triggered manually: just on page load. But it could be done using the openCache key patterns above (I imagine) possibly requiring the extension of the addEventListener method below to respond in special cases.

In the service worker I listen for image requests and check our cache for them (this is React/NextJS BTW): cacheHandler.ts

/**
 * Default fetch listener. Does this override the next-pwa one?
 * We've implemented this one as, even though we seem to be caching images in a
 * very similar way, next-pwa doesn't seem to be returning them. So let's do it
 * ourselves.
 */
self.addEventListener('fetch', event => {
  if ( ! event ) return

  const url = event && new URL(event.request.url) || null
  if (url?.pathname === '/_next/image' ) {
    // https://developer.mozilla.org/en-US/docs/Web/API/FetchEvent/respondWith#examples
    event.respondWith(
      (async () => {
        // Try to get the response from a cache.
        const cachedResponse = await caches.match(url.href);

        // Return it if we found one.
        if (cachedResponse) return cachedResponse

        // If we didn't find a match in the cache, use the network.
        return fetch(event.request)
      })()
    )
  }
})

cache.js

'use strict'

// Workbox RuntimeCaching config: https://developers.google.com/web/tools/workbox/reference-docs/latest/module-workbox-build#.RuntimeCachingEntry
module.exports = [
  {
    urlPattern: /^https:\/\/fonts\.(?:gstatic)\.com\/.*/i,
    handler: 'CacheFirst',
    options: {
      cacheName: 'google-fonts-webfonts',
      expiration: {
        maxEntries: 4,
        maxAgeSeconds: 365 * 24 * 60 * 60 // 365 days
      }
    }
  },
  {
    urlPattern: /^https:\/\/fonts\.(?:googleapis)\.com\/.*/i,
    handler: 'StaleWhileRevalidate',
    options: {
      cacheName: 'google-fonts-stylesheets',
      expiration: {
        maxEntries: 4,
        maxAgeSeconds: 7 * 24 * 60 * 60 // 7 days
      }
    }
  },
  {
    urlPattern: /\.(?:eot|otf|ttc|ttf|woff|woff2|font.css)$/i,
    handler: 'StaleWhileRevalidate',
    options: {
      cacheName: 'static-font-assets',
      expiration: {
        maxEntries: 12,
        maxAgeSeconds: 365 * 24 * 60 * 60 // 365 days
      }
    }
  },
  {
    urlPattern: /\.(?:jpg|jpeg|gif|png|svg|ico|webp)$/i,
    handler: 'StaleWhileRevalidate',
    options: {
      cacheName: 'static-image-assets',
      expiration: {
        maxEntries: 1024,
        maxAgeSeconds: 365 * 24 * 60 * 60 // 365 days
      }
    }
  },
  {
    urlPattern: /\/_next\/image\?url=.+$/i,
    handler: 'StaleWhileRevalidate',
    options: {
      cacheName: 'next-image',
      expiration: {
        maxEntries: 1024,
        maxAgeSeconds: 365 * 24 * 60 * 60 // 365 days
      }
    }
  },
  {
    urlPattern: /\.(?:mp3|wav|ogg)$/i,
    handler: 'CacheFirst',
    options: {
      rangeRequests: true,
      cacheName: 'static-audio-assets',
      expiration: {
        maxEntries: 32,
        maxAgeSeconds: 365 * 24 * 60 * 60 // 365 days
      }
    }
  },
  {
    urlPattern: /\.(?:mp4)$/i,
    handler: 'CacheFirst',
    options: {
      rangeRequests: true,
      cacheName: 'static-video-assets',
      expiration: {
        maxEntries: 32,
        maxAgeSeconds: 365 * 24 * 60 * 60 // 365 days
      }
    }
  },
  {
    urlPattern: /\.(?:js)$/i,
    handler: 'StaleWhileRevalidate',
    options: {
      cacheName: 'static-js-assets',
      expiration: {
        maxEntries: 32,
        maxAgeSeconds: 24 * 60 * 60 // 24 hours
      }
    }
  },
  {
    urlPattern: /\.(?:css|less)$/i,
    handler: 'StaleWhileRevalidate',
    options: {
      cacheName: 'static-style-assets',
      expiration: {
        maxEntries: 32,
        maxAgeSeconds: 24 * 60 * 60 // 24 hours
      }
    }
  },
  {
    /**
     * @queenvictoria
     * - Starts with /_next and allowing query strings (as that is common in
     *   contemporary NextJS)
     * - Removed ^ starts with (as it probably starts with http...)
     **/
    urlPattern: /\/_next\/data\/.+\/.+\.json.*$/i,
    // urlPattern: /\/_next\/data\/.+\/.+\.json$/i,
    handler: 'StaleWhileRevalidate',
    options: {
      cacheName: 'next-data',
      expiration: {
        maxEntries: 256,
        maxAgeSeconds: 24 * 60 * 60 // 24 hours
      }
    }
  },
  {
    urlPattern: /\.(?:json|xml|csv)$/i,
    handler: 'NetworkFirst',
    options: {
      cacheName: 'static-data-assets',
      expiration: {
        maxEntries: 256,
        maxAgeSeconds: 24 * 60 * 60 // 24 hours
      }
    }
  },
  {
    urlPattern: ({ url }) => {
      const isSameOrigin = self.origin === url.origin
      if (!isSameOrigin) return false
      const pathname = url.pathname
      // Exclude /api/auth/callback/* to fix OAuth workflow in Safari without impact other environment
      // Above route is default for next-auth, you may need to change it if your OAuth workflow has a different callback route
      // Issue: https://github.com/shadowwalker/next-pwa/issues/131#issuecomment-821894809
      if (pathname.startsWith('/api/auth/')) return false
      if (pathname.startsWith('/api/')) return true
      return false
    },
    handler: 'NetworkFirst',
    method: 'GET',
    options: {
      cacheName: 'apis',
      expiration: {
        maxEntries: 16,
        maxAgeSeconds: 24 * 60 * 60 // 24 hours
      },
      networkTimeoutSeconds: 10 // fall back to cache if api does not response within 10 seconds
    }
  },
  {
    urlPattern: ({ url }) => {
      const isSameOrigin = self.origin === url.origin
      if (!isSameOrigin) return false
      const pathname = url.pathname
      if (pathname.startsWith('/api/')) return false
      return true
    },
    handler: 'NetworkFirst',
    options: {
      cacheName: 'others',
      expiration: {
        maxEntries: 1024,
        maxAgeSeconds: 24 * 60 * 60 // 24 hours
      },
      networkTimeoutSeconds: 10
    }
  },
  {
    urlPattern: ({ url }) => {
      const isSameOrigin = self.origin === url.origin
      return !isSameOrigin
    },
    handler: 'NetworkFirst',
    options: {
      cacheName: 'cross-origin',
      expiration: {
        maxEntries: 32,
        maxAgeSeconds: 60 * 60 // 1 hour
      },
      networkTimeoutSeconds: 10
    }
  }
]