Implement caching functionality

[Otto-AA]

Discussion summary from https://gitter.im/solid/app-development

The cache functionality should be implemented inside a module (lets say solid-rest-cache) which stores the data in some arbitrary place. It also handles websockets to keep it up to date. the solid-rest-browser accepts app://cache/{url} requests, which it forwards to solid-rest-cache. The module checks if an up to date version is available, if yes return it, else return a new fetch(...) and store the result of it in the cache developers can use it via the solid-auth-client (or some other libraries) by prepending app://cache/ to the url and opionally setting options inside the headers

[Otto-AA]

To clarify:

• [ ] What options do we provide?
• [ ] How exactly the options are passed in the header field
• [ ] How to handle/detect non-pod data
• [ ] How the module can fetch data

[Otto-AA]

How would the module make an authenticated request, in case the requested file is not cached but requires to be logged in? I think this would require a fetch function to be passed to the module. It would be needed to be passed by solid-rest-browser which in turn would needed it to be passed by solid-auth-client/others. Is this something which seems reasonable?

[jeff-zucker]

Some things may apply to all of a user's pods so there should be a way to store in a user-wide place that doesn't go under a pod url. We should decide which things are mandated by our fetches and which are in the dev's control (it's their app's own silo so they may have other ways they want to divide it). So far it's sound like a URL is app:// + sotrageMechanism + / + pod-or-website-url + / + url ... is that how you see it? and should the pod-or-website-url be enforced and expected or should it be something a dev can require for their app's storage or not?

[jeff-zucker]

A fetch wherever it occurs (unless we specifically redirect it) shouldl always go through the chain solid-auth-client -> solid-rest-browser -> solid-rest-cache so if it's a fetch for https:// solid-auth-client will handle it with auth, So no need to pass a fetch function, or am I missing something?

[Otto-AA]

I think caching should only apply to full urls (e.g. https://example.org). If something has to be stored without a pod url, a POST request could be sent to app://lc/foo/bar.ext, which is handled differently than cache requests. From my point of view caching only makes sense with full urls, if for instance only the path is used the caching function couldn't look for newer versions.

[Otto-AA]

If we want to strictly follow the chain solid-auth-client -> solid-rest-browser -> solid-rest-cache solid-rest-cache wouldn't be able to connect to WebSockets, check for updates or fetch an item if it isn't cached. It could only serve as a store. To ensure the cache is up to date, an authenticated fetch function is essential at the solid-rest-cache level.

[jeff-zucker]

Ah, yes, I see what you mean that cache/ will require full url. So maybe solid-rest-browser should not require the full url to accommodate user-wide stuff , or do you think for consistency, it should require a URL as well?

[jeff-zucker]

I'm still missing something on the need to pass a fetch. Why can't solid-rest-cache just call solid.auth.fetch() which, since we're in the browser, should be available.

[Otto-AA]

I don't think it is necessary to restrict solid-rest-browser to full urls.

There are several reasons why I wouldn't directly call solid.auth.fetch: First of all, this would limit solid-rest-cache to only work in combination with solid-auth-client. I would rather like to make this independent of the authentication/fetch library (but requiring a fetch function which is similar to window.fetch) Secondly, I am not sure about this but I think if solid-auth-client is used inside modules it may be in a different scope than solid-rest-cache. This could happen when building via webpack/co, but also by using imports in the browser. Therefore I don't think we can rely on the function being available in our scope, even if solid-auth-client is being used. Thirdly, solid-rest-cache could also be useful in server side caching. I guess it would be possible to provide a fallback data storage in case the browser context is not available.

[jeff-zucker]

"this would limit solid-rest-cache to only work in combination with solid-auth-client" ... so you're thinking of making your own authenticated fetch separate from solid-auth-cleint's authenticated fetch? ... " I don't think we can rely on the function being available in our scope" so do require('solid-auth-client') in your package and it will inherit from the browser if its in scope and require if not

[Otto-AA]

so you're thinking of making your own authenticated fetch separate from solid-auth-cleint's authenticated fetch?

No, definitely not. The best option I currently see, is to pass the fetch function to the solid-rest-cache module. The fetch function could originate form solid-auth-client or any other module, the only limitation is that it mimics the window.fetch function but is authenticated.

so do require('solid-auth-client') in your package and it will inherit from the browser if its in scope and require if not

I am not sure if this works in combination with browser imports. For instance, if script.js looks like this:

import * as solidAuthClient from '/modules/solid-auth-client.js'
await solidAuthClient.login(...)
solidAuthClient.fetch('app://cache/https://example.org/foo/bar.ext')
  .then(console.log)

I think that solidAuthClient would be neither exposed as solid.auth, nor would it be available via require/import('solid-auth-client') (I haven't tried it out though). This may work if solid-auth-client is included via CDN and if it is compiled via a bundler*, but I don't think it works if the import is used directly in the browser (which is already supported by all major browsers).

Another argument speaking against this way is, that solid-auth-client may support being logged in with multiple accounts at the same time (I've opened a PR for it here, with a bit spare time it should be easy to finish). Using solid.auth.fetch directly wouldn't support multiple accounts, while using a passed fetch function does.

[jeff-zucker]

OK, well then you can use the fetch options to pass a fetch function. If you need it to, solid-rest-browser can pass solid-auth-client's fetch along to your package as an option.

[Otto-AA]

Passing it via the headers won't work (because only strings are valid headers), but it should be possible with an extra options object provided by solid-rest-browser and solid-rest-cache.

I'll try to write down a short prototype specification which goes a bit more into the details, so we can discuss it and then move on to the implementation. It maybe will need a few days, because I'm a bit busy with other stuff right now.

[jeff-zucker]

Sounds good. I can certainly pass an extra options object although I don't know of any reason why simply having an extra field (not the headers field) in the options of fetch(url,options) would cause problems.

[Otto-AA]

You are right, both would work. The reason I initially thought of passing an extra parameter is, that I try leave well defined things as they are instead of changing/enhancing them. The RequestInit already has well defined options, so I'd prefer to leave it like that and use another parameter. Also this makes it easier to use with Typescript if it's used.

[Otto-AA]

Something I am currently thinking about is, in how far we could reuse the existing caching mechanism of browsers.

They provide everything I intend to built, except for the WebSockets syncing (which allows sending no requests at all if we are sure it is up to date).So I think it is definitely worth to create this caching library, but I wonder if we could use the Request.cache option to reduce the amount of work we have to do.

The current approach I see:

onStartup() {
  syncedItems = {} // Defines which items are currently synced. Would be stored in localStorage/...
}
onRequest(...) {
  if (syncedItems[url]):
    return fetch(resource, { ...options, cache: 'force-cache' }) // force-cache returns the cached item if available, else makes a new request and caches it
  else {
    syncItem(url)
    return fetch(resource, { ...options, cache: 'no-cache' }) // no-cache tells the browser to send a conditional request (if changed, the server responds with the new item; if not changed the server only responds with 304 and the browser returns the cached value)
  }
}
syncItem(url) {
  // listen for changes via websocket
  // on update: delete syncedItems[url];
  syncedItems[url] = true;
}

I think if we reuse the cache option properly the only real work we have to do is handling the websocket. The rest is pretty much handled by this code. Also this wouldn't require us to store the responses ourselves, but the browser would handle it. This means it is not duplicated. Some things we still have to look into would be, whether or not the browser cache also caches the headers (e.g. permission and link header), how it handles authentication and if it is also divided per origin. These resources may be useful:

• https://developer.mozilla.org/en-US/docs/Web/API/Request/cache
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching
• https://tools.ietf.org/html/rfc7234

[jeff-zucker]

Good points, thanks for explaining.

[Otto-AA]

Here is the draft of the "specification": https://github.com/Otto-AA/solid-rest-cache/blob/master/specification.md

I still have to test several things with the fetch cache options I've mentioned before, depending on how those work out things may change regarding the storing of the cache. But apart from that, what do you think about it? Would you change something?

[jeff-zucker]

It looks good to me. Some questions: how would this be called in an app? How is the Storage object accessed? Is the default Storage object the same as localStorage?

[Otto-AA]

Here's the interface part which describes it: https://github.com/Otto-AA/solid-rest-cache/blob/master/specification.md#interface The cachedFetch method is the only exported variable, and it is used similar to the window.fetch method, except that it takes an additional cacheOptions paramter (which includes the authenticated fetch method). An example:

import { cachedFetch } from 'solid-rest-cache' // or differently if used directly in browser
cachedFetch('https://example.org/private/', {
  "Content-Type": "text/turtle",
  method: "GET"
}, {
  fetch: myAuthenticatedFetch, // e.g. solid.auth.fetch
  cacheStorage: window.localStorage
})
  .then(response => doSomething(response))
  .catch(err => handleErr(err))

I am not entirely sure how I want to implement it, but my current intention is to use up to two different stores: One for caching the responses and one for keeping a list of all items which are currently synced. The store for caching the responses defaults to the internal browser cache (I wouldn't manually store it, but leave it to the fetch request with the cache headers set appropriately ). The store for the synced items list would default to localStorage (if available). Currently the user could change the caching store by providing a storage (like localStorage) to the cacheStorage option.