kumavis
commented
9 years ago Reimagining XmlHttpRequest as a generic data request interface -- Its a nice hack and I think you'd get pretty decent mileage out of it.
Maybe something like https://github.com/mafintosh/tar-fs would help with moving cache between machines. After stripping the protocol, Urls would map well to directories. Something to try, and export should be super easy as its already a tar ball.
We'd also want to load some url matching into the XmlHttpRequest so that multiple cdn servers can be seen as one (e.g.: *.mapbox.com)
as for replacing the XMLHttpRequest. Browserify transform may not be what we're looking for.
I would try overwriting it at first global.XMLHttpRequest = require('./myXhr.js') in your entry point file before requiring any other dependencies. Works in chrome, but not sure if this will in a chromeapp, which are more locked down. Worth testing before we throw all our chips in on an approach. Remember mapfilter already builds as a chromeapp so it should be easy to test it out.
Offline is important for a lot of our tools, and I've been experimenting with digidem/html5-image-cache caching images dynamically in local storage (although I recently found that a lot has already been done here and here)
For data that is dynamically loaded via xmlHttpRequest that is another challenge. To date I had approached it starting to write a custom Backbone sync adapter with js-git, but this is a complex approach. Importantly, what about external libraries that load data, like vector tiles in Mapbox-gl-js (a lightweight way to provide image tiles) or deforestation API data from CartoDB Torque?
For much of this data all we need is the ability to cache the data and potentially have the ability to "side-load" it via a USB rather than an internet connection.
My idea is to write a wrapper for XmlHttpRequest that works similarly to the image cache script: when called the first time it will save to a cache, when called a second time it will load from the cache. That way it could be transparent to any libraries like mapbox or cartodb that we use. We save all requests with a hash of the url, so we always get the same data at the same URL.
Both Mapbox-gl-js and torque use browserify, so we could write a transform to replace XmlHttpRequest with our own version, wrapping the original version and adding a caching layer.
We could also use this for our own apps, whether we use backbone or angular. We just agree that only read offline matters, then just use, for example, the standard Backbone.sync but cache at the XHR layer rather than the Backbone.sync layer.
To populate the cache, we would either need to visit everything with a connection, or call the relevant functions in each library to download the tile we want.
mbtiles would be a good format for side-loading. We would need to write a library to download the needed tiles and save to mbtiles, and have a standard way to encode the original urls into the mbtiles package. That way we could populate the cache of the webapp with the correct url-hashes as the ids, so that as far as the xmlHttpRequest wrapper is concerned, the cache has come from a previous request.
Thoughts on this? Task list:
Thoughts?