Service worker cache as transparent dat proxy

[100ideas]

Hey, I read your blog post dat-in-browsers-without-a-gateway.md and saw your suggestion of using service workers to proxy dat over ws (and also local cache in indexdb).

This is a great idea!

Last year I put together a rough prototype to play around with using sw this way. I guess I didn’t publish the repo, so in the meantime before I do I just wanted to share a little about how it worked:

• ran a local dat service and an express-ws server to replicate it over websockets
• used multiwriter-enabled hyperdb to allow bidirectional sync: browser dat archive (ram or indexdb I forget) replicating <—> random-access-network/ws archive(browser) <—> random-access-network/ws archive (express-ws server) replicating <—> fs dat archive (‘.dat’ folder)
• I avoided using ‘librarian’ or ‘swarm’- related tools because I wanted to see if I could get it working all on local network
• i did’t figure out an effective way to mediate the mulitiwriter key exchange bw browser and server - I just ended up editing the value in idb in browser devtools, or hard coding it
• all objects must be serialized to pass between sw and browser global context. There are three diff message-passing / subscription apis for sw-browser communication that are subtly different... I think only one of them is able to do structural sharing when passing large data objects. I had trouble getting this to work sw->browser (couldn’t figure out how to set up message interface on browser to trigger on sw initiated messages, but the converse was trivial)

My goal was a little different from the one you outlined: I wanted to support static html files that referenced local file assets in relative directories such that everything would load properly without changing the dom if the directory containing the html file(dat archive folder) was served locally by a static http file server. If dat was enabled, I wanted to use the service workers cache to intercept browser fetches for linked resources and serve them from dat. In this way the dat-sw could act almost like a transparent cdn without requiring any source modification (maybe).

I got it partially working but decided to focus on some other projects and come back later when hypercore and multiwriter were more mature. I think using the sw cache is a clever idea but it would probably be easier to instead use a browser-side script to modify the dom to use dat urls directly and register these dependencies with the worker as needed, as you suggest.

I will try to post my src later this week in case you are interested.

I think the thing I find most compelling about this concept in general is imagining a single html file with embedded JavaScript that is able to initialize dat on page load and resolve + sync files and data, perhaps after being unlocked with a passphrase. Maybe some of the imgs and linked files are also included in a lower res or compressed form online in the html - perhaps even as a serialized compressed mini dat “bootstrap” archive. The whole html file could be kept less than 5 megabytes I bet. This could allow some really novel interactions around document collaboration and sharing - just save the page from the browser, email it to a coworker, host it in Dropbox or github, it even works as an imessage attachment.

Food for thought :)

[RangerMauve]

That sounds like a really cool project, I'd love to see the source!

What did the HTTP request URLs look like? I wasn't sure if service workers would be able to intercept dat:// URLs directly. Have you checked out the work in dat-js over the past few months?

[100ideas]

https://github.com/100ideas/hypercore-protocol-proxy-test

I banged on this for a bit today but overall it's probably my ugliest code base. just wanted to get it up for posterity.