Question: How to host a (read-only) provider?

[jaller94]

How can I host my own read-only dat-store provider that others can subscribe to without a login using dat-store set-provider [url] [name]?

The ReadMe only shows how to use hasbase.io accounts as a provider, but i wish to host one myself.

Background

The Dat Installer for Android is a pretty cool project and I hope that dat:// becomes an option in the F-Droid client at some point. To see how well (or badly) this would work, I plan to mirror the F-Droid repo on the dat protocol. That's roughly 1800 archives to keep track of and serve on at least one peer each. To allow my servers and others to subscribe to all archives with one command, I am looking to set up a dat-store provider.

[RangerMauve]

I've been thinking of having dat-stores expose a dat URL which contains a JSON file with all their archives.

From there applications could subscribe to changes in the file and build up a sort of federation of stores that store each others data.

Does that sound like something that'd fit your use case?

Another approach here would be to make use of mounts. Essentially you'd create an archive that represents your registry, and mount other people's archives as folders within it. From there you can just store the root archive and the others will be stored as a result.

This would require the new Dat 2.0 features though that come in the new Hyperdrive version. I'm currently working on adding those to the Dat SDK which should come out towards the end of the month. The dat installer will need to be updated to support the new version of Dat, too.

[jaller94]

Thanks for your ideas and pointing out possible features dat-store may see in the future. For my case both solutions would do the trick. Version 1 seems easy to implement with a little bash script within a few hours.

[RangerMauve]

I'll leave this open for now to keep track. 😁

[anotherjesse]

@RangerMauve are you saying potentially:

• we create a primary hyperdrive, and add it to our dat-store
• and any mounted hyperdrives we add to primary would be peered by our dat-store instance as well?

I was looking at what happens right now (I have a git clone of this repo and running from it)

• after I add primary to dat-store, the hyperdrive has a peer reported (in beaker 1.0)
• mounts - both created before adding and after adding don't report any peers (in beaker 1.0)
• the mounted hyperdrives don't show up in the bin.js list output

Interestingly:

• once I access the mount via the http gateway of dat-store (using curl), a peer shows up with my dat-store instance IP (in beaker 1.0)
• this peering of visited mounts seems to persist stopping and stopping of the dat-store service
• the mount hyperdrive urls still aren't added to the list of hyperdrives in the bin.js list

---

I had been building a small cron job that walked my primary hyperdrive to add mounts (using bin.js add or requesting the appropriate path in dat-store http gateway)

Given that requesting a mount via the gateway seems to trigger peering, I'm wondering if this might be easy to add to dat-store directly?

[RangerMauve]

@anotherjesse As I understood the mounts should have been loaded when dat-store invoked download() on the primary drive. If that's not happening I think either download() isn't being invoked, or something else fishy is going one.

One thing to note is that even if the mounted drives are loaded, they won't necessarily be peering for their discovery keys because dat-sdk only peers on hyperdrive keys that are loaded at the top level.

So if your peer loaded both the primary drive then they could reuse that connection to peer on the mounted drive.

This might be a bug and it could be good to open an issue on dat-sdk to start finding peers on mounted hyperdrives when they're detected.

The reason the mounted drives don't show up in dat-store list is that dat-store is only tracking the top level primary drive URL and only listing it's index.json. It doesn't do anything special for listing mounted drives at the moment.