Hypercore API

[pfrazee]

Starting an issue for a Hypercore API.

Questions:

• What should it be called? Should it follow the Dat* naming scheme like DatArchive? Something like DatCore or DatFeed?
• How should it work? Should it be its own browsery promises style like with DatArchive, or should it try to match the nodejs hypercore API?

cc @RangerMauve

[RangerMauve]

ping @allain

I like DatFeed since it's more obvious that it's a feed of events, but DatCore might make more sense to the power users that are likely to adopt it first.

I think that matching the hypercore API, even though it feels kinda gross as a Web API, would make it easy for developers. Alternately, maybe we could see if we can come to consensus of having a more "Webby" API for hypercore in Node.

Regardless of what the API is, I'd like to focus on getting a polyfill for both it and DatArchive that people can include in web apps to make sure the same code works in legacy Browsers as it does in Beaker.

[allain]

If it exposes a hypercore, which may or may not be a hyperdrive, slapping a Dat prefix on it might confuse.

I think calling it CoreFeed, or just Feed exposes more of the intent.

My 2 cents.

[pfrazee]

I personally prefer a promises-based API, but I know that's currently a pain for people on compatibility. I would also be willing to do an API that supports both callbacks and promises.

Here's the API in the current hypercore readme:

var feed = hypercore(storage, [key], [options])
feed.append(data, [callback])
feed.get(index, [options], callback)
feed.getBatch(start, end, [options], callback)
feed.head([options], callback)
feed.download([range], [callback])
feed.undownload(range)
feed.signature([index], callback)
feed.verify(index, signature, callback)
feed.rootHashes(index, callback)
var number = feed.downloaded([start], [end])
var bool = feed.has(index)
var bool = feed.has(start, end)
feed.clear(start, [end], [callback])
feed.seek(byteOffset, callback)
feed.update([minLength], [callback])
var stream = feed.createReadStream([options])
var stream = feed.createWriteStream()
var stream = feed.replicate([options])
feed.close([callback])
feed.audit([callback])
feed.writable
feed.readable
feed.key
feed.discoveryKey
feed.length
feed.byteLength
feed.on('ready')
feed.on('error', err)
feed.on('download', index, data)
feed.on('upload', index, data)
feed.on('append')
feed.on('sync')
feed.on('close')

A couple other (total PITA) things to consider:

• We can't use the nodejs events system. It has to be browser events. (I hate this but what can you do.)
• We should consider whether there are security concerns to any part of the API being exposed
• All access is actually via an internal RPC, so some attributes (like feed.length) are not possible without doing a stop-the-thread synchronous message, or creating some data-sync mechanism. This is why DatArchive has the getInfo() method.

[RangerMauve]

@karissa @mafintosh Any opinions?

Maybe @noffle coming from Cabal?

[okdistribute]

@substack has thoughts on this

[ghost]

This could be good and I would use it in a version wrapped back into the js hypercore API. Hopefully that wouldn't incur too much extra overhead wrapping from hypercore to whatever browser thing and then back into hypercore. I know this was a huge pain in chrome with leveldb turning IndexedDB which nobody wanted back into leveldb which is a pretty good API but the indirection makes it unusably slow for a lot of use cases. I think here the mapping would be more direct so hopefully it wouldn't be so bad and could also avoid unnecessary data copies and marshaling.

Another way of accomplishing these goals is to unbundle direct block storage and network swarming. Then people can use whichever implementations they want. Then you also get control over the specifics of how to do swarming which is often required to hit the right trade-offs. You could also for example use a wasm version of the rust implementation of hypercore that talks to the block storage and network layers.

[hackergrrl]

What @substack said! I think as long as we can use the hypercore API (indirection or no) and have it be as close to native speeds as possible, I'd be really happy, since it'd make porting cabal and even mapeo[1] to Beaker much simpler.

[1] https://mapeo.world

[pfrazee]

Just adding in some discussion from the comm-comm call:

Beaker has a "library" for managing the user's dats. Hypercores are difficult to manage in the library because they have no standard metadata (no title). Hyperdrives have the dat.json.

The solution we discussed is, attaching Hypercores to origin-based-storage. It would be treated similarly to LocalStorage or IndexedDB data. The core would be swarmed as long as the origin has an open tab. We could then look into APIs for adding the cores to the user's library with some identifying metadata if the site wants to expose it to the user and persist the core's swarming when the tab is closed.

@substack I'm okay with block storage and swarming APIs if that's what folks prefer, but it might take longer to get those APIs done.