Hypercore Header DEP

[pfrazee]

Standardizes a HypercoreHeader message which can be used as the first entry in hypercores to declare the datastructure along with custom fields.

[pfrazee]

I still kind of wish we could deal with nested schemas

I'm not against revisiting this. It should pretty much always be layered (one structure on top of another) so all we need is a standard separator. The downside is that it adds some complexity (multiple types emitted instead of just 1).

I don't remember that much about the wire representation of protobuf, but we will implicitly still be trying to parse a lot of random bytestrings to see if they "match" this schema.

Hmm, yeah that's true.

Should we mention that folks should specifically avoid having a protobuf message that has a string field as the first entry, because that could accidentally get interpreted as a "type" string?

Tricky. If it ends up possible that non-header messages have strings at tag 1, then we have to interpret an unknown type string in the same way we interpret no header at all. We could add some magic numbers to the schema, except that wouldn't be backwards compatible. Might not be a huge deal.

Do we need to set a sane max version length (1024 bytes?) to prevent gnarly/weird parsing?

Is there a way to put a max string length on protobuf fields? @mafintosh

[yoshuawuyts]

Are the proposed version names for hyperdrive / hyperdb also part of the DEP, or do they only serve as an example? If they only serve as an example, it might be good to make it extra clear that the names aren't being specified in this DEP.

edit: typo

[pfrazee]

Are the proposed version names for hyperdrive / hyperdb also part of the DEP, or do they only serve as an example? If they only serve as an example, it might be good to make it extra clear that the names aren't being specified in this DEP.

Agree. They're just examples

[bnewbold]

To summarize the current state of this in my eyes:

• needs a couple tweaks to naming and examples in the DEP text, but I would approve to merge trusting that Paul will make the changes first
• I'd like to move forward, but with the expectation that we should code this up ("working code") and make sure that it allows beaker/dat to disambiguate and handle both old and new hyperdrive before 100% commiting to this scheme (aka, move to Draft ASAP, also prove this out in code to get to Active status ASAP, as getting this correct for the long-term is crucial for the ecosystem)

[pfrazee]

Just fixed that "hypercore-v1" thing and added a note that the strings will be in the data structures' DEPs.

I renamed type to dataStructureType. I'm open to discussion on this. I have some code that's trying to handle dats generally in preparation for more types. In it, I referred to each general item as a "dat" and the instance of hypercore/hyperdrive/hyperdb as that dat's "data structure". So:

var dat = {
  key: '...',
  dataStructureType: 'hyperdrive',
  dataStructure: hyperdrive(...)
}

The word dataStructure is a little clunky but it does seem to work alright.