Open serapath opened 4 years ago
first draft
(work in progress)hypercore creates feeds
(a.k.a logs)
many feeds
are not published via hyperswarm
using their discoveryKey
,
but instead are indirectly published via hyperswarm
using the discoveryKey
of a related feed
metafeed
with a first message to link contentfeed
and further messages form a hypertrie to link mounts
all signed by author
"dat cloud service"
it would now be up to the cloud to request relatedhypercore
data structure types https://github.com/datprotocol/DEPs/blob/master/proposals/0007-hypercore-header.md
Most dataStructureType
s are either using corestore
or multifeed
:
multifeed: https://github.com/kappa-db/multifeed/blob/master/mux.js#L71
corestore: https://github.com/andrewosh/corestore/blob/master/index.js
feed
"dat address"
feed
"dat address"
feed
smetafeed
associated with a corresponding "dat address"
message 0
references a related contentfeed
feeds
via:trie.list(‘mounts’, { hidden: true }, cb)
contentfeed
is published through the swarm of the related metafeed
and it's discoveryKey
hyperdrive
, ...and morefeed
"dat address"
log
and stores them in leveldb
channel!timestamp
, value: feed@seq
kappa-db
, cabal-core
, ...and more...
a generic seeding service
receives a "dat address"
and doesn't know which kind of data structure is behind that address and needs to know all related hypercores
and how to retrieve them and a standard way of making them available to anyone who wants to use the data and data structure behind that "dat address"
a generic seeding service
should not need any data structure specific code to know how to seed the data. This means the seeding service should not need to know all currently existing data structures build on top of hypercore
nor need to be updated for every such data structure created in the future.
this seeding service could also see what the first message they get from a peer is, and from there, figure out whether it's {multifeed,hyperdrive,etc} and do sync from there and just speak the major protocols, not as nice as a unified protocol though :)
- [connect to an address via hyperswarm]
- receive first [protocol] message from a peer to derive the protocol type
- e.g. multifeed, hyperdrive, ...
- support the "major protocols"
RELATED FEEDS
)after reading through `mux.js`, It made me feel that `"manifest"`
might as well be a `manifestfeed` and the `manifest handshake`
could share the `manifest feedkey` so any peer online
could share it and eventually related feeds while the author is offline.
dataStructureType
to identify if a given structure is one of a few supported types or a generic typeAdditionalData
an entry for manifest
which points to a feedkey
of the manifestfeed, which is probably a hypertrie to somehow store all the related feedsor alternatively
main feedkey
manifestkey
manifestkeys
existmain feedkey
part
(e.g. contenfeed)link
(e.g. mounts)origin
to cite reason/proof why a feed is related
To my understanding the reason as to why the hyperdrive structure is how it is relates to performance and memory use. Particularly important is the number of round-trips necessary to until a data structure can be explored. If we have to read the manifest data before we can connect to the feeds it is one entire initial roundtrip added, which can easily mean 200ms added delay. Having the url be the identifier on how the feeds are handled might make it a better best-practice. Maybe we could have "patterns" of feed identification. i.e. the hypertrie pattern of hyperdrive 10 or the cabal pattern, which is part of the url specification.
@martinheidegger I answered here: https://github.com/datproject/comm-comm/issues/134#issuecomment-604806258
I think it'd be cool if the related feeds mechanism was used just for cases when a peer doesn't know your data structure and wants to get related feeds for pinning and stuff, while keeping existing mechanisms that data structures use for performance.
Maybe related feeds is something you opt into loading after you get the initial data or through an extension that only certain peers will bother invoking?
@RangerMauve i hope this is fullfilled by the latest proposal update which i perceive to be: https://github.com/datproject/comm-comm/issues/134#issuecomment-604808606
additional considerations:
certificate feed
so you can update it (e.g. key rotation to point to new certified replacement feeds)certificate feed
e.g. in chunk0
so key compromise can't undo that.On first sight they might both be able to solve the same problem, but they have some nuances especially if that general kind of pointers/links are used for different use cases.
chunk0
that contains among other things information about:
certificate feed
certificate feed
That way, nobody can copy such a message to any other feed to associate ownership over e.g. "child porn" with somebody, because the publickey wouldn't match
Whether people outsource maintanance of certificate feed
to third parties or themselves, in both cases, a certificate feed
could be HOT (especially if it is operated by some kind of trusted third party CA) so that feed having it's own parent certificate feed
would be cool
furthermore, it would be cool to be able to specify a ring of certificate feeds (controlled by the same entity or different), where the majority of keys can vote out or vote in new keys to do key rotation to further improve security in case of lost or stolen private keys for such feeds.
All these issues are technically linking together feeds.
...it's just generally something that is much tougher to bolt onto things later on I believe.
@todo
DDEP-0001:hypercore-header-for-manifestfeed.md
motivation There are many ways why feeds need to be linked parent to dependant to dependencies, dependencies to dependant, domain to content, feed to author, related feeds amongst each other and I think it would be bad to have everyone (app/protocol/datastructure) make those things up instead of following a general standard
ongoing discussions
messy incomplete list of involve community/ecosystem in no particular order (feel free to add/correct the list below by mentioning that in a comment)