set of observables = meta data, worth storing in a separate feed (log)

[tschudin]

Why not have a meta feed as a companion to the main feed (this meta feed could be announced in the very first entry of the main log) for hosting a key/value store inside it, like Hypertrie for Hypercore?

This would allow for efficient lookup of the latest "release=?" or "profile=?" binding etc, and is easy to locate because only one redirection away: fetch the first log entry in the main feed once, then work with the k/v store in the meta feed.

If you are afraid that the meta feed grows too large and you want to drop the history, chain several of them by adding an entry "this meta feed now replaced by feed @_X" before closing the feed, or have a really slow growing meta meta feed containing only a series of "meta=@_X" bindings, the latest being the valid one, or have every 100th entry in the main feed be a "meta=@_X" entry. In the latter two cases, old meta feeds can be discarded.

[arj03]

Thanks for the feedback @tschudin.

Why not have a meta feed as a companion to the main feed

This would assume that we could rely on feeds adhering to this standard always, it can never be later. So it wouldn't work for existing SSB classic feeds. One of the goals of observables is that it works on existing feeds and that it even works if the author of the feed is gone, as others can also make statements about a feed. One could have two feeds, but then the problem becomes linking them (aka same-as).

k/v store

As for bundling things up into 1 data structure I guess is an implementation choice. One could just have 1 main observable and store a k/v store inside this thing. Or one observable per feed. Maybe that part was overspecified.

meta feed grows too large

I see these observables as snapshots and their size are quite small and I don't think their history is interesting. If somebodyis cheating you store those in some kind of proof system, but otherwise only the latest version is needed. It keeps the protocol simpler that you can just rely on the latest version and have that propagate fast through the network using something like EBT.

[tschudin]

thanks @arj03 for the additional color. In defense of the history (and of using a log for propagating observable state):

• is there an implicit assumptions in your argument that having history is slowing the propagation of important news? This does not have to be true, and if observables are seen as more important than other traffic, we could have feed priorities.
• why is history a burden? The consuming ends are free to keep nothing of the log except the latest entry i.e. throwing away the history as it arrives.
• without history you will have out-of-order delivery - isn't this a problem? And if you introduce numbering, we are almost back to logs..
• the fact that "others" are observing my feed would need to be documented somewhere - for all others to know about. This would probably happen in that other's meta feed (if you want to avoid central GitHub-like maintenance of HEAD state). So I think that having an official meta-feed for your main feed is inevitable: without such a base-level observable, we would be back to having to deep-scan all logs in our reach to learn about the observers.

Regarding backwards compatibility where to announce a meta feed I agree that using the first entry in a log is not working for classic feeds. But the "every 100th entry" would be a pragmatic fallback solution if you don't find the meta pointer in a log's first entry?

[arj03]

True, the difference between history and no history basically boils down to: is this a feed or a single value. It is designed to have version numbers, so it doesn't have the problem with out-of-order. I guess conceptually it would be simpler with a feed where one by default is assumed to only store the latest version. I was explicitly trying to make this different as I don't think it should be stored in the main log.

Now the where do you find the entry points to observables is perhaps a more interesting problem. At first I was looking at it from the angle that first one finds the observables, then the data that the observables describe. Instead as you can see in the partial replication proposal, the state of the observables are included in the results as proofs. Of course one could use the partial replication to find all the observables as they have a specific type, this would most relevant for observables that are not proofs of something.

[arj03]

@tschudin I'm closing this issue as I think you got me convinced :-) I wrote a first draft of a https://github.com/arj03/ssb-meta-feed spec.