New Database API

[richburdon]

Next steps: https://github.com/dxos/echo/pull/67

Stage 1

• [x] PartyProcessor class that consumes iterator and processes AdmitMessages (which influence the iterator).
• [x] TestItemEnvelope with itemId used by demuxer.
• [x] PartyProcessor class implements EventListener and fires update event when message consumed/routed to itemDemuxer (for testing). Adapt tests to count these events (via latch) replacing waitForXXX.

API: createPartyStream is a specialized method (not part of FeedStore) that returns an upstream stream that can be piped to the itemDemuxer. It is provided with a functor that present candidate messages, the current set of feed keys, the current anchor (vector clock timestamp) and a callback that advances to the next message -- and optionally updates the set of feedKeys and/or the anchor point. The anchor point allows the stream to skip ahead to a specific position.

const feedStore = new FeedStore();
const feedKeys = [feed1];
const anchor = null;

const partyStream = createPartyStream(feedStore, (messages, feedKeys, anchor, callback) => {
  let [message, feedKey] = containsAdmit(messages);
  if (message) {
    callback(message, [feedKey, ...feedKeys]);
  } else {
    message = chance.pickone(messages);
    callback(message, feedKeys, anchor);
  }
}, feedKeys, anchor);

partyStream.pipe(createItemDemuxer(itemManager));

Stage 2

• [x] Extend proto to include ItemEnvelope that contains vector timestamp.
• [x] Wrapper for David's clock mechanism (generate and compare clocks).
• [x] MessageWriter that writes timestamp to hypercore.
• [x] Processes Item messages based on clock streamed into ItemDemuxers.

Iterator Spec:

• Determine which feeds to consume.
• Determine which message to consume from candidate set from current feeds.
• Halt if nothing selected (indicates a gap); restart on event (e.g., feed added by replicator).

Stage 3

• [ ] Adapt existing multi-node test-suite (simulate peers with protocol replication).
• [ ] Compare throughput tests with existing mechanism.

Stage 4

• [ ] Retrofit/port existing HALO/ECHO mechanisms (e.g., @telackey PartyProcessor).

Hygiene

• [ ] Lint issues (conflicts with babel? Had to commit -n); also incorrect line numbers using jest.
• [ ] TS casts (see TODOs).
• [ ] Typedefs/shims (e.g., Hypercore).
• [ ] Use protobuf TestItemMutation type instead of Message

[tinchoz49]

@richburdon

FeedStore bug (cannot re-open feedstore)

Is there a failing test to check this? because this works:

const { FeedStore } = require('@dxos/feed-store');

(async () => {
  const fs = new FeedStore('db')

  await fs.open()
  console.log(fs.opened) // true

  await fs.close()
  console.log(fs.opened) // false

  await fs.open()
  console.log(fs.opened) // true
})()

and cannot create a second feedstore object with same directory.

It's not possible to create a second feedstore in the same directory like is not possible to create a second hypercore in the same file.

[richburdon]

@tinchoz49 yes: https://github.com/dxos/echo/blob/master/packages/echo-experimental/src/database.test.ts#L124

[tinchoz49]

Fixed bug in database.test.ts: https://github.com/dxos/echo/pull/70

Issues:

• Outdated FeedStore version.
• When you close a FeedStore you close the feeds related to. If you open the FeedStore again and you want your old feeds back you need to open it again explicitly using feedStore.openFeeds

[dboreham]

Message stream ordered by timestamp:

Given a feed set Fn, comprising messages replicated from other nodes, a reader having access to only the "head" message from each feed corresponding to the reader's feed sequence vector with the job of producing a totally ordered output message stream:

Conjecture - for any previous message M1 read from feed Fx having timestamp T1, the next message in timestamp total order will be identifiable from (only) the set of feed head messages, or it will not exist (currently on this node) and reading must pause until it is available.

Proof - intuitively this seems correct because every message Mx read from a feed must necessarily have a higher timestamp than message Mx-1 read from the same feed. Therefore if a head message from any feed isn't the next message in order with respect to some previous message, then subsequent messages on that same feed can't possibly be the next message (because they came after it). Therefore the message we are looking for must either be one of the head messages, or it hasn't been received yet.

[richburdon]

Merged https://github.com/dxos/echo/pull/81

First cut at DB API.

Related

• https://github.com/dxos/halo/issues/40 (HALO design)
• https://github.com/dxos/halo/issues/37 (HALO API)

Process

• [ ] Port https://github.com/dxos/echo/issues/68 to echo-db.
• [ ] Item model metadata processing (separate from custom model).
• [ ] Adapt and test existing models.
• [ ] Basic implementation with FeedStore without HALO processing.
• [ ] Design MetaStore (e.g., which feeds/parties are opened first? Other settings).
• [ ] Inject HALO state machine (extricate FeedStore from HALO).

[richburdon]

Working prototype: https://github.com/dxos/echo/pull/87