"Tail" the feed into an index

[flyingzumwalt]

Whenever anything changes in the feed, write those changes to an index (probably leveldb). Mint unique identifiers/keys for each row.

About the Index

This index is built and maintained separately by each host that's running jawn. The index itself is not replicated across hosts. Instead, each host builds its index based on the blocks in the feeds it receives.

Which db to use?

The dat beta used leveldb/leveldown for its index because its fast, dead simple, and can run in the browser.

Other databases you could consider using: sqlite3, couchdb

Examples

The hyperkv module, which is used by the osm-p2p-db project, does something very similar. Where we want to tail a hypercore feed, hyperkv tails hyperlog feeds. hyperlog is similar to hypercore. mafintosh wrote it shortly before he wrote hypercore.

Design Tip: Separation of Concerns

Ideally, jawn should use an abstraction layer when building and interacting with its index so that you're not stuck with one option for databases. This abstraction layer would let us implement adapters to interact with different databases. Then adopters can choose whichever database suits their needs by choosing the adapter for that database.

To leave that possibility open for the future, focus on defining a module with a clear API for performing any functions that interact with the index. Only interact with the index through that module's API. That way we can swap out that module in the future with implementations that interact with different databases.

This is an example of the software development pattern called "separation of concerns". It's important. Read up on it!

[flyingzumwalt]

Rebooting issues (taking a different approach to roadmap)