Closed parkan closed 7 years ago
@vyzo let's discuss this tomorrow, I think this is a relatively important connecting piece, but willing to hear counterarguments
@parkan batch reading is supported with a query, no need to add a separate interface. We can easily add a batch write interface.
Also, I don't think we should store bare statements outside the db, there is no need for that. We will need a KV store for the actual metadata objects though.
Nonetheless, the database backing the concat node can be shared when using an RDBMS server.
OK, we should discuss the relationship between statements and actual metadata then, it sounds like the latter is what needs to be shared with indexer.
Ok, given that we've decided to move forward with RocksDB, we can't have multiple writers and so this approach is not possible.
Perhaps we can have a datastore implementation that uses the IPFS datastore through the API, but that's unworkable for large datasets.
The symmetric approach is that concat can act as a datastore through the API (with hash-based retrieval) for IPFS. We can't implement the crazy go-datastore interface however, but does IPFS really use all that crazy stuff with hierarchical keys and queries?
Also of note, is that we can temporarily shutdown the concat node and directly load RocksDB for bulk ingestion. It has a special config optimized for bulk writes, that disables compaction and stores everything as level 0. When you are done, you manually run compaction, and reopen the db for general purpose access.
Punting on this, and may in fact not do it directly at all; thinking now is potentially focused on https://github.com/facebook/rocksdb/wiki/Creating-and-Ingesting-SST-files
I've been thinking about how to best create interop between the components in our stack:
Can we use a shared memory-mapped db like LMDB? Imagine a shared "object store" (in IPFS parlance) which is a KV that fulfills go-datastore interface and contains the bare statements, while IPFS, concat, indexer each have their own meta-metadata (provider records, statement envelopes, vectors) referring to them. This is potentially not entirely compatible with reconciliation/bag of statements/wire format, need to think more about that, but overall I think this is a worthwhile question.