search

mediachain / concat

Mediachain daemons
MIT License
42 stars 13 forks source link

Local store + Index #10

Closed parkan closed 7 years ago

parkan commented 7 years ago

Local store for statements and index

Probably SQLite but maybe something like rocksdb

Do we need to consider long-term/large dataset storage optimizations at all yet?

vyzo commented 7 years ago

SQLite should be sufficient for the immediate future. I don't think rocks.db is a good choice here (except perhaps for raw statement storage); we need an index and query capabilities.

In terms of scaling, SQLite can takes us far enough I think, and it should be easy to swap in a more heavyweight db solution if/when the need arises.

parkan commented 7 years ago

What protocol level metadata are we thinking about? Signer, timestamp, what else?

Down with SQLite, but should keep the scaling to billions of rows/batch processing/potentially columnar store case in the back of our minds.

vyzo commented 7 years ago

As a baseline I think we need indexing by namespace, id, publisher, timestamp, tags, and WKI references.

Well, the plan is to have it simple to use for lightweight usage. For aggregators/big nodes we will probably want to use a database server or RDS.