Why not RocksDB for streaming storage? - Githubissues

Sunt-ing / database-system-readings

:yum: A curated reading list about database systems

MIT License

465 stars 31 forks source link

Why not RocksDB for streaming storage? #102

Closed Sunt-ing closed 2 years ago

Sunt-ing commented 2 years ago

https://materialize.com/blog-rocksdb/

Sunt-ing commented 2 years ago

RocksDB Primer

Functions: point lookups, inserts, and range scans under strong isolation guarantees at high concurrent performance
Users: Main OLTP engines (CockroachDB, YugabyteDB, and TiKV), Stream processors (Flink, Kafka Streams)

Why Materialize does not use RocksDB?

Flink and Kafka Streams use a core (or several) dedicated to each operator
Timely dataflow cores are sharded differently, intentionally to minimize cross-core data movement
- Disadvantage: Operators on a worker are cooperatively scheduled, and have to be carefully designed to yield eagerly, or they will block other operators from executing, potentially stalling the entire dataflow graph
- Advantage: less cross-core data transfer