amient / affinity

Library and a Framework for building fast, scalable, fault-tolerant Data APIs based on Akka, Avro, ZooKeeper and Kafka
Apache License 2.0
26 stars 4 forks source link

Utilize RocksDB flush to stay asynchronous and ensure guarantees #248

Open michal-harish opened 6 years ago

michal-harish commented 6 years ago

https://github.com/facebook/rocksdb/wiki/Basic-Operations#asynchronous-writes

This could lead to a single point of control of at-least-once processing possible, e.g. potentially turning off write-ahead log completely on rocksdb (since kafka writes are primary they serve as a wal) and use a single rodksdb synchronous write to participate in the input-stream-processing work commit.