confluentinc / confluent-kafka-dotnet

Confluent's Apache Kafka .NET client
https://github.com/confluentinc/confluent-kafka-dotnet/wiki
Apache License 2.0
66 stars 861 forks source link

Accessing local rocksdb #713

Closed pablocastilla closed 5 years ago

pablocastilla commented 5 years ago

Hi!

I am creating a PoC about implementing microservices with kafka. One thing I understand is out of the scope is to migrate streams to dotnet, but on the other hand I think maybe it is feasible to implement a dotnet access to the rockdb local store (just for read and set in a transactional way as streams do). That would be useful for implementing stateful services.

Thanks!

mhowlett commented 5 years ago

I think even better might be FASTER from Microsoft https://www.microsoft.com/en-us/research/blog/microsoft-unveils-faster-key-value-store-large-state-management/

rocksdb integration is out of scope for this repo. There are other bindings you could use though: https://github.com/warrenfalk/rocksdb-sharp

pablocastilla commented 5 years ago

Thanks for answering.

Just to clarify, my point was on having a persistence that works with the Kafka transaction, which I understand is what Kafka streaming does, that way we can work in a exactly once manner, not having to do idempotence tricks in the service storage.