search

sirixdb / sirix

SirixDB is an an embeddable, bitemporal, append-only database system and event store, storing immutable lightweight snapshots. It keeps the full history of each resource. Every commit stores a space-efficient snapshot through structural sharing. It is log-structured and never overwrites data. SirixDB uses a novel page-level versioning approach.
https://sirix.io
BSD 3-Clause "New" or "Revised" License
1.11k stars 250 forks source link

Issue #537 (alternative kafka approach) #728

Closed FayKounara closed 2 weeks ago

FayKounara commented 3 months ago

As discussed under the issue https://github.com/sirixdb/sirix/issues/537 me and @ElenaSkep would like to propose an enhancement and use apache kafka as an alternative for backend storage.

What it does: Kafka listens to the changes made in a db/resource and stores them in a topic which takes the name of the db.Doing that you can see from the topic of your db the history of all the changes.

Prerequisite:Please download the zip file of apache kafka and run it in the bash with these 2 commands $ bin/zookeeper-server-start.sh config/zookeeper.properties and this $ bin/kafka-server-start.sh config/server.properties.

After that whenever you change something in your db you can see the changes by doing this $ bin/kafka-console-consumer.sh --topic your-db-name --from-beginning --bootstrap-server localhost:9092

JohannesLichtenberger commented 3 months ago

This could be part of an external connector or change data connector (Maybe a separate repository using sirix-core as a dependency). That said as in the case with the websocket client (which sould rather be the server where a TypeScript based client might subscribe to a specific database/resource via a dedicated route in sirix-rest-api) we have to invert the dependency, as it should be no requirement to run Kafka.

JohannesLichtenberger commented 3 months ago

Hope that makes sense and that you understand the issue a bit more