search

replikativ / datahike

A fast, immutable, distributed & compositional Datalog engine for everyone.
https://datahike.io
Eclipse Public License 1.0
1.63k stars 97 forks source link

add FoundationDB integration #21

Closed kordano closed 3 years ago

corporatepiyush commented 5 years ago

it would be great if it is achieved through a pluggable instance of Protocol which respects the generic Key-Value database API semantics.

kordano commented 5 years ago

That's the plan with #32

FalseProtagonist commented 5 years ago

Am I right to think that this would be the first durable DB backend other than a flat file? If it's equally hard I imagine that dynamodb would provide more reach?

whilo commented 5 years ago

@corporatepiyush @FalseProtagonist Just to give a bit of context. The KV-semantics can usually be mapped to konserve, which is already support for our default index datastructure, the hitchhiker-tree. Writing a DynamoDB backend should be enough to store datahike there (similar to the konserve CouchDB, Redis or Riak backends), but we still need to provide a networked transactor instance to become fully distributed.

FalseProtagonist commented 5 years ago

@whilo thanks for the reply, so there are already backends for all those databases? Is there a doc somewhere on how to set them up? + where's the code location in case one wanted to write one for DynamoDB

kordano commented 5 years ago

@FalseProtagonist Currently I'm working on the protocols for that. Currently only an in-memory and file-based backend is implemented. I will show you how you could add different backends, once I finished that.

whilo commented 5 years ago

The projects are referenced here: https://github.com/replikativ/konserve. Looking at memory and the filestore.clj backend should give you a reasonable idea of how to integrate it. Exposing that in datahike will be comparatively simple. @kordano is working on exposing different index semantics directly, which makes datahike portable beyond the hitchhiker-tree, but I think you want to use DynamoDB just as a blob store, right?

kordano commented 3 years ago

Moving that issue to our roadmap.