search

nats-io / nats-streaming-server

NATS Streaming System Server
https://nats.io
Apache License 2.0
2.5k stars 285 forks source link

EdgeFS as decentralized, secure and immutable NoSQL store #961

Open dyusupov opened 4 years ago

dyusupov commented 4 years ago

Dear NATS developers!

It would be great to provide support for EdgeFS S3X NoSQL as NATS store.

EdgeFS is high-performance, low-latency, small memory footprint, decentralized data fabric system released under Apache License v2.0 developed in C/Go.

EdgeFS is built around decentralized immutable metadata consistency to sustain network partitioning up to many days and yet provide consistent global data fabric namespace via unique versions reconciliation technique.

EdgeFS advanced globally immutable versioning with file-level granularity unlimited snapshots, global (LAN/WAN) data deduplication and geo-transparent data access enables a variety of decentralized Edge/Fog/Cloud Computing use cases.

EdgeFS GitHub link: https://github.com/Nexenta/edgefs

I'm willing to code up a prototype and send a PR, if anyone is interested? Appreciate if some starting pointers and examples (like null driver etc) can be provided...

kozlovic commented 4 years ago

@dyusupov Thank you for your interest on NATS Streaming. The first issue I see is that so far we build our servers (NATS Server and NATS Streaming Server) with CGO disabled. This is because historically we build docker images from SCRATCH.

The other issue is that unless we make use of plugin, if you add a new store implementation and it is part of the mainline, it will have to be imported in the project as a dependency. Except for the CGO thing, would have to make sure that your deps are ok license wise.

Finally, at this point since NATS Streaming uses RAFT for consensus, I guess it would not leverage the fact that your store is already built to be decentralized, no?

So I am not sure if the PR would be merged (CGO is definitively and issue at this point), but if you want to have a sense of how easy/difficult it would be, as documented we have the Store interface (https://docs.nats.io/nats-streaming-concepts/store-interface) that you would need to implement and you can look at some of the stores implementation that make use of "common store" implementations. And again, since we don't use plugin, the new store type would have to be instantiated in RunServerWithOpts(). Let me know if that helps.