search

superfly / litefs

FUSE-based file system for replicating SQLite databases across a cluster of machines
Apache License 2.0
3.92k stars 95 forks source link

zookeeper support #256

Open dastbe opened 1 year ago

dastbe commented 1 year ago

In the same vein as #116 it would be very beneficial to use zookeeper for leader election

alternatively, if there was a stable api to bring your own leader election implementation that might get around having to support of all the providers?

benbjohnson commented 1 year ago

@dastbe We're holding off on adding additional lease provider for the moment. One issue is that bundling in clients will may the litefs binary size explode. We've had requests for etcd, Kubernetes Lease API, and now Zookeeper and that's a lot of additional binary size.

Another issue is that it's a lot to support. I'll keep this issue open in case anyone else wants to chime in with thoughts but it's currently lower on our priority list.

darthShadow commented 1 year ago

Perhaps this could be accomplished via some kind of plugin system.

Go's plugin system has too many drawbacks so perhaps the hashicorp/go-plugin package may be a viable alternative instead?

benbjohnson commented 1 year ago

Yeah, Go's plugin system kinda sucks. :)

I haven't seen Hashicorp's before but it seems a little complicated since it runs over gRPC. I'll keep it in mind though in the future if we add support for more lease systems. Thanks!

balchua commented 1 year ago

Instead of relying on external services, perhaps using memberlist will suffice making litefs very low ops.

benbjohnson commented 1 year ago

@balchua LiteFS needs a concept of a leader since SQLite is single-writer. AFAICT, memberlist doesn't provide anything like leader election. It's mainly for eventually consistent gossip messaging.

balchua commented 1 year ago

@benbjohnson aaaa right!! 👍