search

logv / sybil

columnar storage + NoSQL OLAP engine | https://logv.org
https://logv.org
Other
305 stars 26 forks source link

Status of Distributed Sybil? #118

Closed wang502 closed 4 years ago

wang502 commented 4 years ago

Wondering what is the status and planing of distributed version of Sybil.

Is there any active development?

okayzed commented 4 years ago

There's currently not active development here, but it does work. I use it across 3 VPS in the cloud for my own personal usage.

There's msybil (a python script which snorkel calls into) which is able to call out to multiple sybil instances over SSH (remember to use persistent connections) and then combine their results together. msybil also supports round robin ingestion.

It's pretty simple to get msybil working, but my feeling is that putting up a distributed sybil (across 10s of machines) will require some debugging and a few months of effort in terms of hunting down bugs / behavior.

The current focus for sybil has been on supporting larger datasets (100 - 500mm rows + hundreds of columns).

If you want to sync offline, feel free to email me or ping me on workplace