search

logv / sybil

columnar storage + NoSQL OLAP engine | https://logv.org
https://logv.org
Other
305 stars 25 forks source link

[distributed] investigate grpc service for sybil #37

Open okayzed opened 6 years ago

tmc commented 6 years ago

Refs #42

cpu commented 3 years ago

Hi folks, :wave:

I notice that a gRPC service implementation landed in-tree in 2018 with https://github.com/logv/sybil/commit/364be19d8ddae6ecf25f07b629d93d42bb55a29b but was later replaced with the Go API library in 2020 with https://github.com/logv/sybil/commit/08beb86d6d6a259f2107f1d0e7db2a700bb2cedb.

I wasn't able to find any commentary about the removal (were the proto dependencies too heavy/technology specific?) Could someone summarize why the gRPC service was dropped & how it relates to the future of this issue? Is this a WONTFIX for the project or a TODO?

Thanks!

okayzed commented 3 years ago

the gRPC service essentially was a new flag on the sybil binary that allows you to specify a remote host to connect to and execute a sybil query on. the sybild service would accept connections and receive the list of all options to run in the query. it then ran the query (by invoking the sybil binary) and returned the results over grpc to the client.

the main reason for removal was extra upkeep + not enough usage + it was tied to gRPC.

sybil is still capable of running queries remotely on hosts over SSH but we don't have a formal RPC API inside sybil itself. that can be built as a separate service that invokes sybil binaries and transfers results around as the sybil binary itself is capable of combining results from multiple runs on different machines.

cpu commented 3 years ago

@okayzed That makes sense, thanks! 👍

sybil is still capable of running queries remotely on hosts over SSH but we don't have a formal RPC API inside sybil itself. that can be built as a separate service that invokes sybil binaries and transfers results around as the sybil binary itself is capable of combining results from multiple runs on different machines.

I'm weighing the pros/cons of developing a fronting service like this but I think we've settled on a model that directly invokes one sybil binary for the time being.