search

startreedata / pinot-client-go

Apache Pinot Golang Client managed by StarTree
Apache License 2.0
28 stars 10 forks source link

gRPC communication with Pinot #3

Open jpohanka opened 3 years ago

jpohanka commented 3 years ago

Issue type: feature request

Pinot version: 0.8.0

Description:

We are using the Golang module for connecting with Pinot in our internal applications. So far, the JSON-based communication does not pose a bottleneck for smaller result sets (order of magnitude: 10^3).

However, for moderate result sets (order of magnitude: 10^4-10^5), the JSON-based communication starts to be inefficient.

To mitigate this, it would be convenient to implement a gRPC communication in this module, since Pinot has gRPC communication implemented internally and few other systems (i.e. PrestoDB) use this for fast fetching of the result sets.

xiangfu0 commented 3 years ago

Thanks for bringing this up! Right now pinot broker has only one endpoint, which is http-json based.

GRPC endpoint on Pinot server is for internal usage, not exposed as a public query endpoint.

  1. From query issuer, client side needs to understand which Pinot server to query and what are the segments to query on each server.
  2. Client side needs to reduce all the partial results from each Pinot server
  3. Need to handle hybrid table query split as well.

Presto can interpret the query and use grpc to fetch big trunk of data then perform more computations.

Pinot community is also considering adding a grpc endpoint for Pinot broker. Once that is added, we will support grpc client as well.

jpohanka commented 3 years ago

@xiangfu0 Thank you very much for the explanation!

khalidDaoud commented 1 day ago

Hi @xiangfu0, any progress on this front!