search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.42k stars 1.27k forks source link

Removing ZKClient dependency from Pinot Client #7256

Closed mayankshriv closed 2 years ago

mayankshriv commented 3 years ago

Currently, Pinot client uses ZK client to watch on list of available brokers to send the request to. While this works well to keep the live broker list up-to-date, the dependency on ZK connectivity may not be desirable. One way to eliminate this would be to:

xiangfu0 commented 3 years ago

Another issue documented in #7166 is about the broker address we are publishing to client is for inter-Pinot cluster communications. It’s not guaranteed that client side is within same network. E.g. Pinot client outside k8s cluster doesn’t have access to broker exposed through controller API.

On Thu, Aug 5, 2021 at 09:11 Mayank Shrivastava @.***> wrote:

Currently, Pinot client uses ZK client to watch on list of available brokers to send the request to. While this works well to keep the live broker list up-to-date, the dependency on ZK connectivity may not be desirable. One way to eliminate this would be to:

  • Use controller apis to get the list of brokers to query for a table.
  • Use polling to keep the broker list up-to-date
  • Additionally, also update broker list in case of errors (related to broker connectivity).

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/apache/pinot/issues/7256, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJFPSH6MFQ42PO7VWD34XTT3KZ2LANCNFSM5BUFJEHA .

saurabhd336 commented 2 years ago

I'm working on the changes to support controller based broker selector in the client. Initial PR: https://github.com/apache/pinot/pull/8467/files @mayankshriv