apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.31k stars 1.24k forks source link

Decouple pinot-tools dependency on pinot-server, pinot-controller and pinot-broker #5692

Open tmwangi opened 4 years ago

tmwangi commented 4 years ago

It would be a great idea to break this dependency by introducing some api modules or some other way. The result would be a smaller pinot-tools with none of the vulnerabilities or license issues that come with including the server, controller or broker. See https://github.com/apache/incubator-pinot/blob/master/pinot-tools/pom.xml for the dependencies included.

For example in order to get the Pinot Table Creation url, I have to access the class https://github.com/apache/incubator-pinot/blob/47ff09b4c5d726355a4516d2e1eb72c9705b81b9/pinot-controller/src/main/java/org/apache/pinot/controller/helix/ControllerRequestURLBuilder.java which defines the url and is in the pinot-controller module. This could be defined in a much smaller pinot-controller-client or pinot-controller-api to avoid having to include all of pinot controller.

Jackie-Jiang commented 4 years ago

Thanks for the suggestion. It is not possible to remove pinot-controller/broker/server dependency from pinot-tools because the tools package need to support starting controller/broker/server instance. But it is a great idea to add a separate client api module for admin purpose and have minimal dependencies. We are planning to re-organize pinot-common and pinot-core and split them into api/spi modules and implementation modules. Will add another client api module when we design the new package hierarchy.

tmwangi commented 4 years ago

Thanks @Jackie-Jiang . Our use case has to do with creating Pinot Tables and I think that part of Pinot Tools can be carved out into a module of its own as the dependencies on controller, server and broker are not needed for this.