Enhance PQL -- Closer to SQL

winedepot / pinot

Apache Pinot (Incubating) - A realtime distributed OLAP datastore

https://pinot.apache.org

Apache License 2.0

1 stars 0 forks source link

Enhance PQL -- Closer to SQL #14

Open kishoreg opened 5 years ago

kishoreg commented 5 years ago

Our eventual goal should be able to support most of the sql features on a single table (excluding joins).

Currently, adding a new feature requires us to change pql grammar, broker request, and the execution layer. This means there is a high turn around time for each feature.

The right end state would be - "change only the execution layer to add new features - grammar and broker request should be complete and flexible". I am proposing we move to calcite and leverage the full sql support and parser that comes with Calcite.

Proposal

Create BrokerRequestV2.
Embed existing BrokerRequest inside BrokerRequestV2
AST Parser to convert PQL to BrokerRequestV2
Build a converter to convert BrokerRequestV2 to BrokerRequest and set it inside BrokerRequestV2.
Change the execution layer to use BrokerRequestV2
Introduce another end point that takes calcite sql and converts it into brokerrequestv2

pql 2 calcite

mayankshriv commented 5 years ago

This is definitely a good idea, and the proposal looks fine as well. Is the plan to make brokerRequestV2 SQL complete from the beginning? If not, could you elaborate on what does flexible mean? ie Would it eliminate forward/backward compatibility issues?

kishoreg commented 5 years ago

Yes, it should be able to handle all sql use cases on a single table. Apart from nested sql, joins and few other things, we should be able to model everything as Expression(function, literal, identifier). Hope is that we dont have to change brokerrequestV2 to support new functionality.

mayankshriv commented 5 years ago

Looking at the implementation of requestV2.thrift in #15, it seems that the only significant difference between the two is that the newer version allows for expressions in selection and filter.

Wondering if there's a way to re-use the existing request.thrift along with versioning information to interpret it accordingly? For example, request.thrift can also have all newly added fields in requestv2, along with a flag/version which indicates which fields to look at? And once the change is propagated on both broker/server side, the flag along with older fields can be dropped after marking the versions incompatible with prior ones.

kishoreg commented 5 years ago

I did that in the other PR. I thought this was much easier to review and less intrusive.

This also allows to start afresh instead of trying to patch existing things. We might find few things as we add more PRs.

Another plus point about this is we don’t have to touch anything in existing request.thrift

mayankshriv commented 5 years ago

@fx19880617 @kishoreg Is there a way to move this issue to Apache Pinot, as we are already working on this in the main repo? Also, could we add sections on migration/roll-out from PQL to SQL?