Proposal for separating components into sub-modules

[dai-chen]

Currently all our code is in a single Gradle module at root. As the code base keeps growing, we should consider split major components into different sub-modules to have clear boundary and dependency.

Based on classic compiler and database engine architecture, here is one option:

1. Front end - query parser: perform syntax and semantic check and generate parse tree (or annotated AST).
2. Middle end - relational expression: generate relational expression (logical plan) and perform optimization based on relation algebra.
3. Back end - execution plan: generate and choose optimal execution plan (physical plan) and execute it in ES engine.

Apart from 3 major stages as above, we can also separate utility function into common module. And other candidate modules includes ES domain layer, documentation, benchmark etc.

Reference: https://en.wikipedia.org/wiki/Compiler and project structure in Calcite and CrateDb.

[dai-chen]

Recently we added doctest and correctness folder for doc generation and comparison test. These two along with integration test may be candidates for individual module too.

[penghuo]

Proposed sub-modules

• plugin: the module is the adapter to Elasticsearch plugin framework, e.g. it include RestPPLQueryHandler and TransportPPLQueryHandler. it depend on query sql/ppl module to execute the actual query.

• core: the core module corresponding to the middle-end and back-end of classic compiler and database engine architecture. e.g. LogicalPlan, PhysicalPlan, optimizer, execution framework.

• ppl: the implement of PPLEngine which include PPL specific front-end module, e.g. antrl, parser. It depend on core and es module to plan and execute the PPL query.

• sql: the implement of SQLEngine which include SQL specific front-end module, e.g. antrl, parser. It depend on core and es module to plan and execute the SQL query.

• common: the utility library could be used by all module. e.g.

  • protocol, the collection of formatter for different data format, e.g., jdbc, ion,
  • metric, the metric collection and publisher.
  • circuit breaker, the resource manager.

• es, the abstract layer of ES operation. which could include mapping, query??

• docs

• integration-test: integration test

• benchmark: TBD

• JDBC/ODBC/CLI: Should we merge JDBC/ODBC/CLI into one repository? One benefit is we can easily do the integration test in SQL project level.