search

oap-project / gazelle_plugin

Native SQL Engine plugin for Spark SQL with vectorized SIMD optimizations.
Apache License 2.0
256 stars 77 forks source link

The compile procedure must be redesign for Arrow, Arrow Data Source, and Native SQL Engine #176

Open weiting-chen opened 3 years ago

weiting-chen commented 3 years ago

Current install procedure is

  1. compile & install Arrow Data Source
  2. compile & install Arrow
  3. compile & package Native SQL Engine

There is a potential issue since Arrow Data Source is also depending on Arrow. The Current design is to leverage the existing installed Arrow path. A better way is to redesign the installation procedure like below

  1. compile & install Arrow
  2. compile & install Arrow Data Source
  3. compile & package Native SQL Engine
weiting-chen commented 3 years ago

The current procedure is to control Arrow installation via CMakeList.txt, but no cpp source code in Arrow Data Source. Since Arrow Data Source is also using Arrow library and a must dependency for Native SQL Engine, An initial idea is

  1. Remove the code from Native SQL Engine.
  2. Move the Arrow installation procedure to Arrow Data Source as a dependency.