Enhance Data Ingestion Engine

[Jackie-Jiang]

Motivation:

• Offline segment generation and real-time in-memory consuming to on-disk committed segment conversion are very similar process, and they are both achieved by the same segment creation engine. We should try to unify the pipeline so that most of the features can be shared for both offline and real-time tables, including sorting data on a given column.
• Currently offline segment generation scans the records twice (record transformations also happen twice), which has low performance and is not friendly to streaming input.
• Currently real-time segment conversion reads the raw records from the segment, which can be significantly optimized to save both CPU and memory cost because all the useful information is already stored in the forward index.

End goal:

• Segment generation should purely based on the table config and schema, no extra parameters should be accepted (there is no way to pass extra parameters for real-time segment conversion).
• Offline data ingestion should only scans the records once and store all the information in memory (similar to real-time consuming segment).
• Offline data ingestion should support sorting segment on a given column (same as real-time segment conversion).
• Real-time segment conversion should use much less system resources and have less impact to queries.
• Design an interface (close to the idea of the stats collector) to store all the column data. Offline data ingestion pipeline creates them by scanning the data; real-time data ingestion pipeline creates them by wrapping around the in-memory forward-index, inverted-index, dictionary etc. The segment creation engine directly works on the new interface to create the segment so that offline and real-time pipeline can share the features of the segment creation engine.
• Data ingestion engine should support the following types of indexes:
  • Dictionary
  • Forward Index (both dictionary encoded & raw)
  • Inverted Index
  • Null Value Vector
  • Text Index
  • Star-Tree Index (v2)
  • (Bloom Filter is in memory only right now)

[mcvsubbu]

Related Issue #4036

[Jackie-Jiang]

Cleanup the unused old star-tree: #5086

[snleee]

To add here, I think that we should introduce the column-based interface for data indexing (maybe it's the same idea as Design an interface (close to the idea of the stats collector) to store all the column data).

If the input data is based on the columnar format, we will be able to generate dictionary/indices column by column. This will probably consume much less heap because we don't need to store all column data at the same time. Also, we can add the parallelization config to make the engine process multiple columns concurrently to speed up.