Refactor pinot-common and pinot-core

[kishoreg]

After extracting the plugins, the next logical step is to clean up pinot-core and pinot-common. Similar to pinot-spi, we should extract all interfaces from common and core into something like pinot-query-spi.

Creating this issue to list all the interfaces we want to extract out.

Here is the list (will keep this up to date based on the comments)

• Plan
• Operator
• Datasource
• Block
• TransformFunction
• AggregationFunction

What else?

[xiangfu0]

For data part, we also need to make index writer logic pluggable.

[kishoreg]

Grepped for all interfaces in pinot-core. We can start by selecting the interfaces that should be part of query-spi.

• [ ] PeriodicTask extends Runnable
• [ ] IdMap
• [ ] BufferCleaner
• [x] PlanNode
• [x] PlanMaker
• [x] Plan
• [ ] SegmentPreIndexStatsCollector extends SegmentPreIndexStatsContainer
• [x] InvertedIndexCreator extends Closeable
• [ ] SegmentCreationDataSource
• [ ] SingleValueForwardIndexCreator extends ForwardIndexCreator
• [ ] SegmentPreIndexStatsContainer
• [x] SegmentCreator extends Closeable
• [ ] SingleValueRawIndexCreator extends ForwardIndexCreator
• [x] SegmentIndexCreationDriver
• [ ] MultiValueForwardIndexCreator extends ForwardIndexCreator
• [ ] ColumnStatistics
• [ ] ForwardIndexCreator extends Closeable
• [ ] VirtualColumnProvider
• [ ] SegmentFormatConverter
• [ ] ColumnIndexContainer
• [ ] DefaultColumnHandler
• [ ] InvertedIndexReader extends Closeable
• [ ] Dictionary extends Closeable
• [ ] NullValueVectorReader
• [ ] SegmentMetadata
• [ ] SegmentNameGenerator
• [ ] MutableSegment extends IndexSegment
• [ ] ImmutableSegment extends IndexSegment
• [ ] IndexSegment
• [ ] SortedIndexReader extends SingleColumnSingleValueReader, InvertedIndexReader
• [ ] ReaderContext
• [ ] DataFileReader extends Closeable
• [ ] SingleColumnSingleValueReader extends DataFileReader
• [ ] SingleValueMultiColumnReader extends DataFileReader
• [ ] SingleColumnMultiValueReader extends DataFileReader
• [ ] ValueReader
• [ ] DataFileWriter extends Closeable
• [ ] SingleValueMultiColumnWriter extends DataFileWriter
• [ ] SingleColumnSingleValueWriter extends DataFileWriter
• [ ] SingleColumnMultiValueWriter extends DataFileWriter
• [ ] ChunkDecompressor
• [ ] ChunkCompressor
• [ ] PinotDataBufferMemoryManager extends Closeable
• [ ] BloomFilter
• [ ] StarTreeNode
• [ ] StarTree
• [ ] StarTreeV2
• [ ] SingleTreeBuilder extends Closeable
• [ ] RecordAggregator
• [ ] RecordTransformer
• [ ] RecordPartitioner
• [ ] RecordPurgerFactory
• [ ] RecordPurger
• [ ] RecordModifierFactory
• [ ] RecordModifier
• [ ] ValueAggregator
• [ ] ObjectSerDe
• [ ] BlockMetadata
• [ ] Operator
• [ ] BlockDocIdIterator
• [ ] BlockDocIdValueIterator
• [ ] BlockDocIdSet
• [ ] BlockValIterator
• [ ] DataSourceMetadata
• [ ] BlockDocIdValueSet
• [ ] BlockValSet
• [ ] Block
• [ ] AggregationResultHolder
• [ ] AggregationExecutor
• [ ] AggregationFunction<IntermediateResult, FinalResult extends Comparable>
• [ ] GroupByExecutor
• [ ] ValueToIdMap
• [ ] GroupByResultHolder
• [ ] GroupKeyGenerator
• [ ] SegmentPruner
• [ ] DataTableReducer
• [ ] SchedulerGroupFactory
• [ ] SchedulerPriorityQueue
• [ ] SchedulerGroup extends SchedulerGroupAccountant
• [ ] SchedulerGroupAccountant extends Comparable
• [ ] SchedulerGroupMapper
• [ ] QueryExecutor
• [ ] ValueAggregator<R, A>
• [ ] TableDataManager
• [ ] InstanceDataManagerConfig
• [ ] SegmentCommitter
• [ ] InstanceDataManager
• [ ] Table
• [ ] SegmentSorter
• [ ] PartitionFunction
• [ ] RecordTransformer
• [ ] MetadataExtractor
• [ ] FilterBlockDocIdSet extends BlockDocIdSet
• [ ] ScanBasedDocIdSet extends FilterBlockDocIdSet
• [ ] ScanBasedDocIdIterator extends BlockDocIdIterator
• [ ] IndexBasedDocIdIterator extends BlockDocIdIterator
• [ ] PredicateEvaluator
• [ ] DataTransformer<I, O>
• [ ] TimeUnitTransformer extends DataTransformer<long[], long[]>
• [ ] TransformFunction

[yupeng9]

+1

I'm interested in adding a pluggable geospatial module, and I think it demands several pluggability points.

• Query operator and expression function evaluation, this may require TransformFunction Operator Datasource Block and their dependencies (e.g. DataSourceMetadata, DataFileReader ,BlockDocIdSetand many others)
• Introduce a new type of index, this may require an IndexBuilder interface that SegmentIndexCreationDriverImpl can call to build the index. In addition, some configuration interface to specifying the index-specific configurations
• Some sub interface in IndexSegment for loading the index structures (similar to how getStarTrees is added to IndexSegment)
• Some pluggable rules in logical plan and physical plan building for the pluggable operators.