search

xuchuanyin / workbench

0 stars 0 forks source link

2018-08-06 external format #56

Open xuchuanyin opened 6 years ago

xuchuanyin commented 6 years ago

For CSV external format:

  1. create/load/query 1.1 block update/delete 1.2 block add/drop/rename columns 1.3 support compaction （compaction时需要不搬迁数据） 1.4 support delete segment （不删除数据） 1.5 support query on specified segment 1.6 support flat floder (通过ADD SEGMENT的方式，应该天然支持了) 1.7 support csv file with compression

  2. support datamaps 2.1 datamap is file-level （datamap索引的结果是一个文件，而不是一个blocklet。但要和当前datamap兼容） 2.2 support incremental load: datamap writer needs to be reconsiderd to minimize I/O during data loading 2.3 support rebuild datamap on existed data 2.4 support query with datamap 2.5 support compaction with datamap

  3. Support MV 3.1 support MV on external format 3.2 support incremental load 3.3 support build MV on existed data 3.4 support query with MV 3.5 support compaction with MV

  4. Support datamaps on MV 4.1 support create datamap on MV 4.2 support incremental load 4.3 support rebuild datamap 4.4 support query with datamap on MV 4.5 support compaction?

  5. Support parquet external format

xuchuanyin commented 6 years ago

Get Splits by Segment

  1. Explain Info Collector
  2. Query Statistic Collector
  3. get splits