metrico / quackpipe

QuackPipe is an OLAP API built on top of DuckDB with a few extra ClickHouse compatibility bits
https://quackpipe.fly.dev
MIT License
149 stars 6 forks source link

Parse and process MergeTree CREATE TABLE expression #31

Open akvlad opened 1 month ago

akvlad commented 1 month ago

What

Desired functionality:

  1. A user makes a create table request:

    POST /qp/query
    ...
    create_table: experiment
    fields:
    a: UInt64
    b: String
    c: Float64
    engine: Merge
    order_by: 
    - a
    timestamp:
    field: a
    precision: ms
    partition_by:
    - b

    Or the json request with the same data.

  2. The processing component should create the following entry int the tables table:

  name: experimental, 
  path /tmp/experimental,
  field_names ['a', 'b', 'c'],
  field_types ['UInt64', 'String', 'Float64'],
  order_by ['a'],
  engine 'Merge',
  timestamp_field: 'a',
  timestamp_precision: 'ns',
  partition_by: ['b'],
  1. The processing component should create the /tmp/experimental/tmp and /tmp/experimental/data folders.

How

  1. Create an endpoint /qp/query .
  2. Add the following checks to /qp/query endpoint
    • Content-type should be either yaml or json
    • If the decoded structure doesn't have "create_table" attribute, return 400 - Not supported error
    • Check if the decoded structure has all the attributes and the attributes are of the desired type
    • Check if the name of the table and names of the fields are alphanumeric or _ character and don't start with a number.
    • Check if the fields enumerated in order_by, timestamp, partition_by are present in the fields attributes.
  3. If all the checks are passed, the system should create an entry in the tables table and create the folders on the HD.
github-actions[bot] commented 1 month ago

Thanks for opening an Issue! Please star this repository to motivate developers! :star: