housepower / clickhouse_sinker

Easily load data from kafka to ClickHouse
https://housepower.github.io/clickhouse_sinker
Apache License 2.0
515 stars 118 forks source link

Support for `Nested` data type #102

Closed fzyzcjy closed 3 years ago

fzyzcjy commented 3 years ago

Hi thanks for the lib! I wonder whether it supports the Nested data type in clickhouse? Since it is very common.

toannhu96 commented 3 years ago

it's already supported in #56 here.

fzyzcjy commented 3 years ago

@toannhu96 Sorry it seems that the situation is a bit different?

For example, I want:

CREATE TABLE daily (
    day Date,
   apple Nested(
     a String,
      b UInt64
    )
  ) ...;

and for the data:

{
   "day":1565712000,
   "apple.a": ["aaa","bbb"],
   "apple.b": [100,200]
}
toannhu96 commented 3 years ago

@fzyzcjy this repo is based on https://github.com/ClickHouse/clickhouse-go and this has still not supported Nested Column yet. image

I just found another clickhouse golang driver lib that supported nested datatype https://github.com/mailru/go-clickhouse image

fzyzcjy commented 3 years ago

@toannhu96 I see that. So how do you use it daily? IMHO Nested is quite frequently used.

sundy-li commented 3 years ago

I think we can use Array<T> Instead...

Nested Data Type is tuple of Array<T> in ClickHouse.

sundy-li commented 3 years ago
CREATE TABLE b
(
    `struct` Nested(key String, value Int32, m Float32)
)
ENGINE = Memory

macbook-pro-2.local :)    show create table b;

SHOW CREATE TABLE b

Query id: 5b101be1-3393-454b-953a-4973c8f11238

┌─statement───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ CREATE TABLE default.b
(
    `struct.key` Array(String),
    `struct.value` Array(Int32),
    `struct.m` Array(Float32)
)
ENGINE = Memory │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

1 rows in set. Elapsed: 0.001 sec.
fzyzcjy commented 3 years ago

Thanks!