housepower / clickhouse_sinker

Easily load data from kafka to ClickHouse
https://housepower.github.io/clickhouse_sinker
Apache License 2.0
515 stars 118 forks source link

More Data Types Support #100

Closed yuzhichang closed 2 years ago

yuzhichang commented 3 years ago

ClickHouse and clickhouse-go support:

ClickHouse supports but clickhouse-go not:

There's UUID support in this fork.

fzyzcjy commented 3 years ago

Hi, I need to use Nested (which seems very commonly used), so I wonder when will it be supported? If needed I can also contribute.

yuzhichang commented 3 years ago

@fzyzcjy Nested is actually Array(T). Both clickhouse-go and clickhouse_sinker support Array(T). For example,


CREATE TABLE test
(
    `@time` DateTime,
    `@item_guid` String,
    `@metric_name` LowCardinality(String),
    `@alg_name` LowCardinality(String),
    `detail` Nested(    is_p_item UInt8,     sequency_id UInt16),
    `upper` Array(Float64)
)
ENGINE = ReplacingMergeTree
PARTITION BY toYYYYMMDD(`@time`)
ORDER BY (`@time`, `@item_guid`, `@metric_name`)

SELECT 
    name,
    type,
    default_kind
FROM system.columns
WHERE table = 'test'

┌─name───────────────┬─type───────────────────┬─default_kind─┐
│ @time              │ DateTime               │              │
│ @item_guid         │ String                 │              │
│ @metric_name       │ LowCardinality(String) │              │
│ @alg_name          │ LowCardinality(String) │              │
│ detail.is_p_item   │ Array(UInt8)           │              │
│ detail.sequency_id │ Array(UInt16)          │              │
│ upper              │ Array(Float64)         │              │
└────────────────────┴────────────────────────┴──────────────┘
fzyzcjy commented 3 years ago

@yuzhichang Thanks!

josepowera commented 3 years ago

Array from "csv" in csv - see segments below

publisherid, userid, segments 1,2032030,"[40,30,11]"

yuzhichang commented 3 years ago

@josepowera master 090a1b0 added array support in csv parser.

yuzhichang commented 3 years ago

master 60101302721919da9245295769a360c8bea88fce added Decimal support.

yuzhichang commented 2 years ago

master 0a0199a added UUID and Enum support