search

StarRocks / starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
https://starrocks.io
Apache License 2.0
8.96k stars 1.8k forks source link

Can't create iceberg table with zstd parquet compression #52296

Open zig opened 2 weeks ago

zig commented 2 weeks ago

Steps to reproduce the behavior (Required)

CREATE TABLE  IF NOT EXISTS test
(
    id bigint NOT NULL,
    time datetime NOT NULL,
    session_id text,
    truncated_time text
)
PARTITION BY (truncated_time)
PROPERTIES (
"write.parquet.compression-codec" = "zstd",
"write.parquet.compression-level" = "3"
);

Expected behavior (Required)

Success

Real behavior (Required)

SQL Error [1064] [42000]: Multiple entries with same key: write.parquet.compression-codec=gzip and write.parquet.compression-codec=zstd

StarRocks version (Required)

3.3.5

zig commented 2 weeks ago

If I write instead : 

CREATE TABLE  IF NOT EXISTS test
(
    id bigint NOT NULL,
    time datetime NOT NULL,
    session_id text,
    truncated_time text
)
PARTITION BY (truncated_time)
PROPERTIES (
"compression_codec" = "zstd",
"write.parquet.compression-level" = "3"
);

then it works.

However, if I INSERT OVERWRITE the table, the generated parquet files are not compressed. Also SHOW CREATE TABLE does not show any of the compression property.