search

StarRocks / starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
https://starrocks.io
Apache License 2.0
8.99k stars 1.81k forks source link

flink cdc iceberg,starrocks query is errors that is Unsupported iceberg file content:2 #40043

Closed 13453547819 closed 2 months ago

13453547819 commented 9 months ago

【Details】flink cdc collect mysql data to iceberg and starrocks query iceberg catalog，but mysql row data be update, starrocks query is errors that is Unsupported iceberg file content:2。 【StarRocks version]】3.1.3 【Flink version】1.15.2 【lceberg version】1.4.2

  1. create table IF NOT EXISTS test( dt INT NOT NULL name varchar. PRIMARY KEY (dt) NOT ENFORCED )with ('format-version' = '2', 'write.upsert.enabled' = 'true', 'engine.hive.enabled' = 'true' );
  2. Flink cdc iceberg
  3. starrocks query: select * from iceberg catalog.db.test;

    Expected behavior (Required)

    image

Real behavior (Required)

image

StarRocks version (Required)

image

dirtysalt commented 9 months ago

it means we don't support equality delete feature. And it's been recently supported in this PR: https://github.com/StarRocks/starrocks/pull/38529, and released in 3.1.7

chengyi3192 commented 9 months ago

it means we don't support equality delete feature. And it's been recently supported in this PR: #38529, and released in 3.1.7

@dirtysalt Why is 3.1.7 already released, but this PR is not included in it?

chrisfw commented 9 months ago

FYI, I just encountered the same issue in 3.2 today.

chengyi3192 commented 9 months ago

FYI, I just encountered the same issue in 3.2 today.

@chrisfw The problem will be solved in sections 3.1.8 and 3.2.3, But limited to ORC format only If it is in parquet format, Official response needs to wait for version 3.3

chrisfw commented 9 months ago

Thank you for the update @chengyi3192. I appreciate your assistance.

Regards, Chris

github-actions[bot] commented 3 months ago

We have marked this issue as stale because it has been inactive for 6 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to StarRocks!