search

StarRocks / starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
https://starrocks.io
Apache License 2.0
9.06k stars 1.82k forks source link

paimon catalog 不支持的字段时表不能查询 #45812

Open Oloong opened 6 months ago

Oloong commented 6 months ago

Steps to reproduce the behavior (Required)

MYSQL源表： CREATE TABLE dinky_user ( id int(11) NOT NULL AUTO_INCREMENT COMMENT 'ID', username varchar(50) NOT NULL COMMENT 'username', user_type int(11) DEFAULT '1', password varchar(50) DEFAULT NULL COMMENT 'password', nickname varchar(50) DEFAULT NULL COMMENT 'nickname', worknum varchar(50) DEFAULT NULL COMMENT 'worknum', avatar blob DEFAULT NULL COMMENT 'avatar', mobile varchar(20) DEFAULT NULL COMMENT 'mobile phone', enabled tinyint(1) NOT NULL DEFAULT '1' COMMENT 'is enable', super_admin_flag tinyint(4) DEFAULT '0' COMMENT 'is super admin(0:false,1true)', is_delete tinyint(1) NOT NULL DEFAULT '0' COMMENT 'is delete', create_time datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time', update_time datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT 'update time', PRIMARY KEY (id) USING BTREE ) ENGINE=InnoDB AUTO_INCREMENT=14 DEFAULT CHARSET=utf8mb4 ROW_FORMAT=DYNAMIC COMMENT='user'; 源MYSQL表字段是 blob 经FLINKCDC 到paimon表 转换为byte

drop CATALOG mypaimon; CREATE EXTERNAL CATALOG mypaimon comment "外部catalog数据湖paimon" PROPERTIES ("paimon.catalog.type" = "filesystem", "type" = "paimon", "paimon.catalog.warehouse" = "hdfs://bd/data/paimon" ); set catalog mypaimon; desc dinky.dinky_user; image -- 查全部 报错 select * from dinky.dinky_user

1064 - Getting analyzing error. Detail message: Datatype of external table column [avatar] is not supported!. 时间: 0.008s

-- 无数据时 不查问题列 正常 select id,username from dinky.dinky_user -- 有数据时 不查问题列 依然报错 select id,username from dinky.dinky_user

1064 - java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: Index 13 out of bounds for length 13 时间: 0.081s 1064 - java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException 时间: 0.072s select count(*) from dinky.dinky_user; 1064 - java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: Index 13 out of bounds for length 13 时间: 0.081s 1064 - java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException 时间: 0.072s 上面两种报错信息随机出现

调整好字段(源表调整成varchar)，依然报错！！！ set catalog mypaimon; desc dinky.dinky_user; image

无数据时，查询正常

有数据时，同样报错！！！ select * from dinky.dinky_user

1064 - java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: Index 13 out of bounds for length 13 时间: 0.064s

敢情PAIMON表都查不了？？？

Expected behavior (Required)

至少不在不涉及到未识别字段时不报错 建议对于不识别的字段统一转成STRING类型，或做个配置让用户自行决定是否兼容

Real behavior (Required)

报错，整个表都废了

StarRocks version (Required)

v3.1.11

github-actions[bot] commented 4 days ago

We have marked this issue as stale because it has been inactive for 6 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to StarRocks!