search

StarRocks / starrocks

StarRocks, a Linux Foundation project, is a next-generation sub-second MPP OLAP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics, and ad-hoc queries.
https://starrocks.io
Apache License 2.0
8.6k stars 1.74k forks source link

show tables will fail against iceberg catalog when a iceberg table's metadata json file is lost #43154

Open zhangbutao opened 5 months ago

zhangbutao commented 5 months ago

Step to reproduce this issue:

Env: main branch

1. Create the iceberg catalog:

create external catalog icebergcat
PROPERTIES 
("type"="iceberg","iceberg.catalog.type"="hive", 
"hive.metastore.uris"="thrift://127.0.01:9083");

2. create a iceberg tabel in hive db testsrdb: CREATE TABLE icebergcat.testsrdb.ice_par (action varchar(20),id int,dt date) PARTITION BY (id,dt);

3. delete the iceberg table's metadata json file: hdfs dfs -rm -r hdfs://127.0.0.1:8028/user/hive/warehouse/external/hiveicetest/testsrdb.db/ice_par/metadata/*

4. restart fe to avoid cache effects As fe cache will load the table's metadata file when creating table, so this step just to invalidate cache to keep fe reload the iceberg table.

5. show tables

 use icebergcat.testsrdb;
 show tables;

Then, the mysql client occurs exception:

MySQL [icebergcat.testsrdb]> show tables;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id:    14
Current database: icebergcat2.testsrdb3

ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id:    15
Current database: icebergcat2.testsrdb3

ERROR 2006 (HY000): MySQL server has gone away

And check the fe server side's log, found exception: org.apache.iceberg.exceptions.NotFoundException: File does not exist: 

2024-03-26 09:12:57.924Z WARN (starrocks-mysql-nio-pool-2|244) [ReadListener.lambda$handleEvent$0():81] Exception happened in one session(com.starrocks.mysql.nio.NConnectContext@4c250990).
org.apache.iceberg.exceptions.NotFoundException: File does not exist: hdfs://127.0.0.1:8028/user/hive/warehouse/external/hiveicetest/testsrdb.db/ice_par/metadata/00000-ec3abcdd-bf4a-4bfb-a2a5-671d968c2a68.metadata.json
        at org.apache.iceberg.hadoop.HadoopInputFile.lazyStat(HadoopInputFile.java:164) ~[iceberg-core-1.5.0.jar:?]
        at org.apache.iceberg.hadoop.HadoopInputFile.getLength(HadoopInputFile.java:175) ~[iceberg-core-1.5.0.jar:?]
        at com.starrocks.connector.iceberg.io.IcebergCachingFileIO$CachingInputFile.getLength(IcebergCachingFileIO.java:530) ~[starrocks-fe.jar:?]
        at com.starrocks.connector.iceberg.io.IcebergCachingFileIO$CachingInputFile.newStream(IcebergCachingFileIO.java:537) ~[starrocks-fe.jar:?]
        at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:279) ~[iceberg-core-1.5.0.jar:?]
        at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:273) ~[iceberg-core-1.5.0.jar:?]
        at org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$0(BaseMetastoreTableOperations.java:189) ~[iceberg-core-1.5.0.jar:?]
        at org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$1(BaseMetastoreTableOperations.java:208) ~[iceberg-core-1.5.0.jar:?]
        at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:413) ~[iceberg-core-1.5.0.jar:?]
        at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:219) ~[iceberg-core-1.5.0.jar:?]
        at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:203) ~[iceberg-core-1.5.0.jar:?]
        at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196) ~[iceberg-core-1.5.0.jar:?]
        at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:208) ~[iceberg-core-1.5.0.jar:?]
        at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:185) ~[iceberg-core-1.5.0.jar:?]
        at org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:180) ~[iceberg-core-1.5.0.jar:?]
        at org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:166) ~[iceberg-hive-metastore-1.5.0.jar:?]
        at org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:97) ~[iceberg-core-1.5.0.jar:?]
        at org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:80) ~[iceberg-core-1.5.0.jar:?]
        at org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:49) ~[iceberg-core-1.5.0.jar:?]
        at com.starrocks.connector.iceberg.hive.IcebergHiveCatalog.getTable(IcebergHiveCatalog.java:103) ~[starrocks-fe.jar:?]
        at com.starrocks.connector.iceberg.IcebergMetadata.getTable(IcebergMetadata.java:284) ~[starrocks-fe.jar:?]
        at com.starrocks.connector.CatalogConnectorMetadata.getTable(CatalogConnectorMetadata.java:117) ~[starrocks-fe.jar:?]
        at com.starrocks.server.MetadataMgr.lambda$getTable$3(MetadataMgr.java:348) ~[starrocks-fe.jar:?]
        at java.util.Optional.map(Optional.java:265) ~[?:?]
        at com.starrocks.server.MetadataMgr.getTable(MetadataMgr.java:348) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.handleFieldList(ConnectProcessor.java:472) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:616) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:901) ~[starrocks-fe.jar:?]
        at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://127.0.0.1:8028/user/hive/warehouse/external/hiveicetest/testsrdb.db/ice_par/metadata/00000-ec3abcdd-bf4a-4bfb-a2a5-671d968c2a68.metadata.json
        at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1833) ~[hadoop-hdfs-client-3.3.6.jar:?]
        at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1826) ~[hadoop-hdfs-client-3.3.6.jar:?]
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-3.3.6.jar:?]
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1841) ~[hadoop-hdfs-client-3.3.6.jar:?]
        at org.apache.iceberg.hadoop.HadoopInputFile.lazyStat(HadoopInputFile.java:162) ~[iceberg-core-1.5.0.jar:?]
stephen-shelby commented 5 months ago

what the value of list-all-tables in your catalog properties.

zhangbutao commented 5 months ago

what the value of list-all-tables in your catalog properties.

This problem occurs with either true or false.