trinodb / trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
https://trino.io
Apache License 2.0
10.42k stars 3k forks source link

Support drop iceberg table when metadata file is missing #15007

Closed krvikash closed 1 year ago

krvikash commented 1 year ago

Dropping a corrupted iceberg table fails with the below error.

Query 20221112_205321_00041_rqcsp failed: Failed to open input stream for file: hdfs://hadoop-master:9000/user/hive/warehouse/test2-abbacdb6cc4f44a5b80004f8e409ff96/metadata/00001-30eb3899-262b-46de-aee3-c9a1af192c4f.metadata.json
org.apache.iceberg.exceptions.NotFoundException: Failed to open input stream for file: hdfs://hadoop-master:9000/user/hive/warehouse/test2-abbacdb6cc4f44a5b80004f8e409ff96/metadata/00001-30eb3899-262b-46de-aee3-c9a1af192c4f.metadata.json
    at io.trino.filesystem.fileio.ForwardingInputFile.newStream(ForwardingInputFile.java:55)
    at org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:265)
    at io.trino.plugin.iceberg.catalog.AbstractIcebergTableOperations.lambda$refreshFromMetadataLocation$1(AbstractIcebergTableOperations.java:221)
    at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:402)
    at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:212)
    at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:196)
    at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:189)
    at io.trino.plugin.iceberg.catalog.AbstractIcebergTableOperations.refreshFromMetadataLocation(AbstractIcebergTableOperations.java:220)
    at io.trino.plugin.iceberg.catalog.AbstractIcebergTableOperations.refresh(AbstractIcebergTableOperations.java:125)
    at io.trino.plugin.iceberg.catalog.AbstractIcebergTableOperations.current(AbstractIcebergTableOperations.java:108)
    at io.trino.plugin.iceberg.catalog.hms.TrinoHiveCatalog.lambda$loadTable$10(TrinoHiveCatalog.java:333)
    at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708)
    at io.trino.plugin.iceberg.catalog.hms.TrinoHiveCatalog.loadTable(TrinoHiveCatalog.java:331)
    at io.trino.plugin.iceberg.IcebergMetadata.getTableHandle(IcebergMetadata.java:343)
    at io.trino.plugin.iceberg.IcebergMetadata.getTableHandle(IcebergMetadata.java:255)
    at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorMetadata.getTableHandle(ClassLoaderSafeConnectorMetadata.java:1094)
    at io.trino.metadata.MetadataManager.lambda$getTableHandle$4(MetadataManager.java:280)
    at java.base/java.util.Optional.flatMap(Optional.java:289)
    at io.trino.metadata.MetadataManager.getTableHandle(MetadataManager.java:274)
    at io.trino.metadata.MetadataManager.getRedirectionAwareTableHandle(MetadataManager.java:1541)
    at io.trino.metadata.MetadataManager.getRedirectionAwareTableHandle(MetadataManager.java:1533)
    at io.trino.execution.DropTableTask.execute(DropTableTask.java:79)
    at io.trino.execution.DropTableTask.execute(DropTableTask.java:37)
    at io.trino.execution.DataDefinitionExecution.start(DataDefinitionExecution.java:145)
    at io.trino.execution.SqlQueryManager.createQuery(SqlQueryManager.java:249)
    at io.trino.dispatcher.LocalDispatchQuery.lambda$startExecution$7(LocalDispatchQuery.java:143)
    at io.trino.$gen.Trino_dev____20221112_200441_2.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)

Native Iceberg supports dropping a corrupted iceberg table via https://github.com/apache/iceberg/pull/5510

ebyhr commented 1 year ago

Is this the same issue as #12318? By the way, "corrupted" is ambiguous. It would be nice if we can use more clear keyword.

krvikash commented 1 year ago

Yes, this is the same as https://github.com/trinodb/trino/issues/12318.

findepi commented 1 year ago

Closing in favor of https://github.com/trinodb/trino/issues/12318