Use Iceberg Metadata table APIs while implementing Iceberg Metadata tables in Trino.
Right now in Trino, we are using the underlying iceberg APIs like icebergTable.newScan() for $files and snapshot.allManifiests() to implement $manifests metadata table.
Instead of the underlying APIs, we can use the org.apache.iceberg.MetadataTableUtils.createMetadataTableInstance to get the relevant metadata table object and then scan this table.
This way we will get benefit from the underlying Iceberg metadata table implementation, and any change will be reflected in Trino as well, so less regression.
Spark engine is also using similar metadata table implementations/API to support Iceberg's metadata tables, reference.
osscm
commented
2 years ago
Use Iceberg Metadata table APIs while implementing Iceberg Metadata tables in Trino. Right now in Trino, we are using the underlying iceberg APIs like
icebergTable.newScan()for $files andsnapshot.allManifiests()to implement $manifests metadata table.Instead of the underlying APIs, we can use the
org.apache.iceberg.MetadataTableUtils.createMetadataTableInstanceto get the relevant metadata table object and then scan this table. This way we will get benefit from the underlying Iceberg metadata table implementation, and any change will be reflected in Trino as well, so less regression. Spark engine is also using similar metadata table implementations/API to support Iceberg's metadata tables, reference.related issue: https://github.com/trinodb/trino/issues/11172