apache / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.
https://gravitino.apache.org
Apache License 2.0
1.1k stars 344 forks source link

[Improvement] Trino connector doesn't support Iceberg catalog that backend type is 'rest' #5514

Open danhuawang opened 2 weeks ago

danhuawang commented 2 weeks ago

What would you like to be improved?

trino> call gravitino.system.create_catalog(
    ->     'ice_rest0',
    ->     'lakehouse-iceberg',
    ->     map(
    ->         array['uri', 'catalog-backend', 'warehouse','io-impl','s3-endpoint','trino.bypass.hive.s3.aws-access-key','trino.bypass.hive.s3.aws-secret-key', 'trino.bypass.hive.s3.region', 's3-access-key-id', 's3-secret-access-key', 's3-region'],
    ->         array['http://iceberg-rest:9001/iceberg/', 'rest', 's3://trino-s3-test/rest/user/iceberg/warehouse/','org.apache.iceberg.aws.s3.S3FileIO','http://s3.us-east-1.amazonaws.com/','****', '****', 'us-east-1','***', '***', 'us-east-1']  ));

Query 20241107_100201_00094_nfhgp failed: Create catalog failed. Create catalog failed due to the loading process fails
io.trino.spi.TrinoException: Create catalog failed. Create catalog failed due to the loading process fails
    at org.apache.gravitino.trino.connector.system.storedprocdure.CreateCatalogStoredProcedure.createCatalog(CreateCatalogStoredProcedure.java:118)
    at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:733)
    at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:758)
    at io.trino.execution.CallTask.execute(CallTask.java:210)
    at io.trino.execution.CallTask.execute(CallTask.java:70)
    at io.trino.execution.DataDefinitionExecution.start(DataDefinitionExecution.java:145)
    at io.trino.execution.SqlQueryManager.createQuery(SqlQueryManager.java:256)
    at io.trino.dispatcher.LocalDispatchQuery.startExecution(LocalDispatchQuery.java:145)
    at io.trino.dispatcher.LocalDispatchQuery.lambda$waitForMinimumWorkers$2(LocalDispatchQuery.java:129)
    at io.airlift.concurrent.MoreFutures.lambda$addSuccessCallback$12(MoreFutures.java:568)
    at io.airlift.concurrent.MoreFutures$3.onSuccess(MoreFutures.java:543)
    at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1133)
    at io.trino.$gen.Trino_435____20241107_095109_2.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: io.trino.spi.TrinoException: Create catalog failed due to the loading process fails
    at org.apache.gravitino.trino.connector.system.storedprocdure.CreateCatalogStoredProcedure.createCatalog(CreateCatalogStoredProcedure.java:100)
    ... 15 more
Caused by: io.trino.jdbc.$internal.client.FailureInfo$FailureException: Unsupported backend type: rest
    at org.apache.gravitino.trino.connector.catalog.iceberg.IcebergCatalogPropertyConverter.gravitinoToEngineProperties(IcebergCatalogPropertyConverter.java:317) ~[gravitino-trino-connector-0.8.0-incubating-SNAPSHOT.jar:?]
    at org.apache.gravitino.trino.connector.catalog.iceberg.IcebergConnectorAdapter.buildInternalConnectorConfig(IcebergConnectorAdapter.java:49) ~[gravitino-trino-connector-0.8.0-incubating-SNAPSHOT.jar:?]
    at org.apache.gravitino.trino.connector.catalog.CatalogConnectorContext$Builder.build(CatalogConnectorContext.java:132) ~[gravitino-trino-connector-0.8.0-incubating-SNAPSHOT.jar:?]
    at org.apache.gravitino.trino.connector.catalog.CatalogConnectorManager.createConnector(CatalogConnectorManager.java:298) ~[gravitino-trino-connector-0.8.0-incubating-SNAPSHOT.jar:?]
    at org.apache.gravitino.trino.connector.GravitinoConnectorFactory.create(GravitinoConnectorFactory.java:102) ~[gravitino-trino-connector-0.8.0-incubating-SNAPSHOT.jar:?]
    at io.trino.connector.DefaultCatalogFactory.createConnector(DefaultCatalogFactory.java:200) ~[?:?]
    at io.trino.connector.DefaultCatalogFactory.createCatalog(DefaultCatalogFactory.java:118) ~[?:?]
    at io.trino.connector.LazyCatalogFactory.createCatalog(LazyCatalogFactory.java:43) ~[?:?]
    at io.trino.connector.CoordinatorDynamicCatalogManager.lambda$createCatalog$4(CoordinatorDynamicCatalogManager.java:272) ~[?:?]
    at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) ~[?:?]
    at io.trino.connector.CoordinatorDynamicCatalogManager.createCatalog(CoordinatorDynamicCatalogManager.java:270) ~[?:?]
    at io.trino.execution.CreateCatalogTask.execute(CreateCatalogTask.java:98) ~[?:?]
    at io.trino.execution.CreateCatalogTask.execute(CreateCatalogTask.java:42) ~[?:?]
    at io.trino.execution.DataDefinitionExecution.start(DataDefinitionExecution.java:145) ~[?:?]
    at io.trino.execution.SqlQueryManager.createQuery(SqlQueryManager.java:256) ~[?:?]
    at io.trino.dispatcher.LocalDispatchQuery.startExecution(LocalDispatchQuery.java:145) ~[?:?]
    at io.trino.dispatcher.LocalDispatchQuery.lambda$waitForMinimumWorkers$2(LocalDispatchQuery.java:129) ~[?:?]
    at io.airlift.concurrent.MoreFutures.lambda$addSuccessCallback$12(MoreFutures.java:568) ~[?:?]
    at io.airlift.concurrent.MoreFutures$3.onSuccess(MoreFutures.java:543) ~[?:?]
    at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1133) ~[guava-32.1.3-jre.jar:?]
    at io.trino.$gen.Trino_435____20241107_095109_2.run(Unknown Source) ~[?:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
    at java.base/java.lang.Thread.run(Thread.java:1583) ~[?:?]

How should we improve?

Trino connector can support  Iceberg catalog that backend type is 'rest'