search

apache / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.
https://gravitino.apache.org
Apache License 2.0
1.1k stars 345 forks source link

[Bug report]In some case, initialize Catalog Tables e.g. iceberg_tables, iceberg_namespace_properties failed #2687

Open danhuawang opened 8 months ago

danhuawang commented 8 months ago

Version

main branch

Describe what's wrong

  1. create a schema named mysql_schema_test in mysql backend db
  2. create iceberg catalog in gravitino with mysql backend ,parameter as following
catalog-backend jdbc
jdbc-user root
jdbc-driver com.mysql.cj.jdbc.Driver
jdbc-password mysql
warehouse file:///user/hive/warehouse-hive/
uri jdbc:mysql://127.0.0.1:3306/mysql_schema_test
  1. There's error: Failed to execute query: SELECT DISTINCT table_namespace FROM iceberg_tables WHERE catalog_name = ?
image

Error message and/or stacktrace

2024-03-26 17:54:18.181 INFO [Gravitino-webserver-93] [org.apache.iceberg.CatalogUtil.loadFileIO(CatalogUtil.java:302)] - Loading custom FileIO implementation: org.apache.iceberg.hadoop.HadoopFileIO
2024-03-26 17:54:19.870 WARN [Gravitino-webserver-57] [com.datastrato.gravitino.server.web.rest.ExceptionHandlers$SchemaExceptionHandler.handle(ExceptionHandlers.java:143)] - Failed to operate schema(s) operation [LIST] under catalog [ice888], reason [Failed to execute query: SELECT DISTINCT table_namespace FROM iceberg_tables WHERE catalog_name = ?]
java.lang.RuntimeException: org.apache.iceberg.jdbc.UncheckedSQLException: Failed to execute query: SELECT DISTINCT table_namespace FROM iceberg_tables WHERE catalog_name = ?
    at com.datastrato.gravitino.catalog.CatalogOperationDispatcher.doWithCatalog(CatalogOperationDispatcher.java:754) ~[gravitino-core-0.5.0-SNAPSHOT.jar:?]
    at com.datastrato.gravitino.catalog.CatalogOperationDispatcher.listSchemas(CatalogOperationDispatcher.java:105) ~[gravitino-core-0.5.0-SNAPSHOT.jar:?]
    at com.datastrato.gravitino.server.web.rest.SchemaOperations.lambda$listSchemas$0(SchemaOperations.java:74) ~[gravitino-server-0.5.0-SNAPSHOT.jar:?]
    at com.datastrato.gravitino.lock.TreeLockUtils.doWithTreeLock(TreeLockUtils.java:35) ~[gravitino-core-0.5.0-SNAPSHOT.jar:?]
    at com.datastrato.gravitino.server.web.rest.SchemaOperations.lambda$listSchemas$1(SchemaOperations.java:71) ~[gravitino-server-0.5.0-SNAPSHOT.jar:?]
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
    at java.base/javax.security.auth.Subject.doAs(Subject.java:439) ~[?:?]
    at com.datastrato.gravitino.utils.PrincipalUtils.doAs(PrincipalUtils.java:25) ~[gravitino-core-0.5.0-SNAPSHOT.jar:?]
    at com.datastrato.gravitino.server.web.Utils.doAs(Utils.java:121) ~[gravitino-server-0.5.0-SNAPSHOT.jar:?]
    at com.datastrato.gravitino.server.web.rest.SchemaOperations.listSchemas(SchemaOperations.java:66) ~[gravitino-server-0.5.0-SNAPSHOT.jar:?]
    at jdk.internal.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) ~[?:?]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
    at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.41.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) ~[jersey-server-2.41.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) ~[jersey-server-2.41.jar:?]
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176) ~[jersey-server-2.41.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) ~[jersey-server-2.41.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) ~[jersey-server-2.41.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) ~[jersey-server-2.41.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ~[jersey-server-2.41.jar:?]
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) ~[jersey-server-2.41.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[jersey-common-2.41.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[jersey-common-2.41.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.41.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.41.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[jersey-common-2.41.jar:?]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[jersey-common-2.41.jar:?]
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) ~[jersey-server-2.41.jar:?]
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) ~[jersey-server-2.41.jar:?]
    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[jersey-container-servlet-core-2.41.jar:?]
    at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[jersey-container-servlet-core-2.41.jar:?]
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358) ~[jersey-container-servlet-core-2.41.jar:?]
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311) ~[jersey-container-servlet-core-2.41.jar:?]
    at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[jersey-container-servlet-core-2.41.jar:?]
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
    at com.datastrato.gravitino.server.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:59) ~[gravitino-server-common-0.5.0-SNAPSHOT.jar:?]
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
    at com.datastrato.gravitino.server.web.VersioningFilter.doFilter(VersioningFilter.java:97) ~[gravitino-server-0.5.0-SNAPSHOT.jar:?]
    at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) ~[jetty-security-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
    at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: org.apache.iceberg.jdbc.UncheckedSQLException: Failed to execute query: SELECT DISTINCT table_namespace FROM iceberg_tables WHERE catalog_name = ?
    at org.apache.iceberg.jdbc.JdbcCatalog.fetch(JdbcCatalog.java:543) ~[?:?]
    at org.apache.iceberg.jdbc.JdbcCatalog.listNamespaces(JdbcCatalog.java:308) ~[?:?]
    at org.apache.iceberg.rest.CatalogHandlers.listNamespaces(CatalogHandlers.java:98) ~[?:?]
    at com.datastrato.gravitino.catalog.lakehouse.iceberg.ops.IcebergTableOps.listNamespace(IcebergTableOps.java:85) ~[?:?]
    at com.datastrato.gravitino.catalog.lakehouse.iceberg.IcebergCatalogOperations.listSchemas(IcebergCatalogOperations.java:131) ~[?:?]
    at com.datastrato.gravitino.catalog.CatalogOperationDispatcher.lambda$listSchemas$0(CatalogOperationDispatcher.java:107) ~[gravitino-core-0.5.0-SNAPSHOT.jar:?]
    at com.datastrato.gravitino.catalog.CatalogManager$CatalogWrapper.lambda$doWithSchemaOps$0(CatalogManager.java:95) ~[gravitino-core-0.5.0-SNAPSHOT.jar:?]
    at com.datastrato.gravitino.utils.IsolatedClassLoader.withClassLoader(IsolatedClassLoader.java:69) ~[gravitino-core-0.5.0-SNAPSHOT.jar:?]
    at com.datastrato.gravitino.catalog.CatalogManager$CatalogWrapper.doWithSchemaOps(CatalogManager.java:90) ~[gravitino-core-0.5.0-SNAPSHOT.jar:?]
    at com.datastrato.gravitino.catalog.CatalogOperationDispatcher.lambda$listSchemas$1(CatalogOperationDispatcher.java:107) ~[gravitino-core-0.5.0-SNAPSHOT.jar:?]
    at com.datastrato.gravitino.catalog.CatalogOperationDispatcher.doWithCatalog(CatalogOperationDispatcher.java:749) ~[gravitino-core-0.5.0-SNAPSHOT.jar:?]
    ... 74 more
Caused by: java.sql.SQLSyntaxErrorException: Table 'mysql_schema_test.iceberg_tables' doesn't exist
    at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120) ~[mysql-connector-java-8.0.27.jar:8.0.27]
    at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) ~[mysql-connector-java-8.0.27.jar:8.0.27]
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:953) ~[mysql-connector-java-8.0.27.jar:8.0.27]
    at com.mysql.cj.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:1009) ~[mysql-connector-java-8.0.27.jar:8.0.27]
    at org.apache.iceberg.jdbc.JdbcCatalog.lambda$fetch$17(JdbcCatalog.java:533) ~[?:?]
    at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:58) ~[?:?]
    at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51) ~[?:?]
    at org.apache.iceberg.jdbc.JdbcCatalog.fetch(JdbcCatalog.java:524) ~[?:?]
    at org.apache.iceberg.jdbc.JdbcCatalog.listNamespaces(JdbcCatalog.java:308) ~[?:?]
    at org.apache.iceberg.rest.CatalogHandlers.listNamespaces(CatalogHandlers.java:98) ~[?:?]
    at com.datastrato.gravitino.catalog.lakehouse.iceberg.ops.IcebergTableOps.listNamespace(IcebergTableOps.java:85) ~[?:?]
    at com.datastrato.gravitino.catalog.lakehouse.iceberg.IcebergCatalogOperations.listSchemas(IcebergCatalogOperations.java:131) ~[?:?]
    at com.datastrato.gravitino.catalog.CatalogOperationDispatcher.lambda$listSchemas$0(CatalogOperationDispatcher.java:107) ~[gravitino-core-0.5.0-SNAPSHOT.jar:?]
    at com.datastrato.gravitino.catalog.CatalogManager$CatalogWrapper.lambda$doWithSchemaOps$0(CatalogManager.java:95) ~[gravitino-core-0.5.0-SNAPSHOT.jar:?]
    at com.datastrato.gravitino.utils.IsolatedClassLoader.withClassLoader(IsolatedClassLoader.java:69) ~[gravitino-core-0.5.0-SNAPSHOT.jar:?]
    at com.datastrato.gravitino.catalog.CatalogManager$CatalogWrapper.doWithSchemaOps(CatalogManager.java:90) ~[gravitino-core-0.5.0-SNAPSHOT.jar:?]
    at com.datastrato.gravitino.catalog.CatalogOperationDispatcher.lambda$listSchemas$1(CatalogOperationDispatcher.java:107) ~[gravitino-core-0.5.0-SNAPSHOT.jar:?]
    at com.datastrato.gravitino.catalog.CatalogOperationDispatcher.doWithCatalog(CatalogOperationDispatcher.java:749) ~[gravitino-core-0.5.0-SNAPSHOT.jar:?]
    ... 74 more

How to reproduce

  1. create a schema named mysql_schema_test in mysql backend db
  2. create iceberg catalog in gravitino with mysql backend ,using the db which created in step1

Additional context

In this situation , need to create iceberg_tables, iceberg_namespace_properties in mysql backend by hand

FANNG1 commented 8 months ago

In this situation , need to create iceberg_tables, iceberg_namespace_properties in mysql backend by hand

You mean if you create these tables by hand, the error not exists?

danhuawang commented 8 months ago

In this situation , need to create iceberg_tables, iceberg_namespace_properties in mysql backend by hand

You mean if you create these tables by hand, the error not exists?

Yes, create these tables by hand, the error not exists.

FANNG1 commented 8 months ago

it's expected to create when load catalog. are you set jdbc-initialize to false explicitly? do you save gravitino server logs now?

danhuawang commented 8 months ago

it's expected to create when load catalog. are you set jdbc-initialize to false explicitly? do you save gravitino server logs now?

jdbc-initialize using the default value.

danhuawang commented 8 months ago

gravitino-server.log

You can see the logs after 2024-03-28 10:29:34.277 When I reproduce the issues

FANNG1 commented 7 months ago

There a bug of Iceberg JDBC catalog when creating two JDBC catalogs using different database names, I guess you had create another Iceberg Jdbc catalog earlier with different database, YES?

danhuawang commented 7 months ago

There a bug of Iceberg JDBC catalog when creating two JDBC catalogs using different database names, I guess you had create another Iceberg Jdbc catalog earlier with different database, YES?

Yes.

FANNG1 commented 3 months ago

delay to 0.7

FANNG1 commented 4 weeks ago

Tracked in https://github.com/apache/iceberg/issues/11423