search

apache / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.
https://gravitino.apache.org
Apache License 2.0
1.09k stars 343 forks source link

[Bug report] Gravitino container load metalake encounter: NullPointerException: Cannot invoke "java.util.Map.size()" because "m" is null #5237

Closed danhuawang closed 4 weeks ago

danhuawang commented 4 weeks ago

Version

main branch

Describe what's wrong

Run trino test failed.

./trino_test.sh --auto=none --ignore_failed --postgresql_uri=jdbc:postgresql://pg-16-postgresql.relational-catalog:5432 --mysql_uri=jdbc:mysql://my-8-mysql.relational-catalog:3306 --hdfs_uri=hdfs://hadoop-hdfs-namenode.hadoop:8020 --hive_uri=thrift://hadoop-metastore.hadoop:9083 --gravitino_uri=[http://*-1128096743.***.elb.amazonaws.com:8090](http://*-1128096743.%2A%2A%2A.elb.amazonaws.com:8090/) --trino_uri=[http://*-1494066321.***.elb.amazonaws.com:8080](http://acd3e8739d6a54c3088e4da8f496ed8d-1494066321.%2A%2A%2A.elb.amazonaws.com:8080/)
image

Error message and/or stacktrace

2024-10-24 07:13:07.434 INFO [main] [org.apache.gravitino.metrics.MetricsSystem.register(MetricsSystem.java:78)] - Register gravitino-server to metrics system
2024-10-24 07:13:07.448 INFO [main] [org.eclipse.jetty.server.Server.doStart(Server.java:375)] - jetty-9.4.51.v20230217; built: 2023-02-17T08:19:37.309Z; git: b45c405e4544384de066f814ed42ae3dceacdd49; jvm 17.0.2+8-86
2024-10-24 07:13:07.815 INFO [main] [org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:921)] - Started o.e.j.s.ServletContextHandler@48268eec{/,null,AVAILABLE}
2024-10-24 07:13:07.826 INFO [main] [org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:333)] - Started ServerConnector@27585351{HTTP/1.1, (http/1.1)}{0.0.0.0:9001}
2024-10-24 07:13:07.826 INFO [main] [org.eclipse.jetty.server.Server.doStart(Server.java:415)] - Started @1541ms
2024-10-24 07:13:07.826 WARN [main] [org.apache.gravitino.server.web.JettyServer.start(JettyServer.java:199)] - Users would better use HTTPS to avoid token data leak.
2024-10-24 07:13:07.826 INFO [main] [org.apache.gravitino.server.web.JettyServer.start(JettyServer.java:202)] - iceberg-rest web server started on host 0.0.0.0 port 9001.
2024-10-24 07:13:07.827 INFO [main] [org.apache.gravitino.iceberg.RESTService.serviceStart(RESTService.java:128)] - Iceberg REST service started
2024-10-24 07:13:07.827 INFO [main] [org.eclipse.jetty.server.Server.doStart(Server.java:375)] - jetty-9.4.51.v20230217; built: 2023-02-17T08:19:37.309Z; git: b45c405e4544384de066f814ed42ae3dceacdd49; jvm 17.0.2+8-86
2024-10-24 07:13:07.904 INFO [main] [org.eclipse.jetty.webapp.StandardDescriptorProcessor.visitServlet(StandardDescriptorProcessor.java:277)] - NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
2024-10-24 07:13:07.908 INFO [main] [org.eclipse.jetty.server.session.DefaultSessionIdManager.doStart(DefaultSessionIdManager.java:334)] - DefaultSessionIdManager workerName=node0
2024-10-24 07:13:07.909 INFO [main] [org.eclipse.jetty.server.session.DefaultSessionIdManager.doStart(DefaultSessionIdManager.java:339)] - No SessionScavenger set, using defaults
2024-10-24 07:13:07.910 INFO [main] [org.eclipse.jetty.server.session.HouseKeeper.startScavenging(HouseKeeper.java:132)] - node0 Scavenging every 600000ms
2024-10-24 07:13:08.036 INFO [main] [org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:921)] - Started o.e.j.w.WebAppContext@544e3679{gravitino-web,/,jar:file:///root/gravitino/web/gravitino-web-0.7.0-incubating-SNAPSHOT.war!/,AVAILABLE}{/root/gravitino/web/gravitino-web-0.7.0-incubating-SNAPSHOT.war}
2024-10-24 07:13:08.037 INFO [main] [org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:333)] - Started ServerConnector@6889f56f{HTTP/1.1, (http/1.1)}{0.0.0.0:8090}
2024-10-24 07:13:08.038 INFO [main] [org.eclipse.jetty.server.Server.doStart(Server.java:415)] - Started @1752ms
2024-10-24 07:13:08.038 WARN [main] [org.apache.gravitino.server.web.JettyServer.start(JettyServer.java:199)] - Users would better use HTTPS to avoid token data leak.
2024-10-24 07:13:08.038 INFO [main] [org.apache.gravitino.server.web.JettyServer.start(JettyServer.java:202)] - Gravitino-webserver web server started on host 0.0.0.0 port 8090.
2024-10-24 07:13:08.038 INFO [main] [org.apache.gravitino.server.GravitinoServer.main(GravitinoServer.java:174)] - Done, Gravitino server started.
2024-10-24 07:14:07.403 INFO [tree-lock-cleaner-0] [org.apache.gravitino.lock.LockManager.lambda$startNodeCleaner$3(LockManager.java:159)] - Total tree lock node count: 1
2024-10-24 07:14:07.411 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:114)] - Start to check the dead lock...
2024-10-24 07:14:07.411 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:116)] - Finish to check the dead lock...
2024-10-24 07:14:55.093 INFO [Gravitino-webserver-34] [org.apache.gravitino.server.web.rest.MetalakeOperations.listMetalakes(MetalakeOperations.java:84)] - Received list metalakes request.
2024-10-24 07:14:55.173 INFO [Gravitino-webserver-34] [org.apache.gravitino.server.web.rest.MetalakeOperations.lambda$listMetalakes$1(MetalakeOperations.java:94)] - List 0 metalakes in Gravitino
2024-10-24 07:14:55.247 INFO [Gravitino-webserver-45] [org.apache.gravitino.server.web.rest.MetalakeOperations.createMetalake(MetalakeOperations.java:109)] - Received create metalake request for test
2024-10-24 07:14:55.263 INFO [Gravitino-webserver-45] [org.apache.gravitino.server.web.rest.MetalakeOperations.lambda$createMetalake$3(MetalakeOperations.java:123)] - Metalake created: test
2024-10-24 07:15:07.403 INFO [tree-lock-cleaner-0] [org.apache.gravitino.lock.LockManager.lambda$startNodeCleaner$3(LockManager.java:159)] - Total tree lock node count: 1
2024-10-24 07:15:07.411 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:114)] - Start to check the dead lock...
2024-10-24 07:15:07.411 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:116)] - Finish to check the dead lock...
2024-10-24 07:15:59.853 INFO [Gravitino-webserver-45] [org.apache.gravitino.server.web.rest.MetalakeOperations.loadMetalake(MetalakeOperations.java:138)] - Received load metalake request for metalake: test
2024-10-24 07:15:59.883 WARN [Gravitino-webserver-45] [org.apache.gravitino.server.web.rest.ExceptionHandlers$MetalakeExceptionHandler.handle(ExceptionHandlers.java:340)] - Failed to operate metalake(s) [test] operation [LOAD], reason [Cannot invoke "java.util.Map.size()" because "m" is null]
java.lang.NullPointerException: Cannot invoke "java.util.Map.size()" because "m" is null
        at java.base/java.util.HashMap.putMapEntries(HashMap.java:495) ~[?:?]
        at java.base/java.util.HashMap.<init>(HashMap.java:484) ~[?:?]
        at com.google.common.collect.Maps.newHashMap(Maps.java:263) ~[guava-32.1.3-jre.jar:?]
        at org.apache.gravitino.metalake.MetalakeNormalizeDispatcher.newMetalakeWithResolvedProperties(MetalakeNormalizeDispatcher.java:136) ~[gravitino-core-0.7.0-incubating-SNAPSHOT.jar:?]
        at org.apache.gravitino.metalake.MetalakeNormalizeDispatcher.loadMetalake(MetalakeNormalizeDispatcher.java:69) ~[gravitino-core-0.7.0-incubating-SNAPSHOT.jar:?]
        at org.apache.gravitino.listener.MetalakeEventDispatcher.loadMetalake(MetalakeEventDispatcher.java:81) ~[gravitino-core-0.7.0-incubating-SNAPSHOT.jar:?]
        at org.apache.gravitino.server.web.rest.MetalakeOperations.lambda$loadMetalake$4(MetalakeOperations.java:146) ~[gravitino-server-0.7.0-incubating-SNAPSHOT.jar:?]
        at org.apache.gravitino.lock.TreeLockUtils.doWithTreeLock(TreeLockUtils.java:49) ~[gravitino-core-0.7.0-incubating-SNAPSHOT.jar:?]
        at org.apache.gravitino.server.web.rest.MetalakeOperations.lambda$loadMetalake$5(MetalakeOperations.java:145) ~[gravitino-server-0.7.0-incubating-SNAPSHOT.jar:?]
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
        at java.base/javax.security.auth.Subject.doAs(Subject.java:439) ~[?:?]
        at org.apache.gravitino.utils.PrincipalUtils.doAs(PrincipalUtils.java:39) ~[gravitino-core-0.7.0-incubating-SNAPSHOT.jar:?]
        at org.apache.gravitino.server.web.Utils.doAs(Utils.java:188) ~[gravitino-server-0.7.0-incubating-SNAPSHOT.jar:?]
        at org.apache.gravitino.server.web.rest.MetalakeOperations.loadMetalake(MetalakeOperations.java:140) ~[gravitino-server-0.7.0-incubating-SNAPSHOT.jar:?]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[jersey-common-2.41.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[jersey-common-2.41.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.41.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.41.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[jersey-common-2.41.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[jersey-common-2.41.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[jersey-container-servlet-core-2.41.jar:?]
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[jersey-container-servlet-core-2.41.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358) ~[jersey-container-servlet-core-2.41.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311) ~[jersey-container-servlet-core-2.41.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[jersey-container-servlet-core-2.41.jar:?]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.apache.gravitino.server.authentication.AuthenticationFilter.doFilter(AuthenticationFilter.java:86) ~[gravitino-server-common-0.7.0-incubating-SNAPSHOT.jar:?]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.apache.gravitino.server.web.VersioningFilter.doFilter(VersioningFilter.java:111) ~[gravitino-server-0.7.0-incubating-SNAPSHOT.jar:?]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) ~[jetty-security-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]

How to reproduce

  1. Build latest gravitino image, launch gravitino trino test environement
  2. Run trino test by command line

It can be reproduced by github action

Additional context

No response

danhuawang commented 4 weeks ago

@mchades Can you help check this issue?

mchades commented 4 weeks ago

will be fixed after #5241 merged