search

apache / gravitino

World's most powerful open data catalog for building a high-performance, geo-distributed and federated metadata lake.
https://gravitino.apache.org
Apache License 2.0
1.03k stars 320 forks source link

[Bug report] Start the Gravitino Server failed in gravitino container due to java.lang.IllegalArgumentException: null (Daily pipeline 20240822) #4659

Closed danhuawang closed 5 days ago

danhuawang commented 2 months ago

Version

main branch

Describe what's wrong

Start the Gravitino Server

Log dir doesn't exist, create /root/gravitino/logs
Gravitino Server is not running

Failed to start iceberg-rest web server

Error message and/or stacktrace

2024-08-23 10:03:44.506 INFO [main] [org.apache.gravitino.iceberg.RESTService.serviceInit(RESTService.java:95)] - Iceberg REST service init.
2024-08-23 10:03:44.508 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:114)] - Start to check the dead lock...
2024-08-23 10:03:44.508 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:116)] - Finish to check the dead lock...
2024-08-23 10:03:44.508 INFO [main] [org.apache.gravitino.GravitinoEnv.initialize(GravitinoEnv.java:147)] - Gravitino Environment is initialized.
2024-08-23 10:03:44.515 INFO [main] [org.apache.gravitino.server.web.JettyServer.initializeWebAppServletContextHandler(JettyServer.java:309)] - Gravitino Webapp path: /tmp/GravitinoWar6263486368816173318
2024-08-23 10:03:44.518 INFO [main] [org.apache.gravitino.metrics.MetricsSystem.register(MetricsSystem.java:78)] - Register gravitino-server to metrics system 
2024-08-23 10:03:44.521 INFO [main] [org.eclipse.jetty.server.Server.doStart(Server.java:375)] - jetty-9.4.51.v20230217; built: 2023-02-17T08:19:37.309Z; git: b45c405e4544384de066f814ed42ae3dceacdd49; jvm 17.0.2+8-86
2024-08-23 10:03:44.594 WARN [main] [org.eclipse.jetty.server.handler.ContextHandler$Context.log(ContextHandler.java:2374)] - unavailable
java.lang.IllegalArgumentException: null
        at jersey.repackaged.org.objectweb.asm.ClassVisitor.<init>(ClassVisitor.java:79) ~[jersey-bundle-1.19.3.jar:1.19.3]
        at jersey.repackaged.org.objectweb.asm.ClassVisitor.<init>(ClassVisitor.java:64) ~[jersey-bundle-1.19.3.jar:1.19.3]
        at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener$AnnotatedClassVisitor.<init>(AnnotationAcceptingListener.java:172) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener$AnnotatedClassVisitor.<init>(AnnotationAcceptingListener.java:156) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener.<init>(AnnotationAcceptingListener.java:124) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener.newJaxrsResourceAndProviderListener(AnnotationAcceptingListener.java:96) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.ResourceConfig.scanClasses(ResourceConfig.java:903) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.ResourceConfig._getClasses(ResourceConfig.java:864) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.ResourceConfig.getClasses(ResourceConfig.java:773) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.ResourceConfig$RuntimeConfig.<init>(ResourceConfig.java:1216) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.ResourceConfig$RuntimeConfig.<init>(ResourceConfig.java:1187) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.ResourceConfig.createRuntimeConfig(ResourceConfig.java:1183) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler$RuntimeConfigConfigurator.init(ApplicationHandler.java:184) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$0(ApplicationHandler.java:294) ~[jersey-server-2.41.jar:?]
        at java.base/java.util.Arrays$ArrayList.forEach(Arrays.java:4204) ~[?:?]
        at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:294) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.<init>(ApplicationHandler.java:261) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.servlet.WebComponent.<init>(WebComponent.java:311) ~[jersey-container-servlet-core-2.41.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:154) ~[jersey-container-servlet-core-2.41.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:339) ~[jersey-container-servlet-core-2.41.jar:?]
        at javax.servlet.GenericServlet.init(GenericServlet.java:244) ~[javax.servlet-api-3.1.0.jar:3.1.0]
        at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:632) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:415) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:750) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357) ~[?:?]
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:510) ~[?:?]
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:310) ~[?:?]
        at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735) ~[?:?]
        at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) ~[?:?]
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:774) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) ~[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) ~[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.Server.start(Server.java:423) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) ~[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.Server.doStart(Server.java:387) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.apache.gravitino.server.web.JettyServer.start(JettyServer.java:183) ~[gravitino-server-common-0.7.0-incubating-SNAPSHOT.jar:?]
        at org.apache.gravitino.iceberg.RESTService.serviceStart(RESTService.java:103) ~[gravitino-iceberg-rest-server-0.7.0-incubating-SNAPSHOT.jar:?]
        at org.apache.gravitino.auxiliary.AuxiliaryServiceManager.lambda$serviceStart$7(AuxiliaryServiceManager.java:201) ~[gravitino-core-0.7.0-incubating-SNAPSHOT.jar:?]
        at org.apache.gravitino.auxiliary.AuxiliaryServiceManager.lambda$doWithClassLoader$4(AuxiliaryServiceManager.java:172) ~[gravitino-core-0.7.0-incubating-SNAPSHOT.jar:?]
        at org.apache.gravitino.utils.IsolatedClassLoader.withClassLoader(IsolatedClassLoader.java:86) ~[gravitino-core-0.7.0-incubating-SNAPSHOT.jar:?]
        at org.apache.gravitino.auxiliary.AuxiliaryServiceManager.doWithClassLoader(AuxiliaryServiceManager.java:169) ~[gravitino-core-0.7.0-incubating-SNAPSHOT.jar:?]
        at org.apache.gravitino.auxiliary.AuxiliaryServiceManager.lambda$serviceStart$8(AuxiliaryServiceManager.java:201) ~[gravitino-core-0.7.0-incubating-SNAPSHOT.jar:?]
        at java.base/java.util.HashMap.forEach(HashMap.java:1421) [?:?]
        at org.apache.gravitino.auxiliary.AuxiliaryServiceManager.serviceStart(AuxiliaryServiceManager.java:199) [gravitino-core-0.7.0-incubating-SNAPSHOT.jar:?]
        at org.apache.gravitino.GravitinoEnv.start(GravitinoEnv.java:311) [gravitino-core-0.7.0-incubating-SNAPSHOT.jar:?]
        at org.apache.gravitino.server.GravitinoServer.start(GravitinoServer.java:145) [gravitino-server-0.7.0-incubating-SNAPSHOT.jar:?]
        at org.apache.gravitino.server.GravitinoServer.main(GravitinoServer.java:167) [gravitino-server-0.7.0-incubating-SNAPSHOT.jar:?]
2024-08-23 10:03:44.604 ERROR [main] [org.apache.gravitino.server.web.JettyServer.start(JettyServer.java:194)] - Failed to start iceberg-rest web server.
javax.servlet.ServletException: org.glassfish.jersey.servlet.ServletContainer-44550792==org.glassfish.jersey.servlet.ServletContainer@eee110e{jsp=null,order=-1,inst=true,async=true,src=EMBEDDED:null,STARTED}
        at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:650) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:415) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:750) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357) ~[?:?]
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:510) ~[?:?]
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.forEachRemaining(StreamSpliterators.java:310) ~[?:?]
        at java.base/java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:735) ~[?:?]
        at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762) ~[?:?]
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:774) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) ~[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73) ~[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) ~[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.Server.start(Server.java:423) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) ~[jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230

How to reproduce

It can be reproduced by daily github action: https://github.com/datastrato/gravitino-test/actions/runs/10514056573/job/29131685115

or you can docker run gravitino with this image: datastratosandbox/gravitino:0.7.0-incubating-SNAPSHOT-147

Additional context

No response

danhuawang commented 2 months ago

@FANNG1 Would you like to help check this issue ?

FANNG1 commented 2 months ago

ranger plugin introduces some jersery jars, maybe fixed by https://github.com/apache/gravitino/pull/4651

FANNG1 commented 1 week ago

@danhuawang could you verify this?