[BUG] Spark 3.3.1 UI is broken when loading in vertica spark connector 3.3.4 through spark.driver.extraClassPath and spark.executor.extraClassPath

[KevinAppelBofa]

Environment

• Spark version: 3.3.1
• Hadoop version: 3.3.4 / 3.2.2
• Vertica version:
• Vertica Spark Connector version: 3.3.4
• Java version: 1.8
• Additional Environment Information:

---

Problem Description

• Describe the issue in as much details as possible, so it is possible to reproduce it. We are using the spark configs spark.executor.extraClassPath and spark.driver.extraClassPath to load in the various jdbc jars and other jars we need into the spark, when having Spark 3.3.1 and the spark-vertica-connector-all-3.3.4.jar, this is throwing an error in the Spark UI when trying to click on executors tab. api: unavailable java.lang.IllegalArgumentException at jersey.repackaged.org.objectweb.asm.ClassVisitor.(ClassVisitor.java:79) ... Removing this jar, the Spark ui is functional; it appears there is glassfish code in that jar file that is taking precedent over the similar code within the spark 3.3.1

1. Steps to reproduce: Include spark-vertica-connector-all-3.3.4.jar as part of spark.executor.extraClassPath and spark.driver.extraClassPath, start spark job and and open the ui, click on executors tab and it is blank and generates stacktrace
2. Expected behaviour: the executors tab to provide the information
3. Actual behaviour: blank page and stack trace
4. Error message/stack trace:
5. 22/11/28 21:20:13 WARN api: unavailable java.lang.IllegalArgumentException at jersey.repackaged.org.objectweb.asm.ClassVisitor.(ClassVisitor.java:79) at jersey.repackaged.org.objectweb.asm.ClassVisitor.(ClassVisitor.java:64) at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener$AnnotatedClassVisitor.(AnnotationAcceptingListener.java:172) at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener$AnnotatedClassVisitor.(AnnotationAcceptingListener.java:156) at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener.(AnnotationAcceptingListener.java:124) at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener.newJaxrsResourceAndProviderListener(AnnotationAcceptingListener.java:96) at org.glassfish.jersey.server.ResourceConfig.scanClasses(ResourceConfig.java:902) at org.glassfish.jersey.server.ResourceConfig._getClasses(ResourceConfig.java:863) at org.glassfish.jersey.server.ResourceConfig.getClasses(ResourceConfig.java:772) at org.glassfish.jersey.server.ResourceConfig$RuntimeConfig.(ResourceConfig.java:1213) at org.glassfish.jersey.server.ResourceConfig$RuntimeConfig.(ResourceConfig.java:1186) at org.glassfish.jersey.server.ResourceConfig.createRuntimeConfig(ResourceConfig.java:1182) at org.glassfish.jersey.server.ApplicationHandler$RuntimeConfigConfigurator.init(ApplicationHandler.java:182) at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$0(ApplicationHandler.java:290) at java.util.Arrays$ArrayList.forEach(Arrays.java:3880) at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:290) at org.glassfish.jersey.server.ApplicationHandler.(ApplicationHandler.java:259) at org.glassfish.jersey.servlet.WebComponent.(WebComponent.java:311) at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:154) at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:347) at javax.servlet.GenericServlet.init(GenericServlet.java:180) at org.sparkproject.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:632) at org.sparkproject.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:486) at org.sparkproject.jetty.servlet.ServletHolder.prepare(ServletHolder.java:759) at org.sparkproject.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549) at org.sparkproject.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.sparkproject.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.sparkproject.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) at org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.sparkproject.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.sparkproject.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.sparkproject.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) at org.sparkproject.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234) at org.sparkproject.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.sparkproject.jetty.server.Server.handle(Server.java:516) at org.sparkproject.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.sparkproject.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.sparkproject.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.sparkproject.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.sparkproject.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.sparkproject.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.sparkproject.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.sparkproject.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) at org.sparkproject.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.sparkproject.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.lang.Thread.run(Thread.java:750) 22/11/28 21:20:13 WARN HttpChannel: /api/v1/applications/application_1666540628437_1325/allexecutors javax.servlet.ServletException: org.glassfish.jersey.servlet.ServletContainer-3a1ec476==org.glassfish.jersey.servlet.ServletContainer@ef6d77c0{jsp=null,order=-1,inst=true,async=true,src=EMBEDDED:null,STARTED} at org.sparkproject.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:650) at org.sparkproject.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:486) at org.sparkproject.jetty.servlet.ServletHolder.prepare(ServletHolder.java:759) at org.sparkproject.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549) at org.sparkproject.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.sparkproject.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.sparkproject.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) at org.sparkproject.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.sparkproject.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.sparkproject.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.sparkproject.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) at org.sparkproject.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234) at org.sparkproject.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.sparkproject.jetty.server.Server.handle(Server.java:516) at org.sparkproject.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.sparkproject.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.sparkproject.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.sparkproject.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.sparkproject.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.sparkproject.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.sparkproject.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.sparkproject.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.sparkproject.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) at org.sparkproject.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.sparkproject.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.lang.Thread.run(Thread.java:750) Caused by: java.lang.IllegalArgumentException at jersey.repackaged.org.objectweb.asm.ClassVisitor.(ClassVisitor.java:79) at jersey.repackaged.org.objectweb.asm.ClassVisitor.(ClassVisitor.java:64) at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener$AnnotatedClassVisitor.(AnnotationAcceptingListener.java:172) at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener$AnnotatedClassVisitor.(AnnotationAcceptingListener.java:156) at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener.(AnnotationAcceptingListener.java:124) at org.glassfish.jersey.server.internal.scanning.AnnotationAcceptingListener.newJaxrsResourceAndProviderListener(AnnotationAcceptingListener.java:96) at org.glassfish.jersey.server.ResourceConfig.scanClasses(ResourceConfig.java:902) at org.glassfish.jersey.server.ResourceConfig._getClasses(ResourceConfig.java:863) at org.glassfish.jersey.server.ResourceConfig.getClasses(ResourceConfig.java:772) at org.glassfish.jersey.server.ResourceConfig$RuntimeConfig.(ResourceConfig.java:1213) at org.glassfish.jersey.server.ResourceConfig$RuntimeConfig.(ResourceConfig.java:1186) at org.glassfish.jersey.server.ResourceConfig.createRuntimeConfig(ResourceConfig.java:1182) at org.glassfish.jersey.server.ApplicationHandler$RuntimeConfigConfigurator.init(ApplicationHandler.java:182) at org.glassfish.jersey.server.ApplicationHandler.lambda$initialize$0(ApplicationHandler.java:290) at java.util.Arrays$ArrayList.forEach(Arrays.java:3880) at org.glassfish.jersey.server.ApplicationHandler.initialize(ApplicationHandler.java:290) at org.glassfish.jersey.server.ApplicationHandler.(ApplicationHandler.java:259) at org.glassfish.jersey.servlet.WebComponent.(WebComponent.java:311) at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:154) at org.glassfish.jersey.servlet.ServletContainer.init(ServletContainer.java:347) at javax.servlet.GenericServlet.init(GenericServlet.java:180) at org.sparkproject.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:632) ... 29 more
6. Code sample or example on how to reproduce the issue:

---

Spark Connector Logs

• Add related logs entries here.

[ai-bq]

Hi @KevinAppelBofa, thanks for bringing this up. We'll take a look at it.

[jeremyprime]

@KevinAppelBofa, we made changes to our shaded JAR to ensure there is only one version of Jersey (excluded the older version), and in testing the executors tab no longer throws an exception.

This fix is part of v3.3.5, which has now been released.