search

awslabs / aws-glue-libs

AWS Glue Libraries are additions and enhancements to Spark for ETL operations.
Other
642 stars 304 forks source link

Local glue fails due to collision on javax.servlet-3.0.0.v201112011016.jar #47

Open dheffx opened 4 years ago

dheffx commented 4 years ago

We are using aws-glue-local to allow us to run tests in CodeBuild as a part of our CICD pipeline. After an unrelated change was pushed yesterday, we found the job failing due to this error:

20/03/18 13:49:10 ERROR SparkContext: Error initializing SparkContext.
--
205 | java.lang.SecurityException: class "javax.servlet.FilterRegistration"'s signer information does not match signer information of other classes in the same package
206 | at java.lang.ClassLoader.checkCerts(ClassLoader.java:898)
207 | at java.lang.ClassLoader.preDefineClass(ClassLoader.java:668)
208 | at java.lang.ClassLoader.defineClass(ClassLoader.java:761)
209 | at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
210 | at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
211 | at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
212 | at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
213 | at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
214 | at java.security.AccessController.doPrivileged(Native Method)
215 | at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
216 | at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
217 | at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
218 | at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
219 | at org.spark_project.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:154)
220 | at org.spark_project.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:146)
221 | at org.spark_project.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:140)
222 | at org.spark_project.jetty.servlet.ServletContextHandler.<init>(ServletContextHandler.java:110)
223 | at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:143)
224 | at org.apache.spark.ui.JettyUtils$.createServletHandler(JettyUtils.scala:130)
225 | at org.apache.spark.ui.WebUI.attachPage(WebUI.scala:83)
226 | at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:65)
227 | at org.apache.spark.ui.WebUI$$anonfun$attachTab$1.apply(WebUI.scala:65)
228 | at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
229 | at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
230 | at org.apache.spark.ui.WebUI.attachTab(WebUI.scala:65)
231 | at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:62)
232 | at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:80)
233 | at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:175)
234 | at org.apache.spark.SparkContext.<init>(SparkContext.scala:444)
235 | at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
236 | at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
237 | at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
238 | at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
239 | at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
240 | at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
241 | at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
242 | at py4j.Gateway.invoke(Gateway.java:238)
243 | at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
244 | at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
245 | at py4j.GatewayConnection.run(GatewayConnection.java:238)
246 | at java.lang.Thread.run(Thread.java:748)

After some digging online around this error, and looking at the jar files downloaded, I was able to resolve this issue by adding a command to our buildspec that removes the jar file javax.servlet-3.0.0.v201112011016.jar from the jarsv1 dir.

This had not happened on previous runs, and we did not see any changes in the glue repo or the pom file for GlueETL. This also did not happen when we used CodeBuild local, or when we ran glue lib locally on our laptops.

abaibai921 commented 4 years ago

Thanks for providing your solution. Ran into the same situation and did what you said, problem solved.