Open orf opened 4 years ago
I'm having the same issue. I guess is something with lake formation and the job's permission.
Thanks!
I'm having the same issue. Logs:
Caused by: java.io.IOException: Failed to connect to /172.37.210.78:38871 at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:245)
2021-02-16 22:58:22,531 ERROR [main] glue.ProcessLauncher (Logging.scala:logError(70)): Exception in User Class: java.lang.reflect.UndeclaredThrowableExceptionorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1862)org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:64)org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:188)org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:281)org.apache.spark.executor.CoarseGrainedExecutorBackendPlugin$class.launch(CoarseGrainedExecutorBackendWrapper.scala:10)org.apache.spark.executor.CoarseGrainedExecutorBackendWrapper$$anon$1.launch(CoarseGrainedExecutorBackendWrapper.scala:15)org.apache.spark.executor.CoarseGrainedExecutorBackendWrapper.launch(CoarseGrainedExecutorBackendWrapper.scala:19)org.apache.spark.executor.CoarseGrainedExecutorBackendWrapper$.main(CoarseGrainedExecutorBackendWrapper.scala:5)org.apache.spark.executor.CoarseGrainedExecutorBackendWrapper.main(CoarseGrainedExecutorBackendWrapper.scala)sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)java.lang.reflect.Method.invoke(Method.java:498)com.amazonaws.services.glue.SparkProcessLauncherPlugin$class.invoke(ProcessLauncher.scala:38)com.amazonaws.services.glue.ProcessLauncher$$anon$1.invoke(ProcessLauncher.scala:67)com.amazonaws.services.glue.ProcessLauncher.launch(ProcessLauncher.scala:108)com.amazonaws.services.glue.ProcessLauncher$.main(ProcessLauncher.scala:21)com.amazonaws.services.glue.ProcessLauncher.main(ProcessLauncher.scala) | 2021-02-16 22:58:22,531 ERROR [main] glue.ProcessLauncher (Logging.scala:logError(70)): Exception in User Class: java.lang.reflect.UndeclaredThrowableException org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1862) org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:64) org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:188) org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:281) org.apache.spark.executor.CoarseGrainedExecutorBackendPlugin$class.launch(CoarseGrainedExecutorBackendWrapper.scala:10) org.apache.spark.executor.CoarseGrainedExecutorBackendWrapper$$anon$1.launch(CoarseGrainedExecutorBackendWrapper.scala:15) org.apache.spark.executor.CoarseGrainedExecutorBackendWrapper.launch(CoarseGrainedExecutorBackendWrapper.scala:19) org.apache.spark.executor.CoarseGrainedExecutorBackendWrapper$.main(CoarseGrainedExecutorBackendWrapper.scala:5) org.apache.spark.executor.CoarseGrainedExecutorBackendWrapper.main(CoarseGrainedExecutorBackendWrapper.scala) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) com.amazonaws.services.glue.SparkProcessLauncherPlugin$class.invoke(ProcessLauncher.scala:38) com.amazonaws.services.glue.ProcessLauncher$$anon$1.invoke(ProcessLauncher.scala:67) com.amazonaws.services.glue.ProcessLauncher.launch(ProcessLauncher.scala:108) com.amazonaws.services.glue.ProcessLauncher$.main(ProcessLauncher.scala:21) com.amazonaws.services.glue.ProcessLauncher.main(ProcessLauncher.scala) |
---|
2021-02-16 22:58:22,531 ERROR [main] glue.ProcessLauncher (Logging.scala:logError(70)): Exception in User Class: java.lang.reflect.UndeclaredThrowableExceptionorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1862)org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:64)org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:188)org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:281)org.apache.spark.executor.CoarseGrainedExecutorBackendPlugin$class.launch(CoarseGrainedExecutorBackendWrapper.scala:10)org.apache.spark.executor.CoarseGrainedExecutorBackendWrapper$$anon$1.launch(CoarseGrainedExecutorBackendWrapper.scala:15)org.apache.spark.executor.CoarseGrainedExecutorBackendWrapper.launch(CoarseGrainedExecutorBackendWrapper.scala:19)org.apache.spark.executor.CoarseGrainedExecutorBackendWrapper$.main(CoarseGrainedExecutorBackendWrapper.scala:5)org.apache.spark.executor.CoarseGrainedExecutorBackendWrapper.main(CoarseGrainedExecutorBackendWrapper.scala)sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)java.lang.reflect.Method.invoke(Method.java:498)com.amazonaws.services.glue.SparkProcessLauncherPlugin$class.invoke(ProcessLauncher.scala:38)com.amazonaws.services.glue.ProcessLauncher$$anon$1.invoke(ProcessLauncher.scala:67)com.amazonaws.services.glue.ProcessLauncher.launch(ProcessLauncher.scala:108)com.amazonaws.services.glue.ProcessLauncher$.main(ProcessLauncher.scala:21)com.amazonaws.services.glue.ProcessLauncher.main(ProcessLauncher.scala)
Thanks!
@orf lake permission for a glue database? i'm not sure i understand how is that related... actually, i don't think you can give lake permission per a database, lake permission is related to data lake service.
can you elaborate how did you manage to fix this issue?
Same issue here. Is there a workaround for this?
Hello, I faced the same issue and spent several hours looking for it. If you are sure there is not a permission issue with lake formation or IAM over your glue catalog, then you may have the same issue that I had. When running the job with glue2 for the first time it does fails with the mentioned error, but if you change to glue3 then it works just fine and if you go back to glue 2 and the table already exists it will work again. If you need to use glue2 because you need spark 2.X you can just use glueContext.write_dynamic_frame.from_options to save directly to S3 without updating, create and run a Crawler and then change the job to getSink-UPDATE_IN_DATABASE etc.. and it may work. Almost it did it for me.
I found the same issue using glue studio, I found out it was the csv optimization option in the studio
I found the same issue using glue studio, I found out it was the csv optimization option in the studio
How to fix it?
I found the same issue using glue studio, I found out it was the csv optimization option in the studio
can you please explain how to fix it?
I found the same issue using glue studio, I found out it was the csv optimization option in the studio
How to fix it?
did u fix it?
I resolved this issue by realizing my glue job and database were not on the same region. Make sure all your resources are in the same region!
It turned out that 'enableUpdateCatalog=True' is the real cause. If you do not have the table already created, the job will fail. Tested with Glue 3 and Glue 4.
Glue 4 at least explicitly printed the reason of the failure: the target table does not exist. And a ton of trash stack traces, of course.
Running the job only to produce files and running a crawler afterward can be an option. This is extremely confusing, as other jobs were running without issues. Such a wasted day. And night.
Running the snippet from the creating new tables documentation will throw a
NullPointerException
if your job role does not have LakeFormation permissions over the database:The following exception is thrown:
This is a bug report and something that should help anyone else facing this issue.