Spark-UI docker container startup issue

[RajaShyam]

Hi,

I followed the steps from the readme.md & unable to start spark UI on local, below is the error trace -

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
22/02/22 16:14:17 INFO HistoryServer: Started daemon with process name: 1@d4be68f80dce
22/02/22 16:14:17 INFO SignalUtils: Registering signal handler for TERM
22/02/22 16:14:17 INFO SignalUtils: Registering signal handler for HUP
22/02/22 16:14:17 INFO SignalUtils: Registering signal handler for INT
22/02/22 16:14:17 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
22/02/22 16:14:17 INFO SecurityManager: Changing view acls to: root
22/02/22 16:14:17 INFO SecurityManager: Changing modify acls to: root
22/02/22 16:14:17 INFO SecurityManager: Changing view acls groups to: 
22/02/22 16:14:17 INFO SecurityManager: Changing modify acls groups to: 
22/02/22 16:14:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(root); groups with view permissions: Set(); users  with modify permissions: Set(root); groups with modify permissions: Set()
22/02/22 16:14:17 INFO FsHistoryProvider: History server ui acls disabled; users with admin permissions: ; groups with admin permissions: 
22/02/22 16:14:17 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
22/02/22 16:14:17 INFO MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
22/02/22 16:14:17 INFO MetricsSystemImpl: s3a-file-system metrics system started
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:300)
at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: java.lang.NoSuchFieldError: SERVICE_ID
at com.amazonaws.services.s3.AmazonS3Client.createRequest(AmazonS3Client.java:4772)
at com.amazonaws.services.s3.AmazonS3Client.createRequest(AmazonS3Client.java:4758)
at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1434)
at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1374)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$verifyBucketExists$1(S3AFileSystem.java:381)

[SercanKaraoglu]

please check the pull request that resolves the issue

[RajaShyam]

Thanks @SercanKaraoglu I have verified and its working now.

[zdf1230]

I think this is better as 903 is the newer version. https://github.com/aws-samples/aws-glue-samples/pull/116

[moomindani]

It should have been solved by AWS SDK 1.11.375. We did not use 1.11.903 due to its dependency. Resolving. Please feel free to reopen this if you still see the issue.