search

aws-samples / iceberg-aws-event-based-table-management

The MetricsReporter implementation lets users to collect table activities during writes to make better decisions on how to optimize each table differently.
MIT No Attribution
5 stars 2 forks source link

Cannot initialize MetricsReporter, missing no-arg constructor #1

Closed ram-jayapalan closed 11 months ago

ram-jayapalan commented 11 months ago

I followed the instruction for serverles EMR and I generated the jar using the latest source code. But for some reason, I keep getting this error when I run the spark job in serverless EMR

File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 1440, in sql File "/usr/lib/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1323, in __call__ File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 175, in deco pyspark.errors.exceptions.captured.IllegalArgumentException: Cannot initialize MetricsReporter, missing no-arg constructor: org.apache.iceberg.aws.manage.AwsTableManagementMetricsEvaluator

This is being run on serverless EMR 6.14 and below is the updated versioins in pom.xml `

8 8 UTF-8 1.3.1 31.1-jre 2.7.3 3.4.0 2.21.17 3.4 2.12 ` Update: This seems to work on EMR-6.11 but not on higher version
ram-jayapalan commented 11 months ago

Setting the aws sdk version to 2.20.18 seems to work on EMR 6.14. Here is the updated properties for pom.xml that is compatible with EMR 6.14 `

8 8 UTF-8 1.3.1 31.1-jre 2.7.3 3.4.0 2.20.18 3.4 2.12 `