The MetricsReporter implementation lets users to collect table activities during writes to make better decisions on how to optimize each table differently.
I followed the instruction for serverles EMR and I generated the jar using the latest source code. But for some reason, I keep getting this error when I run the spark job in serverless EMR
File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 1440, in sql File "/usr/lib/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1323, in __call__ File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 175, in deco pyspark.errors.exceptions.captured.IllegalArgumentException: Cannot initialize MetricsReporter, missing no-arg constructor: org.apache.iceberg.aws.manage.AwsTableManagementMetricsEvaluator
This is being run on serverless EMR 6.14 and below is the updated versioins in pom.xml
`
88UTF-81.3.131.1-jre2.7.33.4.02.21.173.42.12
`
Update: This seems to work on EMR-6.11 but not on higher version
I followed the instruction for serverles EMR and I generated the jar using the latest source code. But for some reason, I keep getting this error when I run the spark job in serverless EMR
File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 1440, in sql File "/usr/lib/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1323, in __call__ File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/errors/exceptions/captured.py", line 175, in deco pyspark.errors.exceptions.captured.IllegalArgumentException: Cannot initialize MetricsReporter, missing no-arg constructor: org.apache.iceberg.aws.manage.AwsTableManagementMetricsEvaluator
This is being run on serverless EMR 6.14 and below is the updated versioins in pom.xml `