search

spark-redshift-community / spark-redshift

Performant Redshift data source for Apache Spark
Apache License 2.0
135 stars 62 forks source link

Caused by: java.lang.NoSuchMethodError: 'java.lang.String org.apache.spark.sql.execution.datasources.PartitionedFile.filePath()' #129

Closed jvovk closed 1 year ago

jvovk commented 1 year ago

Writing to s3 from redshift and this error occurs.

Versions of libraries: io.github.spark-redshift-community:spark-redshift_2.12:5.1.0 com.amazon.redshift:redshift-jdbc42:2.1.0.17 org.apache.spark:spark-avro_2.12:3.4.0 org.apache.hadoop:hadoop-aws:3.3.4 org.apache.hadoop:hadoop-common:3.3.4 com.amazonaws:aws-java-sdk-bundle:1.12.262 running on spark 3.4

jsleight commented 1 year ago

spark-redshift_2.12:5.1.0 only supports up to Spark v3.2 (might also work with 3.3).

However, we just released spark-redshift_2.12:6.0.0-spark3.4 yesterday which should be compatible. In addition to Spark 3.4 support, v6.0.0 also adds a ton of new features like caching, predicate pushdown, IAM authentication, etc. Checkout the Changelog for all the details https://github.com/spark-redshift-community/spark-redshift/blob/master/CHANGELOG#L19