search

sparkutils / quality

A Quality Spark DQ Library
https://sparkutils.github.io/quality/
Apache License 2.0
4 stars 2 forks source link

3.5 with delta - NakedLocalFileSystem$NakedLocalFileStatus cannot be cast to FileStatusWithMetadata #27

Closed chris-twiner closed 9 months ago

chris-twiner commented 1 year ago

running the ExtensionsDeltaTest with 3.5 snap causes this to be thrown from many of the tests that do not use onlyWithExistingSession / wrapWithExistingSession (i.e. those to run on Databricks / notebooks).

java.lang.ClassCastException: com.globalmentor.apache.hadoop.fs.NakedLocalFileSystem$NakedLocalFileStatus cannot be cast to org.apache.spark.sql.execution.datasources.FileStatusWithMetadata

same delta version on 3.4 per latter #25 builds have no issue. (some of them also failed with the npe issue 629)

chris-twiner commented 1 year ago

as that's new for 3.5 an expected new delta would be needed https://issues.apache.org/jira/browse/SPARK-43039

chris-twiner commented 11 months ago

also

java.lang.NoSuchMethodError: org.apache.spark.sql.types.StructType.toAttributes()Lscala/collection/Seq;

    at org.apache.spark.sql.delta.files.TransactionalWrite.getStatsSchema(TransactionalWrite.scala:262)
    at org.apache.spark.sql.delta.files.TransactionalWrite.getStatsSchema$(TransactionalWrite.scala:252)
    at org.apache.spark.sql.delta.OptimisticTransaction.getStatsSchema(OptimisticTransaction.scala:137)
    at org.apache.spark.sql.delta.files.TransactionalWrite.getOptionalStatsTrackerAndStatsCollection(TransactionalWrite.scala:294)
    at org.apache.spark.sql.delta.files.TransactionalWrite.getOptionalStatsTrackerAndStatsCollection$(TransactionalWrite.scala:286)
    at org.apache.spark.sql.delta.OptimisticTransaction.getOptionalStatsTrackerAndStatsCollection(OptimisticTransaction.scala:137)
    at org.apache.spark.sql.delta.files.TransactionalWrite.writeFiles(TransactionalWrite.scala:352)