Closed matthewpick closed 4 years ago
This is an inconsistent error that means there is probably something non-deterministic going on here. The analysis resolution stuff is deterministic, so it's likely that its something to do with the DataFrame generation. Can you update your code to catch this error and print the schema of the DeltaTable object and the newData DataFrame?
In addition, there has actually been significant changes in the analysis part of merge in 0.7.0 because now Spark 3.0 catalyst directly understands Merge logical plan. So would be good to try it out.
@tdas Thanks for the recommendations / insight.
I've been working on upgrading to EMR 6.0.0 and Delta 0.7.0 today and hopefully that helps (I need to upgrade to Spark 3.x at this point anyway). I'll keep this issue up-to-date with my findings.
@tdas your recommendation to compare datalake / new dataframe schemas worked great 👍
My delta writer logic (built for streaming purposes) has to choose between writing a brand new dataset or using a merge statement.
A few of my delta tables (I have a 150 tables or so) were in a weird state where DeltaTable.isDeltaTable()
returned true even though there wasn't any data in the table (and the schema was completely undefined).
By combining the logic of "isDeltaTable" and "isTableEmpty" I was able to properly create/update my delta tables and account for this odd edge case where isDeltaTable
returned true but the table did not have any data or a schema defined.
Here is my new delta writer code: https://gist.github.com/matthewpick/d7ca9504179c84df945d37a8c34ed107#file-deltawriter-scala-L39-L42
Glad to hear!
Environment
AWS EMR 5.28.1 Delta 0.6.1
Problem
Not sure why I keep getting this AnalysisException... the data in S3 and the new spark dataframe have the same schema (even the error message contains the correct column names)
The error is inconsistent but typically shows up after 5-10 minutes of processing the Kinesis stream data.
Logs
EMR Logs
Delta merge statement
DeltaWriter.scala
Delta.io Code references
https://github.com/delta-io/delta/blob/v0.6.1/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/deltaMerge.scala#L253-L262
Thoughts
Maybe this could be caused by case sensitivity in the primaryKey?
deltaMerge.scala
hasn't changed very much between 0.6.1 to 0.7.0 so I am not confident that upgrading to 0.7.0 will fix this problem (maybe the spark 3.0 catalyst code is more robust?).