delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
7.62k stars 1.71k forks source link

[BUG] MERGE operation fails due to AnalysisException when target table contains CHAR(N) columns #3868

Open clee704 opened 1 week ago

clee704 commented 1 week ago

Bug

Which Delta project/connector is this regarding?

Describe the problem

Steps to reproduce

create table target (id char(10), value char(10)) using delta;
create table source (id char(10), value char(10)) using parquet;
merge into target
using source on target.id = source.id
when matched then update set target.value = source.value;

Observed results

org.apache.spark.sql.delta.DeltaAnalysisException: [DELTA_MERGE_RESOLVED_ATTRIBUTE_MISSING_FROM_INPUT] Resolved attribute(s) id#878,id#880,value#887,id#892,value#893 missing from id#872,value#873,id#874,value#875 in operator !DeltaMergeInto (id#878 = id#880), [Update [actions: [`value` = value#887]]], [Insert [actions: [`id` = id#892, `value` = value#893]]], false, StructType(StructField(id,StringType,true),StructField(value,StringType,true)); line 2 pos 0
at org.apache.spark.sql.delta.ResolveDeltaMergeInto$.resolveReferencesAndSchema(ResolveDeltaMergeInto.scala:335)                                                                                                                                                                          at org.apache.spark.sql.delta.DeltaAnalysis$$anonfun$apply$1.applyOrElse(DeltaAnalysis.scala:547)                                                                                                                                                                                         at org.apache.spark.sql.delta.DeltaAnalysis$$anonfun$apply$1.applyOrElse(DeltaAnalysis.scala:81)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$2(AnalysisHelper.scala:170)
...

Expected results

MERGE should be performed.

Environment information

Willingness to contribute

The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?