An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
create table target (id char(10), value char(10)) using delta;
create table source (id char(10), value char(10)) using parquet;
merge into target
using source on target.id = source.id
when matched then update set target.value = source.value;
Observed results
org.apache.spark.sql.delta.DeltaAnalysisException: [DELTA_MERGE_RESOLVED_ATTRIBUTE_MISSING_FROM_INPUT] Resolved attribute(s) id#878,id#880,value#887,id#892,value#893 missing from id#872,value#873,id#874,value#875 in operator !DeltaMergeInto (id#878 = id#880), [Update [actions: [`value` = value#887]]], [Insert [actions: [`id` = id#892, `value` = value#893]]], false, StructType(StructField(id,StringType,true),StructField(value,StringType,true)); line 2 pos 0
at org.apache.spark.sql.delta.ResolveDeltaMergeInto$.resolveReferencesAndSchema(ResolveDeltaMergeInto.scala:335) at org.apache.spark.sql.delta.DeltaAnalysis$$anonfun$apply$1.applyOrElse(DeltaAnalysis.scala:547) at org.apache.spark.sql.delta.DeltaAnalysis$$anonfun$apply$1.applyOrElse(DeltaAnalysis.scala:81)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$2(AnalysisHelper.scala:170)
...
Expected results
MERGE should be performed.
Environment information
Delta Lake version: latest (860438f97bc209296f6b55ab5311d7f3d2bf6d25)
Spark version: 3.5.3
Scala version: 2.12.18
Willingness to contribute
The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?
[x] Yes. I can contribute a fix for this bug independently.
[ ] Yes. I would be willing to contribute a fix for this bug with guidance from the Delta Lake community.
[ ] No. I cannot contribute a bug fix at this time.
Bug
Which Delta project/connector is this regarding?
Describe the problem
Steps to reproduce
Observed results
Expected results
MERGE should be performed.
Environment information
Willingness to contribute
The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?