AbsaOSS / spline-spark-agent

Spline agent for Apache Spark
https://absaoss.github.io/spline/
Apache License 2.0
185 stars 95 forks source link

Fix MergeIntoNodeBuilder #844

Open AndreaCes opened 2 days ago

AndreaCes commented 2 days ago

This pull request addresses a bug in the MergeIntoNodeBuilder, where the functionalExpressions were overridden with an empty sequence.

Based on the merge semantic, it is correct to have an arbitrary expression in the UPDATE clause. For instance, let's consider a scenario where we have a source table with the schema id long, name string and a target delta table with schema id long, text string. In this case, a valid merge operation could be:

MERGE INTO target
USING source 
ON target.id = source.id
WHEN MATCHED THEN UPDATE SET text = lower(source.name)

If functionalExpressions is empty, the output attribute text will have a child expression id resulting from the conversion of lower(source.name), which is not referenced in the expression.functions output lineage section.

sonarcloud[bot] commented 2 days ago

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud