sparkutils / quality

A Quality Spark DQ Library
https://sparkutils.github.io/quality/
Apache License 2.0
4 stars 2 forks source link

update_field doesn't replace nested structs #36

Closed chris-twiner closed 1 year ago

chris-twiner commented 1 year ago

the original sme doesn't handle multiple arg structs directly, this was provided by the methods add_struct_field impl. As such:

    val og = sparkSession.range(1).selectExpr("named_struct('a', 1, 'b', named_struct('c', 4)) s")
    val updated = og.select(update_field(col("s"), ("b.c", lit(40))) as "s")

would not yield the correct results, instead adding a field called b.c instead of nesting. The function impl from #35 should be called and this logic implemented once.

chris-twiner commented 1 year ago

causes issues when using with resolve and compileEvals = false - added to docs, possibly not worth pursuing a fix but #38 raised for it.