sparkutils / quality

A Quality Spark DQ Library
https://sparkutils.github.io/quality/
Apache License 2.0
4 stars 2 forks source link

Investigate lifting no-arg lambda functions to simple expressions #47

Closed chris-twiner closed 11 months ago

chris-twiner commented 1 year ago

When FunN is parsed without an argument the result is a simple expression, this should be lifted directly and not subject to most of the rest of the higher order logic:

  1. aggregate expressions should be possible to define in user functions
  2. lambda for drop-in's should ideally throw - there is no lambda (not sure on how to verify this)
  3. callfun should still work
  4. lambda for non drop-ins should be possible to just replace

This would allow user functions to just substitute out expression trees like snippets without lambdafunction and aggregatefunction clashes or analysis issues when subbing out plain lambdas for plain expressions.

chris-twiner commented 11 months ago

closing as it's not possible directly, possibly with an optimiser rule to sub it out as only at time of bind do you know it's an attribute reference, when it's no longer substitutable in the builder