search

sparkutils / quality

A Quality Spark DQ Library
https://sparkutils.github.io/quality/
Apache License 2.0
4 stars 2 forks source link

SparkPlugin - create a spark plugin to auto-register sql functions #7

Closed chris-twiner closed 1 year ago

chris-twiner commented 1 year ago

Currently functions are only registered during user written code. Many of the functions (with exception of bloom and map creation) can be auto registered and used without other knowledge of Quality internals. This could be true of using auto defined user lambdas in a later release.

The plugin should be defaulted to a call to registerQualityFunctions on the driver plugin side, with the parameters as non final defs to allow simple overriding at the root plugin level.

As the driver plugin framework is only from 3 onwards it's a good opportunity to stub out the 2.4 impl but deprecate 2.4 at the same time for some future removal.

chris-twiner commented 1 year ago

SparkPlugins can't work, needs SparkSessionExtensions instead

chris-twiner commented 1 year ago

tested on 12.2 dbr