search

Nike-Inc / spark-expectations

A Python Library to support running data quality rules while the spark job is running⚡
https://engineering.nike.com/spark-expectations
Apache License 2.0
161 stars 37 forks source link

[FEATURE] Test Spark-Connect Compatibility #102

Open newfront opened 3 months ago

newfront commented 3 months ago

Is your feature request related to a problem? Please describe. In order to continue to work with the Spark ecosystem, we need to ensure we can continue to support the various connection types - including spark-connect.

Describe the solution you'd like The application will run when using the spark-connect backed SparkSession.

Describe alternatives you've considered Zero. We need this.

Additional context This will be a way to easily adapt to the Delta 4.x and Spark 4.x ecosystem.

stampthecoder commented 1 month ago

@newfront I'm interested in this too... This use case is exactly what my company needs. I like the fact nike is keeping it under their umbrella for maintainability ....godspeed.