Nike-Inc / spark-expectations

A Python Library to support running data quality rules while the spark job is running⚡
https://engineering.nike.com/spark-expectations
Apache License 2.0
161 stars 37 forks source link

Introduce an option for quarantine data #75

Closed jskrajareddy21 closed 6 months ago

jskrajareddy21 commented 6 months ago

Describe the solution you'd like having the option to choose whether to quarantine data into an error table is beneficial. There are instances where we may not need to quarantine data, saving time during spark expectation execution.