Extend Quinn's Data Validation functionality

mrpowers-io / quinn

pyspark methods to enhance developer productivity 📣 👯 🎉

https://mrpowers-io.github.io/quinn/

Apache License 2.0

642 stars 99 forks source link

Extend Quinn's Data Validation functionality #182

Open souvik-databricks opened 9 months ago

souvik-databricks commented 9 months ago

Add to Quinn a tool to profile incoming data, check if it adheres to registered schema and do custom data quality checks. At the end of it, a human readable report is autogenerated that can be saved separately.

Config driven data profiling and schema validation
Autogeneration of report after every run
Integration with datadog monitoring system
Extensible and highly customizable.
Very little boiler plate code.
Support for versioned schema validation.

souvik-databricks commented 9 months ago

@MrPowers I will be working on this. Can we get this assigned to me please. Thanks.

MrPowers commented 9 months ago

Have at it @souvik-databricks !