Data quality is used to ensure the accuracy of data in the process of integration and processing. It is also the core component of DataOps. DataVines is an easy-to-use data quality service platform that supports multiple metric.
Need: Maven 3.6.1 and later
$ mvn clean package -Prelease -DskipTests
SQL
check SLA
for check result alertThe platform is based on plug-in design, and the following modules support user-defined plug-ins to expand
MySQL
, Impala
, StarRocks
, Doris
, Presto
, Trino
, ClickHouse
, PostgreSQL
are already supportedSpark
and Local
have been supported. The Spark
engine currently only supports the Spark2.4
version, and the Local
engine is a local execution engine developed based on JDBC
, without relying on other execution engines.MySQL
and local files are already supported (only Local
execution engine is supported)MySQL
, PostgreSQL
and ZooKeeper
datavines-submit.sh
, can be used in conjunction with the scheduling systemMySQL
to start the project and complete the check of data quality operationsServer
node supports horizontal expansion to improve performanceClick Document for more information
Click Document for more information
You can submit any ideas as pull requests or as GitHub issues.
If you're new to posting issues, we ask that you read How To Ask Questions The Smart Way (This guide does not provide actual support services for this project!), How to Report Bugs Effectively prior to posting. Well written bug reports help us help you!
Thank you to all the people who already contributed to Datavines!
Datavines is licensed under the Apache License 2.0. Datavines relies on some third-party components, and their open source protocols are also Apache License 2.0 or compatible with Apache License 2.0. In addition, Datavines also directly references or modifies some codes in Apache DolphinScheduler, SeaTunnel and Dubbo, all of which are Apache License 2.0. Thanks for contributions to these projects.