[DSIP-79][Task] Add Datavines task to better support data quality

xxzuo commented 5 months ago

DataVines is an easy-to-use data quality service platform that supports multiple metric. https://github.com/datavane/datavines

Datavines supports executing multiple metrics in one job.
Datavines supports execution status dashboard and data quality report.
Datavines supports plug-in extensions for components such as metric, data sources, error data storage, and execution engines.
Jdbc engines can be used to execute data quality tasks instead of solely relying on Spark engines.

config data quality job in datavines
get the jobId
Add datavines job node in workflow, and configure the datavines api address and jobId

No response

No response

MYiYang commented 5 months ago

It would be nice if you could submit a task here and see the status of the task in ds and stop it via datavines

zhangp8721 commented 5 months ago

very useful for data pipeLine

xiaoshiqiai commented 5 months ago

If the datavines are incorporated into the ds, it will be easier to integrate project management and data inspection

zixi0825 commented 5 months ago

+1

ruanwenjun commented 4 months ago

You should provide a detail design related of the how to use the new task and how does the task work in ds, rather than some pictures of ui.

xxzuo commented 4 months ago

You should provide a detail design related of the how to use the new task and how does the task work in ds, rather than some pictures of ui.

ok, I will supplement the detail design.

SbloodyS commented 2 weeks ago

ok, I will supplement the detail design.

Hi, are you still working on this?

zixi0825 commented 2 weeks ago

ok, I will supplement the detail design.

Hi, are you still working on this?

I will come to do this.

zixi0825 commented 2 weeks ago

Before the new task plugin is completed, shell tasks can be used to integrate datavines, refer to the following guidelines https://datavane.github.io/datavines-website/docs/integration/dolphin-scheduler

apache / dolphinscheduler