apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
https://dolphinscheduler.apache.org/
Apache License 2.0
12.37k stars 4.49k forks source link

[DSIP-][Task] Add Datavines task to better support data quality #16113

Open xxzuo opened 3 weeks ago

xxzuo commented 3 weeks ago

Search before asking

Motivation

DataVines is an easy-to-use data quality service platform that supports multiple metric. https://github.com/datavane/datavines

Design Detail

Sript mode

  1. config data quality job in datavines image

  2. get the job config scipt file

  3. Add datavines job node in workflow, and configure the script image

API Mode

  1. config data quality job in datavines image

  2. get the jobId

  3. Add datavines job node in workflow, and configure the datavines api address and jobId

Compatibility, Deprecation, and Migration Plan

No response

Test Plan

No response

Code of Conduct

MYiYang commented 3 weeks ago

It would be nice if you could submit a task here and see the status of the task in ds and stop it via datavines

zhangp8721 commented 3 weeks ago

very useful for data pipeLine

xiaoshiqiai commented 3 weeks ago

If the datavines are incorporated into the ds, it will be easier to integrate project management and data inspection

zixi0825 commented 2 weeks ago

+1

ruanwenjun commented 1 week ago

You should provide a detail design related of the how to use the new task and how does the task work in ds, rather than some pictures of ui.