Lineage data kinship

[boandai]

if anybody else also need this feature , please reply "+1", thx

finally, we will decide which version to implement this feature based on the level of attention.

[liuminyt]

+1

[liuGangBestSkip]

+1

[thisnew]

+1

[zhanghuidouble]

+1

if anybody else also need this feature , please reply "+1", thx

finally, we will decide which version to implement this feature based on the level of attention.

[wuchch]

+1

[moranrr]

+1

[ITriangle]

+1

[WangyanxuFp]

+1111111111

[coding-now]

+1

[lianayu]

+1

[lipengyu]

+1

[wangsvip]

+1

[simon824]

+1

[yh2388]

+1

[7eng]

+1

[feiyalun]

+1

[wen-hemin]

I plan to implement data lineage function (Table level).

• The sql node and datax node automatically parse the dependency table and target table.
• The frontend controls whether to enable dependency detection through switch.
• The master server automatically injects dependent nodes, create dependent nodes based on dependencies.
• Rely on the node to set the default number of retries.
• Open the node that dependent detection function, no longer need to manually connect.

Already start a discussion in mail list.

---

• 支持sql节点和datax节点自动分析表之间的依赖关系，通过解析sql的select表和insert表实现，其他节点可以手工维护insert表（如果存在这种需求）
• 前端通过依赖检测开关控制master在任务调度时是否进行依赖解析
• master server自动注入依赖节点，运行时根据依赖关系生成虚拟依赖节点，不修改工作流定义数据，仅针对定时调度上线的工作流进行依赖分析
• 生成的依赖节点设置默认失败重试次数，比如每5分钟检测一下
• 打开依赖解析开关后，节点间不再需要手工连线，master根据依赖关系的顺序进行节点的调度

E.g.

[simon824]

Will this design cause of loops？

------------------ 原始邮件 ------------------ 发件人: "Rubik-W"<notifications@github.com>; 发送时间: 2020年4月15日(星期三) 下午2:58 收件人: "apache/incubator-dolphinscheduler"<incubator-dolphinscheduler@noreply.github.com>; 抄送: "simon"<3656562@qq.com>;"Comment"<comment@noreply.github.com>; 主题: Re: [apache/incubator-dolphinscheduler] Lineage data kinship（Lineage数据血缘关系） (#249)

I plan to implement data lineage function (Table level).

The sql node and etl node automatically parse the dependency table and target table.

The frontend controls whether to enable dependency detection through switch.

The master server automatically injects dependent nodes, create dependent nodes based on dependencies.

Rely on the node to set the default number of retries.

Open the node that dependent detection function, no longer need to manually connect.

Already start a discussion in mail list.

E.g.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[wen-hemin]

Won't loops, Current workflow does not support circular relationships. Checked when saving workflow.

[s751167520]

+1 Can the blood relationship of the table field be resolved ?

[wen-hemin]

The current design is to solve the blood relationship between the tables.

[gzvince]

+1

[wacr2008]

+1

[kangjunxiang]

+1

[yeyudefeng]

+111111111111111

[jetaime-chen]

+1

[geosmart]

if anybody else also need this feature , please reply "+1", thx

finally, we will decide which version to implement this feature based on the level of attention.

+1

[geosmart]

just like airflow ,task have inlets and outlets attribute, user can define the inlets and outlets manualy; and after task finish ,the inlets/outlets will send to metadata server(e.g atlas)

how can I impl this in dolphinscheduler? @dailidong

[liangpingliu]

It has the necessary functions

[dengc367]

+1

[shitoubiao]

+1

[yh2388]

I plan to implement data lineage function (Table level).

• The sql node and etl node automatically parse the dependency table and target table.
• The frontend controls whether to enable dependency detection through switch.
• The master server automatically injects dependent nodes, create dependent nodes based on dependencies.
• Rely on the node to set the default number of retries.
• Open the node that dependent detection function, no longer need to manually connect.

Already start a discussion in mail list.

• 支持sql节点和etl节点自动分析表之间的依赖关系，通过解析sql的select表和insert表实现，其他节点可以手工维护insert表（如果存在这种需求）
• 前端通过依赖检测开关控制master在任务调度时是否进行依赖解析
• master server自动注入依赖节点，运行时根据依赖关系生成虚拟依赖节点，不修改工作流定义数据，仅针对定时调度上线的工作流进行依赖分析
• 生成的依赖节点设置默认失败重试次数，比如每5分钟检测一下
• 打开依赖解析开关后，节点间不再需要手工连线，master根据依赖关系的顺序进行节点的调度

E.g.

Hi @Rubik-W , I am testing your implement, How to set process_definition_json from frontend, Should you show me a sample?

[wen-hemin]

I plan to implement data lineage function (Table level).

• The sql node and etl node automatically parse the dependency table and target table.
• The frontend controls whether to enable dependency detection through switch.
• The master server automatically injects dependent nodes, create dependent nodes based on dependencies.
• Rely on the node to set the default number of retries.
• Open the node that dependent detection function, no longer need to manually connect.

Already start a discussion in mail list.

• 支持sql节点和etl节点自动分析表之间的依赖关系，通过解析sql的select表和insert表实现，其他节点可以手工维护insert表（如果存在这种需求）
• 前端通过依赖检测开关控制master在任务调度时是否进行依赖解析
• master server自动注入依赖节点，运行时根据依赖关系生成虚拟依赖节点，不修改工作流定义数据，仅针对定时调度上线的工作流进行依赖分析
• 生成的依赖节点设置默认失败重试次数，比如每5分钟检测一下
• 打开依赖解析开关后，节点间不再需要手工连线，master根据依赖关系的顺序进行节点的调度

E.g.

Hi @Rubik-W , I am testing your implement, How to set process_definition_json from frontend, Should you show me a sample?

When the front-end creates a workflow and clicks save, it is automatically generated according to the task configuration information.

[BobbySun]

+1, 这个功能很需要，我们还准备开发这样的功能呢

[ljp510016132]

+1 急需

[YiAnCN]

+1急需此功能

[alexcd90]

+1急需此功能

[lyyprean]

No description provided.

+1 +1 +1 急需此功能

[ricemouse]

+1

[glxfeng]

+1

[ntupapaya]

support +1

[brucemen711]

+1 I think we should not embedded with dolphin system. We could intro new module/plugin, so we could gen and put standard data lineage to other systems like apache atlas, marquez(Wework), ... @dailidong @boandai

[irhawks]

+1

[jczk]

+1

[zhujian86]

+1

[yimaixinchen]

+1

[jon-qj]

+1

[huanzui]

+1

[Level1Accelerator]

这个需求很重要，+1