apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
https://dolphinscheduler.apache.org/
Apache License 2.0
12.88k stars 4.63k forks source link

[DSIP-53][Dependent] Suggest add process lineage parsing for dependent task #16255

Closed SbloodyS closed 4 months ago

SbloodyS commented 4 months ago

Search before asking

Motivation

The current dependency task data structure is complex, which is not conducive to lineage analysis. In order to improve the performance of lineage analysis, the extension of lineage application is increased.

Design Detail

CREATE TABLE `t_ds_process_lineage` (
  `id` int NOT NULL AUTO_INCREMENT,
  `process_definition_code` bigint NOT NULL,
  `process_definition_version` int NOT NULL,
  `task_deifnition_code` bigint NOT NULL,
  `task_definition_version` int NOT NULL,
  `dept_project_code` bigint NOT NULL COMMENT 'dependent project code',
  `dept_process_definition_code` bigint NOT NULL COMMENT 'dependent process definition code',
  `dept_task_definition_code` bigint NOT NULL COMMENT 'dependent task definition code',
  `create_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'create time',
  `update_time` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT 'update time',
  PRIMARY KEY (`id`),
  KEY `idx_process_code_version` (`process_definition_code`,`process_definition_version`),
  KEY `idx_task_code_version` (`task_deifnition_code`,`task_definition_version`),
  KEY `idx_dept_code` (`dept_project_code`,`dept_process_definition_code`,`dept_task_definition_code`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

Compatibility, Deprecation, and Migration Plan

No response

Test Plan

Add e2e test to ensure stability.

Code of Conduct