apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
https://dolphinscheduler.apache.org/
Apache License 2.0
12.41k stars 4.5k forks source link

[Feature][API] Data import and export [version 2.0] #5824

Open BoYiZhang opened 2 years ago

BoYiZhang commented 2 years ago

In version 2.0, it is recommended to support the import and export of workflow definition / resource file / data source

[data format compatible with Azkaban.]

At present, the data format of import / export is JSON format

If it is JSON format, there is no way to do offline development / modeling

So, can we use Azkaban's workflow definition method / data format

advantage:

  1. Compatible with Azkaban, achieve seamless docking. Users of Azkaban can migrate seamlessly

  2. Support offline development, not necessarily online through web client development

This allows and supports offline development

At the same time, also solved the development environment to the production environment migration pain point


2.0版本,工作流拆分完成后, 建议支持工作流定义/资源文件/数据源的导入导出.

[ 数据格式兼容azkaban. ]

目前倒入/导出的数据格式都是json格式. 如果是json格式的话,是没有办法做到线下离线开发/建模.

所以,我们是否可以沿用azkaban的工作流定义的方式/数据格式.

优点:

  1. 兼容azkaban, 做到无缝对接. azkaban的用户可以无缝迁移.
  2. 支持离线开发,不一定要在线通过web客户端开发.
  3. 解决了开发环境向生产环境迁移的痛点.
github-actions[bot] commented 2 years ago

In version 2.0, it is recommended to support the import and export of workflow definition / resource file / data source

[data format compatible with Azkaban.]

At present, the data format of import / export is JSON format

If it is JSON format, there is no way to do offline development / modeling

So, can we use Azkaban's workflow definition method / data format

advantage:

  1. Compatible with Azkaban, achieve seamless docking. Users of Azkaban can migrate seamlessly

  2. Support offline development, not necessarily online through web client development

This allows and supports offline development

At the same time, also solved the development environment to the production environment migration pain point


In version 2.0, after the workflow split is completed, it is recommended to support the import and export of workflow definitions/resource files/data sources.

[Data format compatible with azkaban.]

The data format currently imported/exported is in json format. If it is in json format, there is no way to do offline development/modeling.

So, can we continue to use the method/data format defined by azkaban's workflow?

advantage:

  1. Compatible with azkaban, to achieve seamless docking. azkaban users can seamlessly migrate.
  2. Support offline development, not necessarily online development through the web client.
  3. Solved the pain points of the migration from the development environment to the production environment.
github-actions[bot] commented 2 years ago

Hi:

CalvinKirs commented 2 years ago

At that time, we can discuss the specific design together.