search

apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
https://dolphinscheduler.apache.org/
Apache License 2.0
12.89k stars 4.63k forks source link

[DSIP-44] Add owner for provider managed plugins #16093

Open EricGao888 opened 5 months ago

EricGao888 commented 5 months ago

Search before asking

Description

IMHO we could classify current plugins into two categories:

  1. Community-Managed plugins: widely-used plugins such as shell, hive, spark, flink, etc.
  2. Provider-Managed plugins: plugins used in specific scenario, which enriches the open-source ecosystem of Dolphin.

With the growing number of provider-managed plugins, the community do not have enough hands to ensure the stability and security. Therefore, I suggest each provider-managed plugin should have an owner, responsible for the stability and security issues. Before each release of Dolphin, we need the owners to submit a test report for the plugins they cover. Failure to submit test reports within release date for three consecutive times will lead to the retirement of the plugin.

We could call for a vote later to decide which plugins will be managed by the community.

[WIP] Community-Managed Plugins:

[WIP] Provider-Managed Plugins: Name Owner
dataquality-plugin
zeppelin-plugin @EricGao888
jupyter-plugin @EricGao888

......

Are you willing to submit a PR?

Code of Conduct

SbloodyS commented 5 months ago

I'm big +1 on this.

ruanwenjun commented 5 months ago

+1, I added to DSIP

hanhanzhang commented 2 months ago

@EricGao888 Hi, I'm interested in this issue, we also need to support user-defined plugins. Is there any design for this capability?

EricGao888 commented 2 months ago

@EricGao888 Hi, I'm interested in this issue, we also need to support user-defined plugins. Is there any design for this capability?

Hi, this proposal is not a design. This is about maintenance. Dolphin has supported many kinds of plugins since years before. You could check the dolphinscheduler-task-plugin folders if you want to add a customized one.

hanhanzhang commented 2 months ago

@EricGao888 Hi, I'm interested in this issue, we also need to support user-defined plugins. Is there any design for this capability?

Hi, this proposal is not a design. This is about maintenance. Dolphin has supported many kinds of plugins since years before. You could check the dolphinscheduler-task-plugin folders if you want to add a customized one.

Thanks for your reply. Business wants to package task plugin into products, eg: for HTTP tasks, business wants to package HTTP services into product plugins, which can be directly used in multiple scenarios. At present, Dolphin's sub-workflow should also be able to meet the requirements. We are considering whether Dolphin community plans to compare this capability with the UDF in SQL and allow users to customize task plugins.

SbloodyS commented 2 months ago

Hi @hanhanzhang , you can raise a new issue to discuss it first.

hanhanzhang commented 2 months ago

Hi @hanhanzhang , you can raise a new issue to discuss it first.

Thanks for reply@SbloodyS , I have created a new issue