[Feature][plugin] Supports user-defined task plugins

[hanhanzhang]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Description

Dolphin community has provided rich task plugins. In our business scenario, users hope to package the provided task plugins into products . For example, users can package HTTP services into custom task plugins and reuse them in multiple business workflows. Now we can use sub-workflows to meet the requirements, we consider whether we can provide user-defined plugin capabilities, compared to UDFs in SQL.

Use case

I hope build task ClassLoader based on task type, and add a task plugin resource.

Related issues

No response

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[SbloodyS]

For example, users can package HTTP services into custom task plugins and reuse them in multiple business workflows.

I'm not quite understand what is this. Do you mean http task type?

[hanhanzhang]

For example, users can package HTTP services into custom task plugins and reuse them in multiple business workflows.

I'm not quite understand what is this. Do you mean http task type?

Business service has many http request parameters, which are used in multiple workflows. Request parameters need to be configured each time. Business wants to productize the existing http service as a new task plugin (Common parameters provide default configuration, of course, Modify original service could be a solution, but default parameter configuration is constrained by business scenario). In addition, some services do not provide Http calls, but provide RPC calls (of course, through the shell script packaging into products).

[hanhanzhang]

Also based on business requirements, we want to see if community has plans to open up the ability of user-defined task plugins (similar to the UDF capability opened by SQL), of course, through shell scripts task plugin.

[SbloodyS]

I still don't know what is the meaning of Business?

Also based on business requirements, we want to see if community has plans to open up the ability of user-defined task plugins (similar to the UDF capability opened by SQL), of course, through shell scripts task plugin.

We already have the task plugin SPI.

[hanhanzhang]

I still don't know what is the meaning of Business?

Also based on business requirements, we want to see if community has plans to open up the ability of user-defined task plugins (similar to the UDF capability opened by SQL), of course, through shell scripts task plugin.

We already have the task plugin SPI.

Yes, community provides the strength task plugin SPI mechanism, but there is a question: Can the new task plugin be developed without starting worker?

[SbloodyS]

Can the new task plugin be developed without starting worker?

Not possible right now.

[hanhanzhang]

Can the new task plugin be developed without starting worker?

Not possible right now.

Whether there is a plan for this ability

[SbloodyS]

There are no plans to do so right now.

[SbloodyS]

Closing for now.

[hanhanzhang]

Closing for now.

If we want to contribute this code capability, what process should follow?

[SbloodyS]

It's unsafe to change task plugin during runtime. This can lead to a lot of task failures and unstable factors such as different task plugins between different workers, etc. We do not want this kind of feature.

[hanhanzhang]

It's unsafe to change task plugin during runtime. This can lead to a lot of task failures and unstable factors such as different task plugins between different workers, etc. We do not want this kind of feature.

got