apache / linkis

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.
https://linkis.apache.org/
Apache License 2.0
3.27k stars 1.15k forks source link

[Feature][1.5.0] Linkis orchestrator module supports plugin configuration (#5082) #5083

Closed CharlieYan24 closed 1 month ago

CharlieYan24 commented 4 months ago

What is the purpose of the change

In some scenarios, the Linkis orchestrator module is not required, but the orchestrator does not support plugin configuration. Linkis orchestrator supports the use of plugins, and it is also possible to bypass Linkis orchestrator.

Related issues/PRs

Related issues: #5082

Brief change log

Checklist

Test Result

Execute hive task: image

Execute shell task: image

peacewong commented 4 months ago

ping~ @guoshupei

peacewong commented 4 months ago

It looks good to me, there are a few points that can be further optimized:

  1. Need to consider the update of log and progress information dealLog/dealProgressWithResource

  2. Whether classes related to org.apache.linkis.orchestrator.ecm can be reused

  3. callSimpleExecute needs to consider returning an asynchronous response and asynchronous notification of task completion.

  4. Need to support retry scenario

  5. 需要考虑日志和进度信息的更新dealLog/dealProgressWithResource

  6. org.apache.linkis.orchestrator.ecm相关的类是否可以复用

  7. callSimpleExecute需要考虑返回异步响应,任务异步通知完成的方式

  8. 需要支持retry场景

peacewong commented 1 month ago

Repeated PR #5120