search

apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
https://dolphinscheduler.apache.org/
Apache License 2.0
12.74k stars 4.58k forks source link

[Feature][all] We are ready to contribute multiple features #15928

Closed liukunyuan closed 1 month ago

liukunyuan commented 5 months ago

Search before asking

Description

We have these feature within our company and hope to contribute them. What feature communities require?

  1. Worker and master grayscale function based on traffic
  2. Optimization of worker and master load protection strategies (optimizing remaining physical memory and CPU algorithms on the machine) to address the issue of worker downtime
  3. Force skip dependency detection during complement
  4. Heartbeat reporting from master and worker to detect disconnection issues
  5. Write the Yarn appid into the task_instance table during task execution to facilitate worker fault tolerance (solving the problem of worker machine downtime and inability to tolerate faults)
  6. Workflow migration project interface
  7. Recommend users to set the cron for the workflow (query the completion time of all upstream workflows in the last 7 days and then recommend the start time of this workflow)
  8. After the task timeout fails, it is necessary to kill the Yarn task
  9. Change workflow instances and task instances to single table queries to solve the problem of slow queries
  10. Provide a batch recovery failure interface to solve the problem of slow recovery of workflow instances caused by cluster or MySQL issues
  11. Add feature control tables to MySQL. All important functions require parameter control to turn on and off

Are you willing to submit a PR?

Code of Conduct

wangxj3 commented 5 months ago

What causes the fourth problem? And I have some suggest: 1.The fifth problem may cause task loss. A data processing task may contain more then one yarn job. If it fails between yarn tasks, this may cause the fault-tolerant task to succeed but miss some yarn task submissions. 2.Can the eighth one be configured? The easy failure of some long tasks may lead to a waste of complement resources.

ruanwenjun commented 5 months ago

Please Create DSIP to describe why you want to do this change, it's hard to know what you want to do by a simple description, and some of these should already been supported.

liukunyuan commented 5 months ago

What causes the fourth problem? And I have some suggest: 1.The fifth problem may cause task loss. A data processing task may contain more then one yarn job. If it fails between yarn tasks, this may cause the fault-tolerant task to succeed but miss some yarn task submissions. 2.Can the eighth one be configured? The easy failure of some long tasks may lead to a waste of complement resources.

  1. What causes the fourth problem? Mysql network exception for a few minutes
  2. A data processing task may contain more then one yarn job。 It will write multiple appids to the task_instance table
github-actions[bot] commented 2 months ago

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

github-actions[bot] commented 1 month ago

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.