kestra-io / kestra

:zap: Workflow Automation Platform. Orchestrate & Schedule code in any language, run anywhere, 500+ plugins. Alternative to Zapier, Rundeck, Camunda, Airflow...
https://kestra.io
Apache License 2.0
12.66k stars 1.08k forks source link

Investiguate execution processing improvement in the executor #5521

Open loicmathieu opened 3 weeks ago

loicmathieu commented 3 weeks ago

Issue description

The executor, when processing an execution, will sometimes create WorkerTaskResult to emulate a task run by the worker This is the case for example for flowable tasks as they are run by the executor. See for ex: https://github.com/kestra-io/kestra/blob/develop/jdbc/src/main/java/io/kestra/jdbc/runner/JdbcExecutor.java#L437 but there is multiple time this idiom.

As there is nothing wrong here, as we send them to the WorkerTaskResult queue, it will go back to the executor then enter the workerTaskTrigger queue message handling code which in final will lock the execution and create new task run. This seems counter-productive, for these cases, we can directly create the task run instead of sending a WorkerTaskResult as the execution will be updated just after.

The same optimization may apply to the Kafka Runner.

Malaydewangan09 commented 2 weeks ago

Hey @loicmathieu So the proposed solution would be to skip the queue for flowable tasks and update directly, right?

loicmathieu commented 2 weeks ago

This is just an idea, more exploration needs to be done. This is a very complex area as it touches the internal of the executor.

Malaydewangan09 commented 2 weeks ago

Sure Thanks!