search

Netflix / conductor

Conductor is a microservices orchestration engine.
Apache License 2.0
12.83k stars 2.33k forks source link

Push task to workers rather than poll repeatly #2020

Open HenryLauu opened 3 years ago

HenryLauu commented 3 years ago

We have thousands of tasks most of which are executed once a day. These workers keep polling task 24-hours , Any consideration in conductor 3.0 to allow using push mechanic?

dcore94 commented 3 years ago

That would be great even if I suspect it could be a major architectural change with impact on many aspects.... anyway if this really becomes a possibility, please, consider lightweight solutions like web-hooks or websockets and not bloated old-fashioned RPC paradigms ... thanks a lot!

apanicker-nflx commented 3 years ago

Thanks for the suggestions, this is on the long-term roadmap for Conductor task execution. As @dcore94 mentioned above, this would involve major architectural changes and would need a smooth migration strategy for the clients. There is definitely a major value-add but given the prerequisite work needed and the other items on the roadmap, I imagine we will not get to this in the first half of 2021.

leviliangtw commented 1 year ago

Look forward to a push-based Conductor too!

charybr commented 1 year ago

+1 This can be accomplished by introducing TaskStatusListener interface so that when a task is scheduled it can notify implementor of TaskStatusListener interface. Implementor can be a web-hook call. More details in below link https://github.com/Netflix/conductor/discussions/3587

Webhook module PR is work in progress: https://github.com/Netflix/conductor-community/pull/222