Websocket trigger for long-lived deferable operators

[pwachira]

Description

Create a trigger for deferrable operators that uses a websocket client to send a long-lived/long-running task request to an external remote websocket server that handles (or intiates the handling of) the requested task. The Async trigger then triggers when the remote server replies The remote websocket server will be out of the scope of airflow and not implemented here.

Use case/motivation

A common use case is to run long lived tasks asynchronously when the duration of execution is unknown. Http requests are not the best for long lived connections so a http trigger may not be optimal. The currently provided timer trigger does not meet this requirement A websocket request would be better suited for this (I would even go further and propose a messaging system(e.g kafka) is the best option in such a case - I might open a different feature request if this current feature is approved)

Related issues

No response

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

By all means - please add it :). That's definitely something that will make many Defferable Operators easier to implement !

[potiuk]

Assigned you

[potiuk]

And messageing system triggers is the next best thing after that one too, especially if we could make them generic for multiple messaging systems. I am not sure if there are good proven libraries that could provide the right abstraction - I'd think something like Kombu (which we already use through Celery) might be a good candidate but I do not have enough deep experience to make the right call here.