Emit Node/Workflow events to multiple target sinks

[AntoineDao]

Summary

This proposal is very similar to #2274 however we want to emit the events directly to a set of targets rather than rely on Kubernetes events for two reasons:

• Node events fired at rapid successions get bundled together and ignored. I raised this here and used a work around to suit our needs. That workaround is no longer reliable.
• We cannot customize the shape of the events being emitted which means we have to fetch extra information when receiving a workflow event

Use Cases

We would use this to replace our current workflow update tracking system. We currently use a combination of workflow events emitted to Kubernetes and read the logs from the controller to make requests to the server for an updated state of specific nodes of the workflow.

Proposed Implementation

As this is a feature we are interested in for our project we are looking to implement it on our end and propose it as a PR to the argo-workflows repository. The purpose of this issue is to get a feel the likely-hood of such a feature being integrated in the argo-workflows upstream repository

It looks like we could refactor the EventsRecorderManager to take extra configuration to emit events to different sinks with different formatting options according to user preferences. We would initially only implement an HTTP sink which could be used to send events to webhook targets such as slack or argo-events. We would then be in a good position to add more sink types as users need them.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[alexec]

This sounds a bit like #4355 .

[alexec]

Have you considered using a Kubernetes informer for this?