Open umialpha opened 7 months ago
Actually, we implement this feature on our private fork branch, but we don't think it is general approach. i wonder whether the community consider this feature.
@umialpha interested in this, put some notes on https://github.com/argoproj/argo-events/issues/1827
Why not exporting events once a workflow has completed?
Watch Events from external database besides k8s, e.g. ElasticSearch or ClickHouse.
This sounds like a feature request for Argo Events, not Workflows?
sry for confusion. I think it is a feature request of argoworkflow. The events are the kubernetes events.
Currenty, argoworkflow list-watch k8s events from k8s source. But usually k8s events lifecycle is short. In production, k8s events usually are sinked into external database like clickhouse or es.
So, is there any plan to add support to list k8s-events from other databases?
Oh you mean in the UI specifically for historical reasons. Yes currently the UI provides a list watch on k8s events for Workflows to help with debugging etc. As the k8s events you're referring to are historical for completed Workflows, it would just be a list
, no need to watch in that case.
That is currently k8s native (which is ideal), and as far as I know, there is no standard historical k8s events database or standardized API. At my last job, we had historical k8s events in New Relic. That would make it very difficult to integrate as each integration would need its own implementation, connectors, dependencies, etc. I would say that this would potentially be a good use-case for plugins a la #6943. I don't think it would be wise for Argo to support this out-of-the-box as it would be a hefty maintenance burden for a small number of users.
We do already recommend using standard k8s tooling for historical observability, such as for logs, so this would fit into that pattern. Although integrating with existing tooling may be a better paradigm than building an Argo-specific implementation like the archiveLogs
feature. An archiveEvents
feature could provide similar support via an Artifact Repository without too much legwork, but that would also be a not recommended hack and would not provide integration with other events stores as you describe.
Summary
Watch Events from external database besides k8s, e.g. ElasticSearch or ClickHouse.
Usually, k8s doesn't store events too long. In production, we sometimes export events to external database for more use cases. There are some popular exporter to do this. For example, https://github.com/AliyunContainerService/kube-eventer https://github.com/resmoio/kubernetes-event-exporter
There are at least two cases why we need this features:
Use Cases
When would you use this?
To track the failure reason for workflows, especially for archived workflows.
Message from the maintainers:
Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.