Researching Contextual Event Integration in Observability Dashboards

[schwesig]

Motivation

Our current and planned observability infrastructure provides comprehensive metrics, logs, and traces, it lacks the ability to correlate these data points with specific system events. The absence of event context makes it challenging to directly link system changes, such as hardware upgrades or software deployments, with their impact on system reliability. Incorporating context around these events could significantly enhance our understanding and analysis capabilities.

Objective

Exploring methodologies and technologies for integrating contextual event markers into our observability (dashboards). Investigate both the technical feasibility and the potential value of embedding event context (e.g., hardware changes, software updates, configuration modifications) directly within our database and monitoring tools. Minimal wish to have a structured changelog that can be combined with the collected metrics.

Requirements

• Review of existing solutions or plugins within the Prometheus, Thanos, and Grafana ecosystem that allow for event marking.
• Check third-party tools (services???) that could be integrated.

Completion Criteria

• List of solutions (with pros and cons)
• Recommendation
• Possible timeline and resource requirement

Description

The goal of this issue is to hopefully find solutions to create insightful and actionable observability. We aim to enhance our ability to perform root cause analysis and improve system reliability proactively.

Considerations

• Scalability, Customizability, Multicluster usability.
• Need for collaboration open-source communities.
• Balance between granularity of event information and the usability of osbs.

Discussions

• Potential Challenges:
  • Identification of key event types that can have impact. (or just be safe and track all? learning phase?)
  • Designs/Diagrams for presenting event data alongside traditional observability metrics.
  • Possible computational and storage overhead by tracking (and displaying) event data.
  • Ensuring that the system remains intuitive for users.
• Possible Events to Cover:
  • Software upgrades (e.g., operator version changes).
  • Hardware changes (e.g., CPU replacement, RAM or storage upgrades).
  • Network adjustments (e.g., switch replacements, firewall rule changes).
  • Configuration changes (e.g., Kubernetes config adjustments, ACM policy updates).
• Alternative Presentation Ideas:
  • Timeline view with event markers plotted against metric trends.
  • Anomaly detection mode where the system automatically highlights metrics deviations post-event.
  • Comparative analysis tool allowing users to select an event and compare various metrics before and after the event.

[schwesig]

idea Event-Driven Ansible https://www.redhat.com/en/technologies/management/ansible/event-driven-ansible

[computate]

@schwesig Fortunately AAP Operator is installed in NERC! See https://github.com/OCP-on-NERC/nerc-ocp-config/pull/234

[computate]

I have worked on other event driven automation projects in Red Hat Financial Services Consulting that work perfectly with AAP. See three articles about our Event Driven Automation Portfolio Architecture here:

• Event-driven automation: Automation vs. event-driven automation
• Event-driven automation: What’s in an event-driven automation architecture?
• Event-driven automation: How to build an event-driven automation architecture

[schwesig]

https://community.grafana.com/t/how-to-create-vertical-line-in-histogram-chart/82982 https://dev.to/aleksk1ng/java-quarkus-cqrs-and-eventsourcing-microservice-example-2p8h