Event tracing for operators

palourde commented 5 years ago

@cwjohnston commented on Mon Feb 25 2019

Expected Behavior

As an operator I can observe the lifecycle of events as they are processed by the backend.

Current Behavior

The logs generated by the backend expose some but not all of the activities/phases in the event lifecycle.

Possible Solution

Implement a trace command in sensuctl which allows operators to specify one or more patterns for matching based on entity names, subscriptions and/or check names.

When the backend processes events matching one or of these patterns, the event is reported to the operator in such a way that the event ingress from agent, mutation, filtering and handling can all be understood without having to resort to debugging via log messages.

Context

Customers

1. 2. 3. 4.

Your Environment

Sensu version used (sensuctl, sensu-backend, and/or sensu-agent):
Installation method (packages, binaries, docker etc.):
Operating System and version (e.g. Ubuntu 14.04):

@portertech commented on Wed Feb 27 2019

Rough thoughts/reaction: Fix/improve logging around many of the event actions. Build upon a logger abstraction to allow sensuctl/API to observe log events matching certain criteria.

@portertech commented on Wed Jul 03 2019

We need to keep it simple at first, adding the concept of an event id for tracing through the system would be hugely beneficial.

@annaplotkin let's line this one up 👍

@annaplotkin commented on Wed Jul 03 2019

@portertech 👍

@annaplotkin commented on Tue Jul 09 2019

Proposal:

Add mechanism to create a unique ID [UUID] for a given event. Attach backend hostname as a label.
Backend and Agents Events' API, it creates a unique ID on event submission.
Log event field name: event_uuid
Going through the whole system and updating any sort of log events to reference that event ID for context (this will allow us to 'thread the needle' for a particular event)

**** This will be ONLY for backend and agent logs. To trace, we will need to know which backend or agent.

@palourde commented on Tue Jul 09 2019

Issue moved to sensu/sensu-go #3125 via ZenHub

palourde commented 5 years ago

Duplicate of https://github.com/sensu/sensu-go/issues/3125

cwjohnston commented 4 years ago

Reopening, as #3125 and adjacent work represents the first step in implementing this tracing concept.

sensu / sensu-go