Support associating proxy check results with the executing agent entity

[cwjohnston]

Expected Behavior

Results from proxy checks can be associated with the agent entity which ran the check.

Current Behavior

Results from proxy checks are always associated with the proxy entity

Possible Solution

Implement a check config attribute which causes results to be associated with the executing agent entity.

Context

As an operator maintaining multiple databases on a single compute resource, I implement checks to monitor various aspects of each, e.g. cache hit rates, long running queries, etc.

I can create proxy entities representing each of my databases, specifying labels on each entity which help my checks connect to the corresponding ip/port, and configure a series of proxy checks which inserts these values using token substitution.

Unfortunately, when one or more of these database instances begins to alert, I won't know right away which of my database servers this instance is running on. I can retrieve this information from a label or annotation on the proxy entity, but this likely involves an extra step.

Compare this to Nagios where the same check can be applied to a host multiple times with different settings. Although the Nagios approach is less dynamic, the results of these checks are associated with the host where the check has run. In a scenario where multiple databases are alerting, an operator can more easily determine that all of the alerts are coming from a single host which may have experienced a disk failure or some other fault.

[echlebek]

The feature needn't necessarily require the use of proxy entities, although this may be the most straightforward way to achieve it.

The customer would like a way to create different permutations of checks, with varying configuration values, by specifying a single check config, and getting back multiple events.

This is a departure from the Sensu data model, and would need some design analysis.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[echlebek]

edit: post was made in error.

[mcatngena]

Hi, I am not sure if comments from you @echlebek are related to the original request. Maybe I am wrong but my understanding is Sensu Go to support new field in events payload to indicate which real agent executed given proxy check. I can't recall the field from Sensu Core but it was there for 100%. It was very useful for round robin proxy checks where e.g. one machine may have missing libraries for the check plugin and check "randomly" fails - in such situation, with that field, you could identify which real node/agent executed given command instead of going though all nodes/agents manually. Isn't this request just about that? And I vote for this for sure. It's missing in Sensu Go and was very helpful in Sensu Core.

[echlebek]

@mcbsd sorry, you're right, I actually meant to comment on another issue with the above post! I'll delete it to avoid possible confusion.

I'm still in favour of implementing the issue as originally described.