search

sensu / sensu-plugin-sdk

A framework for creating Sensu plugins
MIT License
7 stars 8 forks source link

Feature Enhancement: Include helper functions for handling of prometheus exposition formatted metrics #65

Closed jspaleta closed 2 years ago

jspaleta commented 2 years ago

Feature Description

Check plugin use case

Prometheus exposition format is desirable across several metric check plugins, it would be useful to have a set of SDK functions that check plugins could make use of for consistent well formed Prometheus exposition output instead of having each plugin roll its own process.

Handler plugin use case

Sensu handlers based on this SDK would also benefit from helper function that know how to take Sensu internal metrics format and convert them into Prometheus exposition in a consistent manner.

Considerations

  1. Prometheus metrics exposition format makes use of specially formatted unique comment strings to provide type information decorating a family of related metrics. These comment strings can only show up once per metrics family group.
  2. Sensu agent ingests Prometheus exposition format and converts the comments into specially named Sensu metrics tags that appear on all metrics. To support the handler use case, SDK metrics helper function would need to set/comprehend specially named tags consistent with how sensu-agent currently does the task.

Straw Implementation

Helper function to populate Sensu Metrics object programmatically

Helper function must be able to store optional TYPE and HELP comments consistent with how Sensu Agent ingests prom exposition metrics.

Check plugin use case

Check plugin executable would use this function to populate a Sensu Metrics structure as part of metrics collection business logic.

Handler plugin use case

Handlers are not expected to need this function

Helper function to output Sensu Metrics in prom exposition format

Helper function must be able to take Sensu Metrics object and export it in prometheus exposition formatted string building TYPE and HELP comment strings from Metrics tag information consistent with how sensu-agent currently ingests prometheus comment strings.

Check plugin use case

Check plugin executable would use this function to export metrics collected into Sensu Metrics object as Prometheus exposition formatted string.

Handler plugin use case

Handler plugin executable would use this function to export Sensu Metrics object as Prometheus exposition formatted string.

Minimum Viable Goal

  1. Need to be able to refactor system-check plugin to use these helper functions to create prom exp output: 
./system-check 
# HELP system_cpu_cores [GAUGE] Number of cpu cores on the system
# TYPE system_cpu_cores GAUGE
system_cpu_cores{} 8 1636582440402
# HELP system_cpu_idle [GAUGE] Percent of time all cpus were idle
# TYPE system_cpu_idle GAUGE
system_cpu_idle{cpu="cpu0"} 74.49664429582639 1636582440402
system_cpu_idle{cpu="cpu1"} 78.26086956470425 1636582440402
system_cpu_idle{cpu="cpu2"} 77.07641196137524 1636582440402
system_cpu_idle{cpu="cpu3"} 81.27090300950525 1636582440402
system_cpu_idle{cpu="cpu4"} 76.8456375843412 1636582440402
system_cpu_idle{cpu="cpu5"} 78.1144781152644 1636582440402
system_cpu_idle{cpu="cpu6"} 81.72757475139935 1636582440402
system_cpu_idle{cpu="cpu7"} 81.33333333413934 1636582440402
system_cpu_idle{cpu="cpu-total"} 78.56247388180051 1636582440402
# HELP system_cpu_used [GAUGE] Percent of time all cpus were used
# TYPE system_cpu_used GAUGE
system_cpu_used{cpu="cpu0"} 25.503355704173615 1636582440402
system_cpu_used{cpu="cpu1"} 21.73913043529575 1636582440402
system_cpu_used{cpu="cpu2"} 22.923588038624757 1636582440402
system_cpu_used{cpu="cpu3"} 18.729096990494753 1636582440402
system_cpu_used{cpu="cpu4"} 23.154362415658795 1636582440402
system_cpu_used{cpu="cpu5"} 21.885521884735596 1636582440402
system_cpu_used{cpu="cpu6"} 18.272425248600655 1636582440402
system_cpu_used{cpu="cpu7"} 18.66666666586066 1636582440402
system_cpu_used{cpu="cpu-total"} 21.437526118199486 1636582440402
...
  1. Need to be able to refactor sumologic handler plugin using the same helper functions to take event generated from system-check and ingested by sensu-agent and faithfully reproduce original prom exp format string.
c-kruse commented 2 years ago

@jspaleta can you check my understanding?

A full journey of this would be.

  1. Check Plugin has collected metric points, has some internal object representation of them, and a way to dump them to the text metric format of choice (maybe just prometheus for now?) - including the full annotation capabilities of prom (TYPE and HELP).
  2. sensu-agent's check handler parses the check output and attaches corev2.Metrics to the event before forwarding them on to the backend.
  3. in the backend some pipeline/mutator/handler subscribed to the event can take the event.Metrics object and marshal it to a functionally identical text output

If I've gorked this correctly, I think this needs to be a sensu-go issue rather than a sunsu-plugin-sdk issue because the agent is the component that is unmarshaling the text check output.

jspaleta commented 2 years ago

No this is not a sensu-go issue. I've rewritten the feature request to be make the check plugin executable use case need clearer.

jspaleta commented 2 years ago

Okay I think we have this released now in sdk 0.15.0 version. Closing.