salt_function_status() counter doesn't increment

[rjel2159]

The salt_function_status() counter doesn't increment but always stays 1 - here's an example when running total of 7 states on a single minion:

#  curl -s http://localhost:2112/metrics|grep ^salt_
salt_function_responses_total{function="state.apply",state="highstate",success="true"} 3
salt_function_responses_total{function="state.highstate",state="highstate",success="true"} 2
salt_function_responses_total{function="state.sls",state="sys.passwd",success="true"} 2
salt_function_status{function="state.apply",minion="gen12472",state="highstate"} 1
salt_function_status{function="state.highstate",minion="gen12472",state="highstate"} 1
salt_function_status{function="state.sls",minion="gen12472",state="sys.passwd"} 1
salt_responses_total{minion="gen12472",success="true"} 7
#

[kpetremann]

Hello @rjel2159,

salt_function_status is a gauge, not a counter. 1 = ok 0 = not ok

See the help of the metric:

$ curl -s http://localhost:2112/metrics | grep  salt_function_status
# HELP salt_function_status Last function/state success, 0=Failed, 1=Success

[kpetremann]

see PR #7 for details

[rjel2159]

Hello @rjel2159,

salt_function_status is a gauge, not a counter. 1 = ok 0 = not ok

See the help of the metric:

$ curl -s http://localhost:2112/metrics | grep  salt_function_status
# HELP salt_function_status Last function/state success, 0=Failed, 1=Success

I see, thanks for your response. It'd then be useful to have a metric such as salt_function_responses(minion=, function=, state=) that would record the count.

[kpetremann]

The cardinality would explode.

Why do you need that?

[rjel2159]

Mainly for testing to corelate salt_function_responses_total()'s functions/states counters to individual minions - meant to be used within small environments (i.e. DEV/QA) only. I totally agree with the concern of having a high cardinality which is why I had previously opened https://github.com/kpetremann/salt-exporter/issues/20 to allow disabling all minion-specific metrics.

[kpetremann]

:+1: If your need was only to troubleshoot an issue, I would have suggested either to use the salt-live tool in this repo, or to use dedicated returner to sent some metrics to a Prometheus Gateway.

Let me think about what would be the best approach. I would like to avoid current user to be impacted by such changes.

Something like a "verbose-metric" flag or something like this, which would either:

• create a new metric with more details
• or, most likely, add the minion label to salt_function_responses_total metric

I also need to think about the other current/future metrics. Users might expect to have the same option for these. So I might need to rethink the configuration to better scale the different options, while keeping this exporter simple to use and maintain.

[rjel2159]

Thanks for looking into this request. If you could implement (perhaps optionally) adding the minion label to the salt_function_responses_total() metric, that would be great.

[kpetremann]

Hi @rjel2159,

FYI the work is mostly done, I need to add more tests and update the documentation, as I completely refacto the configuration of the exporter.

You can already test it using the configurable_metrics branch.

You will need a configuration file called config.yml containing at least:

metrics:
  salt_function_responses_total:
    enabled: true
    add-minion-label: true

  salt_scheduled_job_return_total:
    enabled: true
    add-minion-label: true

[rjel2159]

Awesome, thanks so much for the update!

 On Thursday, June 29, 2023 at 12:25:05 PM EDT, Kevin Petremann ***@***.***> wrote:  

Hi @rjel2159,

FYI the work is mostly done, I need to add more tests and update the documentation, as I completely refacto the configuration of the exporter.

You can already test it using the configurable_metrics branch.

You will need a configuration file called config.yml containing at least: metrics: salt_function_responses_total: enabled: true add-minion-label: true salt_scheduled_job_return_total: enabled: true add-minion-label: true — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[kpetremann]

The pre-release is available for testing: https://github.com/kpetremann/salt-exporter/releases/tag/v0.7.0-pre.1

The README has been updated.

[kpetremann]

The feature is now available in the v0.7.0 which has just been released.

@rjel2159 feel free to reopen this issue, or another one, if you encounter any problem.

[rjel2159]

Great, thank you so much. I'll test this new feature shortly.

 On Thursday, July 6, 2023 at 12:26:05 PM EDT, Kevin Petremann ***@***.***> wrote:  

The feature is now available in the v0.7.0 which has just been released.

@rjel2159 feel free to reopen this issue, or another one, if you encounter any problem.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>