Prometheus SNMP setup cannot be ported

[coretemp]

Issue description

Either the example of the snmp-exporter people (https://github.com/prometheus/snmp_exporter) is wrong, or it cannot be expressed with the current module (which requires the source_labels attribute to be set).

scrape_configs:
  - job_name: 'snmp'
    static_configs:
      - targets:
        - 192.168.1.2  # SNMP device.
    metrics_path: /snmp
    params:
      module: [if_mib]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9116  # The SNMP exporter's real hostname:port.

[coretemp]

It appears that one needs snmpd to be installed for example, which is not even available on NixOS.

[thefloweringash]

Can you expand on what's not working, or what the desired setup is? I've been using the the snmp exporter for a while. The scrape_configs example in the snmp_exporter project can be fairly directly ported to services.prometheus.scrapeConfigs:

  services.prometheus = {
    scrapeConfigs = [
      {
        job_name = "snmp";
        metrics_path = "/snmp";
        params = { module = [ "if_mib" ]; };
        relabel_configs = [
          { source_labels = ["__address__"];    target_label = "__param_target"; }
          { source_labels = ["__param_target"]; target_label = "instance"; }
          { source_labels = []; target_label = "__address__"; replacement = "localhost:9116"; }
        ];
        static_configs = [{
          targets = ["switch.example.com"];
        }];
      }
    ];
  };

The exporter itself requires almost no configuration:

  services.prometheus.exporters.snmp = {
    enable = true;
    configuration = null;
    configurationPath = "${pkgs.prometheus-snmp-exporter.src}/snmp.yml";
  };

The exporter returns the results of scraping a remote host

$ curl -s 'http://localhost:9116/snmp?module=if_mib&target=switch.example.com' | grep ifName | head -4
# HELP ifName The textual name of the interface - 1.3.6.1.2.1.31.1.1.1.1
# TYPE ifName gauge
ifName{ifIndex="1",ifName="g1"} 1
ifName{ifIndex="13",ifName="cpu"} 1

And prometheus is including those metrics

$ curl -s 'http://prometheus.example.com:9090/api/v1/query?query=COUNT(ifName)%20by%20(instance)' | gron | grep -e 'value\[1\]' -e instance
json.data.result[0].metric.instance = "switch.example.com";
json.data.result[0].value[1] = "13";

[coretemp]

@thefloweringash In your example, you set source_labels = [];. In the upstream example it is not set at all. That's a deviation from upstream.

Now, I do believe that it might work if one actually has an SNMP implementation, but I wanted to use Prometheus to keep track of bandwidth throughput/second over time (to compute max, average, etc.).

So, there are two issues. One of the is the small deviation from upstream in the module configuration. The other is how to get an SNMP implementation on my NixOS system (or otherwise some method to accomplish the goal described earlier (I don't care about SNMP)).

[thefloweringash]

In your example, you set source_labels = [];. In the upstream example it is not set at all

I see it configured exactly the same in the upstream landing page, your copy-pasted example, and my example config.

I think for tracking network devices on a linux, you want the node exporter's "netdev" collector and a prometheus query like rate(node_network_receive_bytes_total[5m]).

[coretemp]

@thefloweringash Note the last element of the relabel_configs and observe that the source_label is not defined there.

[coretemp]

@thefloweringash Thanks for the query.

[thefloweringash]

@thefloweringash Note the last element of the relabel_configs and observe that the source_label is not defined there.

Indeed, apologies for being slow on the uptake there.

[globin]

cc @WilliButz

[andir]

I discussed that during the weekend with @WilliButz. We believe requiring assigning an empty list ([]) to be a bug..

It is counter-intuitive but semantically seems to be the same. The re-exported Prometheus configuration doesn't list the attribute if it is set like this.

[WilliButz]

@andir with that in mind, to avoid having the users explicitly set source_labels = []; for their relabel_configs we could add it as the default in the prometheus module:

--- a/nixos/modules/services/monitoring/prometheus/default.nix
+++ b/nixos/modules/services/monitoring/prometheus/default.nix
@@ -318,6 +318,7 @@ let
     options = {
       source_labels = mkOption {
         type = types.listOf types.str;
+        default = [];
         description = ''
           The source labels select values from existing labels. Their content
           is concatenated using the configured separator and matched against

Not sure if we wan't to do this though.

[andir]

Shouldn't we instead make it a nullOr (listOf str) and default to null while checking for null further down the file?

On Tue, 20 Nov 2018, 17:12 WilliButz <notifications@github.com wrote:

@andir https://github.com/andir with that in mind, to avoid having the users explicitly set source_labels = []; for their relabel_configs we could add it as the default in the prometheus module:

--- a/nixos/modules/services/monitoring/prometheus/default.nix+++ b/nixos/modules/services/monitoring/prometheus/default.nix@@ -318,6 +318,7 @@ let options = { source_labels = mkOption { type = types.listOf types.str;+ default = []; description = '' The source labels select values from existing labels. Their content is concatenated using the configured separator and matched against

Not sure if we wan't to do this though.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NixOS/nixpkgs/issues/50519#issuecomment-440330411, or mute the thread https://github.com/notifications/unsubscribe-auth/AAm_dKSkqJnFcaBWmueBsFSNV00-44wVks5uxCoAgaJpZM4YnnPO .