Ability to add custom prometheus labels

[megamattzilla]

Ability to add TMOS version as a prometheus metric label

The Prometheus Pull consumer output does not contain any labels that indicate TMOS version. When reviewing the metric data for multiple F5 sytems, it is difficult to determine TMOS version differences.

A flag in prometheus pull consumer config that enables a TMOS version label.

When flag is enabled, metric output includes a version label like this:

# HELP f5_isEnabled isEnabled
# TYPE f5_isEnabled gauge
f5_isEnabled{virtualServers="/Common/Shared/demo-http",version="15.1.1"} 1
f5_isEnabled{virtualServers="/Common/http",version="15.1.1"} 1

Describe alternatives you've considered

The ability to specify custom labels would be very helpfull as well:

    "My_Pull_Consumer": {
        "class": "Telemetry_Pull_Consumer",
        "type": "Prometheus",
        "labels": [{
                "version": "15.1.1"
            },
            {
                "custom": "custom_data"
            }
        ],
        "systemPoller": "My_Poller"
    }

metric output includes custom labels like this:

# HELP f5_isEnabled isEnabled
# TYPE f5_isEnabled gauge
f5_isEnabled{virtualServers="/Common/Shared/demo-http",version="15.1.1",custom="custom_data"} 1
f5_isEnabled{virtualServers="/Common/http",version="15.1.1",custom="custom_data"} 1

[Nachtfalkeaw]

This is another bug in the OpenTelemetry Streaming app. If you use the "OpenTelemetry Push" consumer and you send the default metrics to Prometheus (https://prometheus.io/docs/prometheus/latest/feature_flags/ --enable-feature=otlp-write-receiver) you have information about model, chassis, etc.

If you use Prometheus Pull consumer these information are missing.

[megamattzilla]

wow 2 years! feels just like yesterday ^_^

2024 update:

Customer likes using prometheus pull consumer; however many, many Big-IP metrics have strings for values as seen in the example output.

Prometheus itself only supports integer metrics so any strings need to go into labels. F5 Telemetry streaming is pretty limited in adding labels on prometheus pull consumer metrics to get the equivalent data you would get from a push consumer.

The ability to allow Telemetry Streaming administrators to add global and custom_endpoint labels would be really useful.

1. Global labels example: see example above

2. Custom Endpoint labels:

   "Custom_Endpoints": {
       "class": "Telemetry_Endpoints",
       "items": {
           "detailedCPU": {
               "name": "detailedCPU",
               "path": "/mgmt/tm/sys/host-info",
               "labels": [
                   {
                       "version": "15.1.1"
                   },
                   {
                       "hostname": "F5.local"
                   }
               ]
           }
       }
   },

Example Metric output:

# HELP f5_detailedCPU_sys_host_info_0_activeCpuCount detailedCPU_sys/host-info/0_activeCpuCount
# TYPE f5_detailedCPU_sys_host_info_0_activeCpuCount gauge
f5_detailedCPU_sys_host_info_0_activeCpuCount{version="15.1.1",hostname="F5.local"} 2

BONUS points would be exposing some kind of variable interpolation system so that common things like hostname, pool name, F5 version could be referenced in the custom labels as variables like $hostname, $pool, $tmosVersion. Or allowing administrators to define expressions that would be executed to define variables to reference in label definitions.