search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.51k stars 5.55k forks source link

gnmi to prometheus on ios xe devices fails #11879

Closed peterbaumert closed 1 year ago

peterbaumert commented 1 year ago

Relevant telegraf.conf

[[inputs.gnmi]]
  addresses = ["****"]
  username = "****"
  password = "****"

  encoding = "json_ietf"

  ## redial in case of failures after
  redial = "60s"

[[inputs.gnmi.subscription]]
  name = "oc-interfaces"
  origin = "openconfig"
  path = "/interfaces/interface"
  subscription_mode = "sample"
  sample_interval = "60s"

[[outputs.prometheus_client]]                    
  listen = ":9011"                               
  collectors_exclude = ["gocollector", "process"]
  namepass = ["oc-*"]

Logs from Telegraf

2022-09-26T08:04:57Z E! [outputs.prometheus_client] Error creating prometheus metric: key: oc_interfaces_11_subinterfaces_subinterface_0_index, labels: [0 10.52.239.210 0 unused unused 0 SWITCHNAME Twe2/0/6 0 0 0 0 0 0 0 0 0
 0 TwentyFiveGigE2/0/1 0 TwentyFiveGigE1/0/2 1736112 SPEED_UNKNOWN 0 0 96 0 ethernetCsmacd 0 0 0 1663957156104000000 e0:69:ba:ff:2f:85 STATIC e0:69:ba:ff:2f:9b 0 0 0 0 TwentyFiveGigE1/0/22 0 0 Twe2/0/5 0 0 1663957156070000000 unused UP 0 0 0 ethernetC
smacd 0 0 0 0 0 0 0 0 0 0 0 10.17.60.145 1663957119723000000 unused 0 10.17.63.113 10.25.111.1 UP 0 0 0 0 0 0 0 0 unused TRUNK 0 e0:69:ba:ff:2f:8f 0 692193001 0 0 SPEED_10GB 0 0 46900879 10.17.60.105 STATIC UP 0 0 DISABLE 0 0 0 0 0 ethernetCsmacd Twen
tyFiveGigE1/0/15 GigabitEthernet0/0 0 Port-channel4 0 Port-channel4 DISABLE 1663957157103000000 0 0 0 0 LOWER_LAYER_DOWN Core: SWITCHNAME  Po4 [20G] (toSWITCHNAME  Po1) {Po4} 0 1663957119745000000 0 0 0 0 0 0 0 0 Port-channel3 0 TwentyF
iveGigE2/0/8 0 0 0 0 0 0 0 0 0 Port-channel5 0 0 0 0 TwentyFiveGigE2/0/15 0 1663957119747000000 0 SPEED_UNKNOWN 0 0 HundredGigE2/0/28 0 0 UP 0 10.17.63.158 0 10.17.63.65 e0:69:ba:ff:55:12 VSS: SWITCHNAME  Hun1/0/27 [10G] (to SWITCHNAME  Hu
n2/0/27) 0 0 0 Port-channel5 10.170.2.254 0 1251351790 0 0 0 HundredGigE2/0/28 LOWER_LAYER_DOWN FULL 0 0 e0:69:ba:ff:2f:84 0 0 0 0 SPEED_UNKNOWN UP 0 10.17.63.161 0 1663957157102000000 0 STATIC DISABLE 0 0 0 0 0 0 0 0 0 ethernetCsmacd 0 0 l3ipvlan 138
7015 unused 0 89 UP 0 0 0 0 10.17.63.97 TRUNK unused SPEED_UNKNOWN 0 0 UP 1663957127273000000 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ethernetCsmacd 0 Core: SWITCHNAME  Po4 [20G] (to SWITCHNAME  Po1) {Po4} 0 0 UP 0 0 unused 0 unused 0 0 0 0 0
0 0 0 0 0 e0:69:ba:ff:2f:8a ethernetCsmacd 0 TwentyFiveGigE2/0/18 LOWER_LAYER_DOWN 0 Core: SWITCHNAME  Twe2/0/5 [10G] (to SWITCHNAME  Ten2/0/1) {Po5} 0 0 0 10.17.60.1 0 e0:69:ba:ff:55:04 10097131 unused 0 0 0 0 Port-channel3 UP 22 0 0 0
0 ethernetCsmacd 0 0 UP 0 0 0 0 unused 0 TwentyFiveGigE2/0/4 194730 0 0 0 unused 0 0 0 0 0 SPEED_UNKNOWN unused 0 TwentyFiveGigE1/0/9 e0:69:ba:ff:55:0b 0 0 0 0 0 10.170.2.254 0 14619321 0 HPG0000002006 0 0 Port-channel6 0 0 TwentyFiveGigE1/0/7 0 0 0 e
thernetCsmacd 10.17.60.1 0 0 1663957119723000000 SPEED_UNKNOWN 0 unused 1663956980000000000 0 0 0 0 0 0 SPEED_UNKNOWN 0 SPEED_10GB 0 SPEED_UNKNOWN 0 0 0 0 0 1663957166478000000 0 701078077 0 Core: SWITCHNAME  Twe1/0/6 [1G] (to SWITCHNAME 
GigabitEthernet0/11) {Po6} 0 10.17.63.65 Twe2/0/4 0 0 0 TRUNK 0 12739638878 40342635 0 0 0 0 0 Twe1/0/1 0 0 FULL TwentyFiveGigE2/0/11 10.25.111.1 1663956980000000000 0 0 unused 0 0 0 0 0 0 TRUNK 0 0 0 0 0 0 TRUNK 0 0 0 Port-channel4 e0:69:ba:ff:55:0b
0 0 0 0 unused 0 0 STATIC e0:69:ba:ff:2f:88 0 0 1663957156068000000 0 0 LOWER_LAYER_DOWN 0 0 0 1912273 0 0 SPEED_UNKNOWN 0 0 0 0 1663957119717000000 unused 0 0 TRUNK 1663957157101000000 ethernetCsmacd 0 0 unused 0 0 0 0 10.25.111.1 0 0 TRUNK 0 SPEED_U
NKNOWN 0 0 0 0 1663957118415000000 0 10.17.63.113 0 Port-channel5 0 0 0 4328638 e0:69:ba:ff:55:05 0 0 ethernetCsmacd TwentyFiveGigE2/0/7 0 0 0 LOWER_LAYER_DOWN 0 0 0 0 TwentyFiveGigE1/0/17 0 0 1663957119726000000 0 unused ethernetCsmacd 0 0 FULL UP 7$
412 0 4134439 0 0 TwentyFiveGigE1/0/24 0 1663957131468000000 0 0 0 903467 0 UP 0 0 0 0 unused 0 0 TwentyFiveGigE1/0/17 53312636 UP 0 0 DISABLE 0 0 0 0 0 0 Port-channel5 0 0 0 0 10.17.63.65 0 0 0 0 0 0 0 LOWER_LAYER_DOWN 0 0 0 0 0 0 0 0 0 e0:69:ba:ff:5
5:05 0 unused 0 0 0 UP 0 10.17.63.97 ethernetCsmacd 10.17.63.158 UP 0 0 0 e0:69:ba:ff:55:16 0 159301214 TwentyFiveGigE2/0/18 0 UP e0:69:ba:ff:2f:90 0 0 TwentyFiveGigE2/0/7 e0:69:ba:ff:2f:85 0 0 0 0 0 unused 1663957156100000000 TwentyFiveGigE2/0/4 ethe
rnetCsmacd 0 0 unused STATIC DISABLE 0 0 0 1663957170361000000 1663957127793000000 1663957127273000000 0 UP 0 0 0 0 0 0 0 10.17.60.105 0 STATIC UP 0 0 0 0 0 0 299185 4177051396 DISABLE 0 SPEED_UNKNOWN 0 0 0 TwentyFiveGigE1/0/8 0 0 0 0 0 VSS: SWITCHNAME 
01-uta-1 Hun1/0/28 [10G] (to SWITCHNAME  Hun2/0/28) 0 1663957157076000000 0 0 0 0 10.17.63.113 UP 0 TwentyFiveGigE2/0/15 1663957157072000000 0 0 0 e0:69:ba:ff:55:0a 0 0 0 0 LOWER_LAYER_DOWN unused 0 Core: SWITCHNAME  Po6 [2G] (to SWITCHNAME  Po1) {Po6} 0 0 0 0 0 unused 0 e0:69:ba:ff:2f:96 0 0 DISABLE 0 0 0 0 0 1663957157071000000 0 0 0 Port-channel4 0 0 0 0 e0:69:ba:ff:55:07 0 0 0 0 0 Core: SWITCHNAME  Po4 [20G] (to SWITCHNAME  Po1) {Po4} e0:69:ba:ff:55:0a 0 0 e0:69
:ba:ff:2f:81 STATIC TwentyFiveGigE1/0/23 0 0 1663957118724000000 TRUNK 1312225 0 0 1663957119747000000 0 0 0 0 0 0 SPEED_UNKNOWN 0 FULL 0 0 0 0 0 0 0 UP 0 10.52.239.210 SPEED_UNKNOWN 0 0 0 TRUNK 0 0 e0:69:ba:ff:2f:86 0 0 UP 0 TwentyFiveGigE2/0/12 0 0
unused 0 0 0 0 1663957130663000000 0 0 e0:69:ba:ff:2f:80 89 ethernetCsmacd e0:69:ba:ff:2f:84 0 unused 0 0 0 e0:69:ba:ff:55:0f 0 0 0 0 TRUNK DISABLE TwentyFiveGigE2/0/6 0 0 0 0 1135699 TwentyFiveGigE2/0/19 0 1663957119721000000 604207 0 0 0 0 10.17.63.
161 0 0 0 0 1663957119721000000 0 HundredGigE1/0/25 0 0 0 0 TRUNK 0 0 UP 0 0 0 UP 159325894 0 e0:69:ba:ff:2f:8a 0 0 0 e0:69:ba:ff:55:06 0 277969 TwentyFiveGigE2/0/21 e0:69:ba:ff:2f:85 Vlan20 0 unused 10.170.2.254 ethernetCsmacd TwentyFiveGigE2/0/14 0
e0:69:ba:ff:55:01 0 0 0 0 0 0 0 0 0 0 0 TwentyFiveGigE1/0/6 40777846 0 UP UP 0 0 0 0 0 0 0 0 0 0 10.17.63.158 TwentyFiveGigE1/0/4 0 33373 0 TwentyFiveGigE2/0/18 0 0 0 0 10.17.63.97 0 0 0 TwentyFiveGigE1/0/6 DISABLE Port-channel6 21497231 0 0 0 1663957
119719000000  0 FULL 423933 0 UP 0 0 10.17.60.145 1663957119745000000 TwentyFiveGigE1/0/14 SPEED_1GB 0 0 0 0 TwentyFiveGigE1/0/3 0 0 10.17.63.161 0 Port-channel5 ethernetCsmacd 0 0 0 SPEED_UNKNOWN TwentyFiveGigE1/0/20 10.17.60.145 0 FULL 0 0 0 STATIC
0 e0:69:ba:ff:55:19 0 0 UP 0 0 unused Twe1/0/3 0 0 0 0 0 0 0 0 0 TwentyFiveGigE1/0/10 0 DISABLE STATIC 0 0 0 0 1663957118746000000 0 0 0 UP 1663957119719000000 0 0 0 0 0 unused 0 7993420 ethernetCsmacd 0 0 21440268 0 0 0 0 0 Port-channel3 HundredGigE2
/0/26 0 0 unused Core: SWITCHNAME  Twe1/0/4 [10G] (toSWITCHNAME  Ten1/0/1) {Po4} 0 0 8556 SPEED_UNKNOWN 0 0 0 0 0 1663957157071000000 0 0 SPEED_UNKNOWN TwentyFiveGigE2/0/12 Port-channel6 0 0 0 0 0 0 0 0 0 0 SPEED_UNKNOWN 0 0 0 0 10.17.6
0.1 1663957156067000000 0 0 1663956981000000000 0 0 0 TwentyFiveGigE2/0/18 LOWER_LAYER_DOWN 0 e0:69:ba:ff:2f:83 0 0 ethernetCsmacd 0 0 0 Port-channel6 0 51446 SPEED_UNKNOWN 0 0 0 0 0 0 e0:69:ba:ff:2f:8a 0 LOWER_LAYER_DOWN 0 1663957128636000000 0 0 0 C
ore: SWITCHNAME  Twe2/0/5 [10G] (toSWITCHNAME  Ten2/0/1) {Po5} 0 Port-channel6 0 0 0 0 LOWER_LAYER_DOWN 0 0 STATIC 0 1663957118749000000 TRUNK DISABLE 0 1663957157067000000 10.52.239.210 TwentyFiveGigE1/0/19 0 0 0 0 0 0 0 FULL TwentyFiv
eGigE1/0/14 0 0 FULL 0 TwentyFiveGigE1/0/5 0 UP ethernetCsmacd Port-channel4 TwentyFiveGigE2/0/4 0 0 0 0 TRUNK 0 0 ethernetCsmacd 0 0 0 0 0 3901045642 0 LOWER_LAYER_DOWN unused 1905692400 0 SPEED_UNKNOWN UP 0 0 TRUNK 0 10.17.60.105 0 37914078 0 Port-c
hannel6 0 e0:69:ba:ff:55:0c TwentyFiveGigE1/0/22 0 0 1663957118749000000 LOWER_LAYER_DOWN 0 0 0 0 0 0 0 0 0 0 1663957119721000000 unused 0 0 12372774062 18115742 Core: SWITCHNAME  Po1 [10G] (to SWITCHNAME  Po3) {Po1} 0 166395715707500000
0 0 0 0 e0:69:ba:ff:2f:8b 1663957157097000000 0 UP 0 TwentyFiveGigE1/0/10 0 Port-channel5 0 TwentyFiveGigE2/0/16 ethernetCsmacd 0 UP 0], err: "2_openconfig_vlan_routed_vlan_openconfig_if_ip_ipv6_state_counters_out_octets" is not a valid label name for
 metric "oc_interfaces_11_subinterfaces_subinterface_0_index"

System info

Telegraf 1.24.1 (git: HEAD@bd7d53fb), Debian GNU/Linux 11 (bullseye)

Docker

No response

Steps to reproduce

  1. use ios xe device
  2. create config as above
  3. start

Expected behavior

Telegraf outputs interface stats in prometheus metrics

Actual behavior

only above errors are shown

Additional info

No response

reimda commented 1 year ago

Are you sure the gnmi encoding is right? The config has encoding = "json_ietf" but it doesn't seem to be decoding correctly.

Could you also rerun your config with telegraf --test and post the output so we can see what the gnmi input plugin is producing? The error you included came from the prometheus output. It might be more simple to debug if we avoid using the output plugin entirely.

peterbaumert commented 1 year ago

See log:

telegraf_gnmi.txt

fyi all ips replaced with 0.0.0.0 output is the same for "json" and "json_ietf", ios xe only supports that anyway see Programmability Configuration Guide, Cisco IOS XE Cupertino 17.8.x

srebhan commented 1 year ago

@peterbaumert can you please output the collected metrics as influx-line-protocol e.g. by adding another outputs.file instance with data_format = "influx"!?

srebhan commented 1 year ago

According to prometheus' code labels cannot start with a number but they have to start with a letter or underscore. This is a limitation of prometheus in combination with how your device reports the paths for your subscription (i.e. starting with a number).

There is little from the Telegraf side we can do in the plugins, but you can easily fix the issue by adding a regex processor to your config

[[processors.regex]]
  namepass = ["oc-interfaces"]
  [[processors.regex.field_rename]]
  pattern = "^([^a-zA-Z_].*)$"
  replacement = "if${1}"

Does that solve your problem @peterbaumert?

telegraf-tiger[bot] commented 1 year ago

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Page. Thank you!