sebastianw
commented
6 months ago I just found the dump_responses option for the gnmi plugin and can confirm now that this seems to be a problem with multiple updates.
Checking the output file in prometheus format I see this:
gnmi_sys_memory_used{host="tanagra",path="/system/memory/state",source="arista-switch"} 3.478126592e+09
gnmi_sys_memory_used{host="tanagra",path="/system/memory/state",source="arista-switch"} 3.478450176e+09
gnmi_sys_memory_used{host="tanagra",path="/system/memory/state",source="arista-switch"} 3.479629824e+09
gnmi_sys_memory_used{host="tanagra",source="arista-switch"} 3.479527424e+09
gnmi_sys_memory_used{host="tanagra",path="/system/memory/state",source="arista-switch"} 3.479810048e+09
gnmi_sys_memory_used{host="tanagra",path="/system/memory/state",source="arista-switch"} 3.479003136e+09
gnmi_sys_memory_used{host="tanagra",path="/system/memory/state",source="arista-switch"} 3.478552576e+09
gnmi_sys_memory_used{host="tanagra",path="/system/memory/state",source="arista-switch"} 3.477671936e+09
gnmi_sys_memory_used{host="tanagra",path="/system/memory/state",source="arista-switch"} 3.47871232e+09This corresponds to this debug output from telegraf:
2024-03-06T15:08:53Z D! [inputs.gnmi] Got update_1709737723565746678: {"update":{"timestamp":"1709737723565746678","prefix":{"elem":[{"name":"system"},{"name":"memory"},{"name":"state"}]},"update":[{"path":{"elem":[{"name":"reserved"}]},"val":{"uintVal":"6357975040"}},{"path":{"elem":[{"name":"used"}]},"val":{"uintVal":"3478126592"}}]}}
2024-03-06T15:09:03Z D! [inputs.gnmi] Got update_1709737733566450753: {"update":{"timestamp":"1709737733566450753","prefix":{"elem":[{"name":"system"},{"name":"memory"},{"name":"state"}]},"update":[{"path":{"elem":[{"name":"reserved"}]},"val":{"uintVal":"6358298624"}},{"path":{"elem":[{"name":"used"}]},"val":{"uintVal":"3478450176"}}]}}
2024-03-06T15:09:13Z D! [inputs.gnmi] Got update_1709737743568119333: {"update":{"timestamp":"1709737743568119333","prefix":{"elem":[{"name":"system"},{"name":"memory"},{"name":"state"}]},"update":[{"path":{"elem":[{"name":"reserved"}]},"val":{"uintVal":"6359478272"}},{"path":{"elem":[{"name":"used"}]},"val":{"uintVal":"3479629824"}}]}}
2024-03-06T15:09:23Z D! [inputs.gnmi] Got update_1709737753565697718: {"update":{"timestamp":"1709737753565697718","update":[{"path":{"elem":[{"name":"system"},{"name":"memory"},{"name":"state"},{"name":"used"}]},"val":{"uintVal":"3479527424"}}]}}
2024-03-06T15:09:33Z D! [inputs.gnmi] Got update_1709737763565722269: {"update":{"timestamp":"1709737763565722269","prefix":{"elem":[{"name":"system"},{"name":"memory"},{"name":"state"}]},"update":[{"path":{"elem":[{"name":"reserved"}]},"val":{"uintVal":"6359670784"}},{"path":{"elem":[{"name":"used"}]},"val":{"uintVal":"3479810048"}}]}}
2024-03-06T15:09:43Z D! [inputs.gnmi] Got update_1709737773566721549: {"update":{"timestamp":"1709737773566721549","prefix":{"elem":[{"name":"system"},{"name":"memory"},{"name":"state"}]},"update":[{"path":{"elem":[{"name":"reserved"}]},"val":{"uintVal":"6358863872"}},{"path":{"elem":[{"name":"used"}]},"val":{"uintVal":"3479003136"}}]}}
2024-03-06T15:09:53Z D! [inputs.gnmi] Got update_1709737783565848683: {"update":{"timestamp":"1709737783565848683","prefix":{"elem":[{"name":"system"},{"name":"memory"},{"name":"state"}]},"update":[{"path":{"elem":[{"name":"reserved"}]},"val":{"uintVal":"6358413312"}},{"path":{"elem":[{"name":"used"}]},"val":{"uintVal":"3478552576"}}]}}
2024-03-06T15:10:03Z D! [inputs.gnmi] Got update_1709737793566147217: {"update":{"timestamp":"1709737793566147217","prefix":{"elem":[{"name":"system"},{"name":"memory"},{"name":"state"}]},"update":[{"path":{"elem":[{"name":"reserved"}]},"val":{"uintVal":"6357532672"}},{"path":{"elem":[{"name":"used"}]},"val":{"uintVal":"3477671936"}}]}}
2024-03-06T15:10:13Z D! [inputs.gnmi] Got update_1709737803565776294: {"update":{"timestamp":"1709737803565776294","prefix":{"elem":[{"name":"system"},{"name":"memory"},{"name":"state"}]},"update":[{"path":{"elem":[{"name":"reserved"}]},"val":{"uintVal":"6358564864"}},{"path":{"elem":[{"name":"used"}]},"val":{"uintVal":"3478712320"}}]}}
srebhan
Relevant telegraf.conf
Logs from Telegraf
System info
Telegraf 1.29.5, MacOS Homebrew
Docker
No response
Steps to reproduce
Expected behavior
Measurements have consistent path attributes.
Actual behavior
Measurements sometimes are missing the path attribute. This leads for example to duplicate measurements in the prometheus exporter. The following should be one measurement, not two:
Checking the output file in prometheus format we can see that this changes from time to time for the same measurement:
Checking the same metric in the influxdb output file, it seems that the path only get recorded when there are multiple measurements in one update (in this case
user/minanduser/min_time):Additional info
Testing the same gNMI output from a python script with pygnmi show consistent measurements coming from the switch. The only difference is that updates with multiple measurements have a prefix and relative path while single measurement updates have a full path: