search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.57k stars 5.56k forks source link

Error in plugin: failed to decode: proto: cannot parse invalid wire-format data #13135

Closed tylersiemers closed 1 year ago

tylersiemers commented 1 year ago

Relevant telegraf.conf

# Telegraf Configuration
#
# Telegraf is entirely plugin driven. All metrics are gathered from the
# declared inputs, and sent to the declared outputs.
#
# Plugins must be declared in here to be active.
# To deactivate a plugin, comment out the name and any variables.
#
# Use 'telegraf -config telegraf.conf -test' to see what metrics a config
# file would generate.
#
# Environment variables can be used anywhere in this config file, simply surround
# them with ${}. For strings the variable must be within quotes (ie, "${STR_VAR}"),
# for numbers and booleans they should be plain (ie, ${INT_VAR}, ${BOOL_VAR})

# Global tags can be specified here in key="value" format.
[global_tags]
  database_name = "telemetry" # will tag all metrics with dc=us-east-1

# Configuration for telegraf agent
[agent]
  ## Default data collection interval for all inputs
  interval = "60s"
  ## Rounds collection interval to 'interval'
  ## ie, if interval="10s" then always collect on :00, :10, :20, etc.
  round_interval = true

  ## Telegraf will send metrics to outputs in batches of at most
  ## metric_batch_size metrics.
  ## This controls the size of writes that Telegraf sends to output plugins.
  metric_batch_size = 10000

  ## Maximum number of unwritten metrics per output.
  metric_buffer_limit = 200000

  ## Collection jitter is used to jitter the collection by a random amount.
  ## Each plugin will sleep for a random time within jitter before collecting.
  ## This can be used to avoid many plugins querying things like sysfs at the
  ## same time, which can have a measurable effect on the system.
  collection_jitter = "0s"

  ## Default flushing interval for all outputs. Maximum flush_interval will be
  ## flush_interval + flush_jitter
  flush_interval = "15s"
  ## Jitter the flush interval by a random amount. This is primarily to avoid
  ## large write spikes for users running a large number of telegraf instances.
  ## ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
  flush_jitter = "5s"

  ## By default or when set to "0s", precision will be set to the same
  ## timestamp order as the collection interval, with the maximum being 1s.
  ##   ie, when interval = "10s", precision will be "1s"
  ##       when interval = "250ms", precision will be "1ms"
  ## Precision will NOT be used for service inputs. It is up to each individual
  ## service input to set the timestamp at the appropriate precision.
  ## Valid time units are "ns", "us" (or "µs"), "ms", "s".
  precision = ""

  ## Log at debug level.
  debug = true
  ## Log only error level messages.
  # quiet = false

  ## Log file name, the empty string means to log to stderr.
  # logfile = "/var/log/telegraf/telegraf_log.log"

  ## The logfile will be rotated after the time interval specified.  When set
  ## to 0 no time based rotation is performed.  Logs are rotated only when
  ## written to, if there is no log activity rotation may be delayed.
  # logfile_rotation_interval = "0d"

  ## The logfile will be rotated when it becomes larger than the specified
  ## size.  When set to 0 no size based rotation is performed.
  logfile_rotation_max_size = "100MB"

  ## Maximum number of rotated archives to keep, any older logs are deleted.
  ## If set to -1, no archives are removed.
  logfile_rotation_max_archives = 5

  ## Override default hostname, if empty use os.Hostname()
  hostname = ""

  ## If set to true, do no set the "host" tag in the telegraf agent.
  omit_hostname = false

###############################################################################
#                            OUTPUT PLUGINS                                   #
###############################################################################

# Debug
#
[[outputs.file]]
#  ## Files to write to, "stdout" is a specially handled file.
  files = ["stdout", "/tmp/metrics_cisco_debug.out"]
#
#  ## Data format to output.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
  data_format = "influx"

# Configuration for influxdb server to send metrics to
[[outputs.influxdb]]
  # required
  urls = ["http://influx-telemetry.mycompany.com:8086"]
  database = "cisco_telemetry_debug"
  skip_database_creation = true

  ## HTTP Basic Auth
  username = "telegraf"
  password = "telegraf"
  precision = "s"
  timeout = "5s"

###############################################################################

[[processors.dedup]]
  order = 1
  ## Maximum time to suppress output
  dedup_interval = "600s"

###############################################################################
#                            INPUT PLUGINS                                    #
###############################################################################

# # Cisco model-driven telemetry (MDT) input plugin for IOS XR, IOS XE and NX-OS platforms
[[inputs.cisco_telemetry_mdt]]
#  ## Telemetry transport can be "tcp" or "grpc".  TLS is only supported when
#  ## using the grpc transport.
transport = "tcp"
max_msg_size = 2500000000
#
#  ## Address and port to host telemetry listener
service_address = ":57590"
[inputs.cisco_telemetry_mdt.tags]
    model = "XR"

[[inputs.cisco_telemetry_mdt]]
#  ## Telemetry transport can be "tcp" or "grpc".  TLS is only supported when
#  ## using the grpc transport.
transport = "grpc"
max_msg_size = 2000000000
#
#  ## Address and port to host telemetry listener
service_address = ":57591"
[inputs.cisco_telemetry_mdt.tags]
    model = "XE"

Logs from Telegraf

2023-04-24T17:58:34Z E! [inputs.cisco_telemetry_mdt] Error in plugin: failed to decode: proto: cannot parse invalid wire-format data
2023-04-24T17:58:34Z E! [inputs.cisco_telemetry_mdt] Error in plugin: invalid dialout flags: 32821
2023-04-24T17:58:34Z D! [inputs.cisco_telemetry_mdt] Closed Cisco MDT TCP dialout connection from [2607:f33f:4ff:9999:5::1]:46758
2023-04-24T17:58:34Z D! [inputs.cisco_telemetry_mdt] Accepted Cisco MDT TCP dialout connection from [2607:f33f:4ff:9999:5::1]:60812
2023-04-24T17:58:38Z D! [outputs.file] Buffer fullness: 0 / 200000 metrics
2023-04-24T17:58:44Z D! [outputs.influxdb] Buffer fullness: 0 / 200000 metrics

System info

Telegraf 1.26.1, Ubuntu 20.04.5, Docker 23.0.1, build a5ee5b1

Docker

docker run -d -p 57590:57590 -p 57591:57591 -v /etc/telegraf/telegraf_docker/cisco_mdt_debug.conf:/etc/telegraf/telegraf.conf:ro --name=telegraf_cisco_mdt_debug -h telegraf_cisco_mdt_debug telegraf:1.26.1

Steps to reproduce

  1. Send telemetry data from a Cisco IOS-XR Router (Version 7.5.2) for the following sensor path Cisco-IOS-XR-procmem-oper:processes-memory/nodes/node/process-ids/process-id
  2. Telegraf fails to decode the data
  3. See additional info for raw data

Expected behavior

The MDT plugin should decode the data if it isn't a problem with the Cisco side sending bad data.

Actual behavior

Fails to output any data.

Additional info

What does this message mean and how can a debug it further? Is this something wrong in the plugin or from the router sending the data?

Raw Json Data from the Cisco Router for the sensor path

{ "node_id_str": "LabXrDevice", "subscription_id_str": "app_TEST_200000001", "encoding_path": "Cisco-IOS-XR-procmem-oper:processes-memory/nodes/node/process-ids/process-id", "collection_id": "214228", "collection_start_time": "1682358890712", "msg_timestamp": "1682358892005", "data_json": [ { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5860 } ], "content": { "name": "fab_health_xml_srvr", "jid": 354, "pid": 5860, "text-seg-size": 28, "data-seg-size": 236856, "stack-seg-size": 136, "malloc-size": 593, "dyn-limit": 307200, "shared-mem": 25424, "physical-mem": 40800 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5870 } ], "content": { "name": "ipv6_ea", "jid": 280, "pid": 5870, "text-seg-size": 92, "data-seg-size": 303140, "stack-seg-size": 136, "malloc-size": 592, "dyn-limit": 307200, "shared-mem": 25860, "physical-mem": 41228 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 7024 } ], "content": { "name": "imaedm_server", "jid": 221, "pid": 7024, "text-seg-size": 60, "data-seg-size": 201280, "stack-seg-size": 136, "malloc-size": 581, "dyn-limit": 307200, "shared-mem": 7708, "physical-mem": 10264 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 7790 } ], "content": { "name": "nf_producer", "jid": 301, "pid": 7790, "text-seg-size": 40, "data-seg-size": 303100, "stack-seg-size": 136, "malloc-size": 571, "dyn-limit": 307200, "shared-mem": 37328, "physical-mem": 53112 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5862 } ], "content": { "name": "gcp_mpls_over_udp_ea", "jid": 404, "pid": 5862, "text-seg-size": 8, "data-seg-size": 303160, "stack-seg-size": 136, "malloc-size": 570, "dyn-limit": 307200, "shared-mem": 25120, "physical-mem": 40452 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 6017 } ], "content": { "name": "tamd_proc", "jid": 71553, "pid": 6017, "text-seg-size": 32, "data-seg-size": 302392, "stack-seg-size": 136, "malloc-size": 565, "dyn-limit": 307200, "shared-mem": 27584, "physical-mem": 43544 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5857 } ], "content": { "name": "mpls_io_ea", "jid": 197, "pid": 5857, "text-seg-size": 112, "data-seg-size": 302788, "stack-seg-size": 136, "malloc-size": 560, "dyn-limit": 307200, "shared-mem": 26824, "physical-mem": 42324 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5846 } ], "content": { "name": "fwd_driver_partner", "jid": 317, "pid": 5846, "text-seg-size": 84, "data-seg-size": 201284, "stack-seg-size": 136, "malloc-size": 558, "dyn-limit": 307200, "shared-mem": 9708, "physical-mem": 12460 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 6582 } ], "content": { "name": "vkg_l2fib_evpn_mac", "jid": 251, "pid": 6582, "text-seg-size": 20, "data-seg-size": 237764, "stack-seg-size": 136, "malloc-size": 557, "dyn-limit": 409600, "shared-mem": 26820, "physical-mem": 42868 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5861 } ], "content": { "name": "statsd_server", "jid": 194, "pid": 5861, "text-seg-size": 44, "data-seg-size": 266920, "stack-seg-size": 136, "malloc-size": 543, "dyn-limit": 307200, "shared-mem": 13944, "physical-mem": 16808 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 6585 } ], "content": { "name": "mpls_fwd_show_proxy", "jid": 208, "pid": 6585, "text-seg-size": 228, "data-seg-size": 237140, "stack-seg-size": 136, "malloc-size": 542, "dyn-limit": 307200, "shared-mem": 25844, "physical-mem": 41356 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 7044 } ], "content": { "name": "ssm_process", "jid": 238, "pid": 7044, "text-seg-size": 60, "data-seg-size": 464352, "stack-seg-size": 136, "malloc-size": 538, "dyn-limit": 307200, "shared-mem": 7584, "physical-mem": 10240 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5080 } ], "content": { "name": "shmwin_svr", "jid": 118, "pid": 5080, "text-seg-size": 56, "data-seg-size": 201184, "stack-seg-size": 136, "malloc-size": 525, "dyn-limit": 307200, "shared-mem": 8796, "physical-mem": 11412 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5096 } ], "content": { "name": "sld", "jid": 294, "pid": 5096, "text-seg-size": 112, "data-seg-size": 398604, "stack-seg-size": 136, "malloc-size": 442, "dyn-limit": 307200, "shared-mem": 10760, "physical-mem": 13108 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 7023 } ], "content": { "name": "icpe_sdep", "jid": 313, "pid": 7023, "text-seg-size": 8, "data-seg-size": 267040, "stack-seg-size": 136, "malloc-size": 403, "dyn-limit": 307200, "shared-mem": 7672, "physical-mem": 10204 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5092 } ], "content": { "name": "ltrace_sync", "jid": 408, "pid": 5092, "text-seg-size": 32, "data-seg-size": 399156, "stack-seg-size": 136, "malloc-size": 401, "dyn-limit": 307200, "shared-mem": 6712, "physical-mem": 9140 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5851 } ], "content": { "name": "sh_proc_mem_edm", "jid": 156, "pid": 5851, "text-seg-size": 24, "data-seg-size": 200580, "stack-seg-size": 136, "malloc-size": 395, "dyn-limit": 307200, "shared-mem": 7844, "physical-mem": 10516 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 7020 } ], "content": { "name": "cmp_edm", "jid": 209, "pid": 7020, "text-seg-size": 16, "data-seg-size": 200444, "stack-seg-size": 136, "malloc-size": 390, "dyn-limit": 307200, "shared-mem": 7112, "physical-mem": 9516 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5103 } ], "content": { "name": "fab_si", "jid": 234, "pid": 5103, "text-seg-size": 36, "data-seg-size": 332632, "stack-seg-size": 136, "malloc-size": 386, "dyn-limit": 307200, "shared-mem": 7400, "physical-mem": 10020 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5077 } ], "content": { "name": "cerrno_server", "jid": 332, "pid": 5077, "text-seg-size": 48, "data-seg-size": 201524, "stack-seg-size": 136, "malloc-size": 382, "dyn-limit": 307200, "shared-mem": 7704, "physical-mem": 11156 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 6426 } ], "content": { "name": "tam_entropy", "jid": 164, "pid": 6426, "text-seg-size": 12, "data-seg-size": 200256, "stack-seg-size": 136, "malloc-size": 377, "dyn-limit": 307200, "shared-mem": 9616, "physical-mem": 12852 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 7028 } ], "content": { "name": "netio_debug_partner", "jid": 406, "pid": 7028, "text-seg-size": 24, "data-seg-size": 201056, "stack-seg-size": 136, "malloc-size": 365, "dyn-limit": 307200, "shared-mem": 7412, "physical-mem": 9860 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5879 } ], "content": { "name": "ether_rewrite_helper", "jid": 346, "pid": 5879, "text-seg-size": 24, "data-seg-size": 200964, "stack-seg-size": 136, "malloc-size": 363, "dyn-limit": 307200, "shared-mem": 11548, "physical-mem": 13940 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5853 } ], "content": { "name": "sint_ma", "jid": 267, "pid": 5853, "text-seg-size": 16, "data-seg-size": 200956, "stack-seg-size": 136, "malloc-size": 359, "dyn-limit": 307200, "shared-mem": 7180, "physical-mem": 9536 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5116 } ], "content": { "name": "bundlemgr_checker", "jid": 389, "pid": 5116, "text-seg-size": 76, "data-seg-size": 200516, "stack-seg-size": 136, "malloc-size": 355, "dyn-limit": 307200, "shared-mem": 7240, "physical-mem": 9720 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5876 } ], "content": { "name": "debug_d", "jid": 250, "pid": 5876, "text-seg-size": 24, "data-seg-size": 200504, "stack-seg-size": 136, "malloc-size": 353, "dyn-limit": 307200, "shared-mem": 7232, "physical-mem": 9668 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5111 } ], "content": { "name": "prm_verifier", "jid": 254, "pid": 5111, "text-seg-size": 100, "data-seg-size": 200524, "stack-seg-size": 136, "malloc-size": 352, "dyn-limit": 307200, "shared-mem": 7412, "physical-mem": 9884 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5852 } ], "content": { "name": "sh_proc_threadname_edm", "jid": 158, "pid": 5852, "text-seg-size": 24, "data-seg-size": 200504, "stack-seg-size": 136, "malloc-size": 343, "dyn-limit": 307200, "shared-mem": 7012, "physical-mem": 9424 } }, { "timestamp": "1682358892003", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5855 } ], "content": { "name": "sysmgr_show_proc_all_edm", "jid": 401, "pid": 5855, "text-seg-size": 84, "data-seg-size": 200512, "stack-seg-size": 136, "malloc-size": 342, "dyn-limit": 307200, "shared-mem": 7012, "physical-mem": 9444 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 6601 } ], "content": { "name": "showd_lc", "jid": 135, "pid": 6601, "text-seg-size": 76, "data-seg-size": 201020, "stack-seg-size": 136, "malloc-size": 342, "dyn-limit": 307200, "shared-mem": 7284, "physical-mem": 9792 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5884 } ], "content": { "name": "timezone_notify", "jid": 235, "pid": 5884, "text-seg-size": 12, "data-seg-size": 200508, "stack-seg-size": 136, "malloc-size": 341, "dyn-limit": 307200, "shared-mem": 6804, "physical-mem": 9128 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5104 } ], "content": { "name": "showd_server", "jid": 348, "pid": 5104, "text-seg-size": 12, "data-seg-size": 200504, "stack-seg-size": 136, "malloc-size": 340, "dyn-limit": 307200, "shared-mem": 6744, "physical-mem": 9128 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5873 } ], "content": { "name": "procfind", "jid": 252, "pid": 5873, "text-seg-size": 24, "data-seg-size": 200512, "stack-seg-size": 136, "malloc-size": 340, "dyn-limit": 307200, "shared-mem": 7040, "physical-mem": 9456 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5847 } ], "content": { "name": "mempool_edm", "jid": 165, "pid": 5847, "text-seg-size": 8, "data-seg-size": 200508, "stack-seg-size": 136, "malloc-size": 340, "dyn-limit": 307200, "shared-mem": 6776, "physical-mem": 9140 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5125 } ], "content": { "name": "media_server", "jid": 295, "pid": 5125, "text-seg-size": 16, "data-seg-size": 199920, "stack-seg-size": 136, "malloc-size": 338, "dyn-limit": 307200, "shared-mem": 6496, "physical-mem": 8740 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 6110 } ], "content": { "name": "ixdb_gc", "jid": 181, "pid": 6110, "text-seg-size": 28, "data-seg-size": 199932, "stack-seg-size": 136, "malloc-size": 338, "dyn-limit": 307200, "shared-mem": 35524, "physical-mem": 37836 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5105 } ], "content": { "name": "aipc_cleaner", "jid": 350, "pid": 5105, "text-seg-size": 8, "data-seg-size": 265972, "stack-seg-size": 136, "malloc-size": 335, "dyn-limit": 307200, "shared-mem": 6452, "physical-mem": 8736 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5102 } ], "content": { "name": "meminfo_svr", "jid": 225, "pid": 5102, "text-seg-size": 8, "data-seg-size": 199988, "stack-seg-size": 136, "malloc-size": 334, "dyn-limit": 307200, "shared-mem": 6756, "physical-mem": 9140 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5994 } ], "content": { "name": "tams_proc", "jid": 71530, "pid": 5994, "text-seg-size": 436, "data-seg-size": 200192, "stack-seg-size": 136, "malloc-size": 330, "dyn-limit": 307200, "shared-mem": 9960, "physical-mem": 13180 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 6573 } ], "content": { "name": "syslog_infra_hm", "jid": 306, "pid": 6573, "text-seg-size": 12, "data-seg-size": 199788, "stack-seg-size": 136, "malloc-size": 285, "dyn-limit": 307200, "shared-mem": 6676, "physical-mem": 8944 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 4158 } ], "content": { "name": "sh_proc_memory", "jid": 69694, "pid": 4158, "text-seg-size": 12, "data-seg-size": 134312, "stack-seg-size": 136, "malloc-size": 234, "dyn-limit": 307200, "shared-mem": 6772, "physical-mem": 8960 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 5555 } ], "content": { "name": "udevd", "jid": 71091, "pid": 5555, "text-seg-size": 160, "data-seg-size": 1868, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 332, "physical-mem": 2260 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 4908 } ], "content": { "name": "bash", "jid": 70444, "pid": 4908, "text-seg-size": 1016, "data-seg-size": 1700, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 1372, "physical-mem": 3208 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 4491 } ], "content": { "name": "ds", "jid": 70027, "pid": 4491, "text-seg-size": 52, "data-seg-size": 86552, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 6900, "physical-mem": 26708 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 4467 } ], "content": { "name": "bash", "jid": 70003, "pid": 4467, "text-seg-size": 1016, "data-seg-size": 684, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 1356, "physical-mem": 2172 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 4157 } ], "content": { "name": "sh", "jid": 69693, "pid": 4157, "text-seg-size": 1016, "data-seg-size": 420, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 307200, "shared-mem": 1336, "physical-mem": 1632 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 4156 } ], "content": { "name": "sleep", "jid": 69692, "pid": 4156, "text-seg-size": 32, "data-seg-size": 172, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 292, "physical-mem": 368 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 4145 } ], "content": { "name": "sleep", "jid": 69681, "pid": 4145, "text-seg-size": 32, "data-seg-size": 172, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 292, "physical-mem": 368 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 3934 } ], "content": { "name": "udevd", "jid": 69470, "pid": 3934, "text-seg-size": 160, "data-seg-size": 1868, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 772, "physical-mem": 2704 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 3494 } ], "content": { "name": "sh", "jid": 69030, "pid": 3494, "text-seg-size": 1016, "data-seg-size": 808, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 1400, "physical-mem": 2340 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 3383 } ], "content": { "name": "msixd_static", "jid": 68919, "pid": 3383, "text-seg-size": 1080, "data-seg-size": 184, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 0, "physical-mem": 120 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 3365 } ], "content": { "name": "proxy_attach_static", "jid": 68901, "pid": 3365, "text-seg-size": 1616, "data-seg-size": 221616, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 880, "physical-mem": 3360 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 2975 } ], "content": { "name": "proxy_attach_static", "jid": 68511, "pid": 2975, "text-seg-size": 1616, "data-seg-size": 147884, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 880, "physical-mem": 1276 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 2706 } ], "content": { "name": "crond", "jid": 68242, "pid": 2706, "text-seg-size": 56, "data-seg-size": 748, "stack-seg-size": 528, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 732, "physical-mem": 1356 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 2660 } ], "content": { "name": "libvirtd", "jid": 68196, "pid": 2660, "text-seg-size": 380, "data-seg-size": 418516, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 3888, "physical-mem": 6352 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 2626 } ], "content": { "name": "xinetd", "jid": 68162, "pid": 2626, "text-seg-size": 156, "data-seg-size": 184, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 696, "physical-mem": 880 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 2607 } ], "content": { "name": "klogd", "jid": 68143, "pid": 2607, "text-seg-size": 28, "data-seg-size": 3656, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 320, "physical-mem": 3868 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 2604 } ], "content": { "name": "syslogd", "jid": 68140, "pid": 2604, "text-seg-size": 44, "data-seg-size": 176, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 544, "physical-mem": 688 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 2598 } ], "content": { "name": "rngd", "jid": 68134, "pid": 2598, "text-seg-size": 20, "data-seg-size": 244, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 4, "physical-mem": 140 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 2554 } ], "content": { "name": "auditd", "jid": 68090, "pid": 2554, "text-seg-size": 120, "data-seg-size": 73968, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 428, "physical-mem": 620 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 2544 } ], "content": { "name": "rpcbind", "jid": 68080, "pid": 2544, "text-seg-size": 44, "data-seg-size": 200, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 520, "physical-mem": 716 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 2533 } ], "content": { "name": "sshd", "jid": 68069, "pid": 2533, "text-seg-size": 704, "data-seg-size": 440, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 2008, "physical-mem": 2504 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 2515 } ], "content": { "name": "dbus-daemon", "jid": 68051, "pid": 2515, "text-seg-size": 408, "data-seg-size": 8408, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 644, "physical-mem": 904 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 2460 } ], "content": { "name": "bash", "jid": 67996, "pid": 2460, "text-seg-size": 1016, "data-seg-size": 1124, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 1320, "physical-mem": 2568 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 2276 } ], "content": { "name": "bash", "jid": 67812, "pid": 2276, "text-seg-size": 1016, "data-seg-size": 1148, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 344, "physical-mem": 1616 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 2261 } ], "content": { "name": "bash", "jid": 67797, "pid": 2261, "text-seg-size": 1016, "data-seg-size": 200, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 1344, "physical-mem": 1668 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 1476 } ], "content": { "name": "udevd", "jid": 67012, "pid": 1476, "text-seg-size": 160, "data-seg-size": 1872, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 856, "physical-mem": 2784 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 7018 } ], "content": { "name": "bash", "jid": 161, "pid": 7018, "text-seg-size": 1016, "data-seg-size": 424, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 307200, "shared-mem": 1320, "physical-mem": 1612 } }, { "timestamp": "1682358892004", "keys": [ { "node-name": "0/1/CPU0" }, { "process-id": 1 } ], "content": { "name": "init", "jid": 1, "pid": 1, "text-seg-size": 296, "data-seg-size": 348, "stack-seg-size": 136, "malloc-size": 0, "dyn-limit": 4194303, "shared-mem": 1480, "physical-mem": 1944 } } ], "collection_end_time": "1682358892005" }

tylersiemers commented 1 year ago

Found the structure of the header its pulling apart but don't know how to capture it and verify if the producer of telemetry is sending non zero or not sending complete header.

image

srebhan commented 1 year ago

@tylersiemers it looks like your device does not send protocol-buffer encoded data. Can you please run the binary in PR #13337 once CI finished the tests successfully?! It will show the bad data when running Telegraf with --debug. Can you please post one of those samples (line contains incoming data:) so I can reproduce the issue locally?

srebhan commented 1 year ago

@tylersiemers can you please run the binary in #13337 as requested above!?

tylersiemers commented 1 year ago

@srebhan Yea I am out for the US holiday but have it on my calendar for upcoming Tuesday to run this and get some data.

tylersiemers commented 1 year ago

invalid_parse_13135.txt

Hex attached. Oddly enough I can only replicate this issue when using IPV6. If I switch the Cisco router to use IPV4 it doesn't do this.

srebhan commented 1 year ago

@tylersiemers it seems like your router uses some other encoding for the message above. Do you know if by chance the encoding is GPB K/V (GPB-compact)? There are plenty of protobuf definitions out there (e.g. https://github.com/ios-xr/model-driven-telemetry) and without having a clue on what to look for we will have a hard time decoding that message...

tylersiemers commented 1 year ago

encoding self-describing-gpb protocol tcp

GPB is the encoding.

srebhan commented 1 year ago

@tylersiemers can you please point me to the corresponding protobuf file? I tested a lot of different ones (even wrote a tool to brute-force test them) but couldn't find a matching one...

tylersiemers commented 1 year ago

https://github.com/ios-xr/model-driven-telemetry/tree/master/protos/754

These should be updated to the best of my knowledge. The hex I put above would tie to this

https://github.com/ios-xr/model-driven-telemetry/tree/master/protos/754/cisco_ios_xr_l2vpn_oper/l2vpn_forwarding/nodes/node/l2fibmac_details/l2fibmac_detail

srebhan commented 1 year ago

@tylersiemers sorry for the late reply but I only now found some more time to debug the protobuf issue... When doing a raw analysis of the data you can see that the data start out good image

Looking at the protobuf definition we see the following fields id name value ok
1 node ID CQ3CORPLBBF1 :heavy_check_mark:
3 subscription ID l2vpn-forwarding :heavy_check_mark:
6 encoding path Cisco-IOS-XR-l2vpn-oper:l2vpn-forwarding/nodes/node/l2fibmac-details/l2fibmac-detail :heavy_check_mark:
7 model version 2022-03-14 :heavy_check_mark:
8 collection ID 360834 :heavy_check_mark:
9 collection start time 1685466167987 (Tuesday, 30. May 2023 17:02:47.987 UTC) :heavy_check_mark:
10 message timestamp 1685466167987 (Tuesday, 30. May 2023 17:02:47.987 UTC) :heavy_check_mark:
11 telemetry field (GPBKV) some binary data :heavy_multiplication_x:
... ... ... :heavy_multiplication_x:

So what you can see is that everything up to field 11 is fine, but field 11 should contain a TelemetryField with "[...] payload data [...] encoded in the self-describing GPB-KV format". Looking at the binary data in 11 it seems like a valid TelemetryField to me, but none of the protobuf tools can decode it.

This being said, this issue is either in Cisco's way of defining the protocol-buffer message or a bug in the protocol-buffer implementation... Both being out of scope for Telegraf I fear. Do you have some contact in Cisco I can work with to find out what is going on?

telegraf-tiger[bot] commented 1 year ago

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Forums or provide additional details in this issue and reqeust that it be re-opened. Thank you!

doup123 commented 2 months ago

@powersj did you by any chance find any solution for this with Cisco?

powersj commented 2 months ago

@doup123, I wasn't involved in this issue, but reading through this it looks like it was due to bad data from a Cisco device or something up with the protobuf definition. I don't believe Sven or I have any direct Cisco contacts so not sure this went anywhere.

srebhan commented 2 months ago

@powersj and @doup123 I did not pursue this further as (as @powersj said) I don't have any contact in Cisco to further investigate the issue.

doup123 commented 2 months ago

@powersj and @srebhan, changing the subscription protocol from TCP to gRPC seems to solve the aforementioned issue.

srebhan commented 2 months ago

@doup123 thanks for letting us know! Do you want to add a short statement to plugin's readme?