search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.14k stars 5.52k forks source link

Event Hub output plugin does not reconnect after a link is closed because of transient credit issues. #14162

Open NuMove-JonathanSchmidt opened 8 months ago

NuMove-JonathanSchmidt commented 8 months ago

Relevant telegraf.conf

[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = "0s"
quiet = false
logtarget = "file"
logfile = 'c:\TIGStack\Logs\Telegraf\secondary.log'
logfile_rotation_interval = "24h"
logfile_rotation_max_size = "100MB"
logfile_rotation_max_archives = 5

[[inputs.influxdb_v2_listener]]
service_address = "127.0.0.1:47596"
parser_type = "upstream"

[[outputs.event_hubs]]
connection_string = "$TELEGRAF_APPLIANCE_CONN_STRING"
data_format = "influx"
namedrop = ["Log"]

[[outputs.event_hubs]]
connection_string = "$TELEGRAF_LOG_CONN_STRING"
data_format = "influx"
namepass = ["Log"]

Logs from Telegraf

2023-10-19T21:48:18Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:48:28Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:48:38Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:48:48Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:48:58Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:49:08Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:49:18Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:49:28Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:49:38Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:49:48Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:49:58Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:50:08Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:50:18Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:50:28Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:50:38Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:50:48Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:50:58Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:51:09Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:51:19Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:51:29Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:51:39Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:51:49Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:51:59Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:52:09Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:52:19Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:52:29Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:52:39Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:52:49Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:52:59Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:53:09Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:53:19Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:53:29Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:53:39Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:53:49Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:53:59Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:54:09Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:54:19Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:54:29Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:54:39Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:54:49Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:54:59Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:55:09Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:55:19Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:55:29Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:55:39Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:55:49Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:55:59Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:56:09Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:56:19Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:56:29Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:56:39Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:56:49Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:56:59Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:57:09Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:57:19Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:57:29Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:57:39Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:57:49Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:57:59Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:58:09Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:58:19Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:58:29Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:58:39Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:58:49Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:58:59Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:59:09Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:59:19Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:59:29Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T21:59:39Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:59:49Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T21:59:59Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:00:09Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:00:19Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:00:29Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:00:39Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:00:49Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:00:59Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:01:09Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:01:19Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:01:29Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:01:39Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:01:49Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:01:59Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:02:09Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:02:19Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:02:29Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:02:39Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:02:49Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:02:59Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:03:09Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:03:19Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:03:29Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:03:39Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:03:49Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:03:59Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:04:09Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:04:19Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:04:29Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:04:39Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:04:49Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:04:59Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:05:09Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:05:19Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:05:29Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:05:29Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:05:39Z W! [outputs.event_hubs] Metric buffer overflow; 299 metrics have been dropped
2023-10-19T22:05:49Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:05:59Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:06:09Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:06:19Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:06:29Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:06:39Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:06:49Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:06:59Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:07:09Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:07:19Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:07:29Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:07:39Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:07:50Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:08:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:08:10Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:08:20Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:08:30Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:08:40Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:08:50Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:09:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:09:10Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:09:20Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:09:30Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:09:40Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:09:50Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:10:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:10:10Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:10:20Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:10:30Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:10:40Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:10:50Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:11:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:11:10Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:11:20Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:11:30Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:11:40Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:11:50Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:12:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:12:10Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:12:20Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:12:30Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:12:40Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:12:50Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:13:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:13:10Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:13:10Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:13:20Z W! [outputs.event_hubs] Metric buffer overflow; 217 metrics have been dropped
2023-10-19T22:13:30Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:13:40Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:13:50Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:14:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:14:10Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:14:20Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:14:30Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:14:40Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:14:50Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:15:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:15:10Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:15:20Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:15:20Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:15:30Z W! [outputs.event_hubs] Metric buffer overflow; 231 metrics have been dropped
2023-10-19T22:15:40Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:15:50Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:16:00Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:16:10Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:16:20Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:16:30Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:16:30Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:16:40Z W! [outputs.event_hubs] Metric buffer overflow; 304 metrics have been dropped
2023-10-19T22:16:50Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:17:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:17:10Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:17:20Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:17:30Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:17:40Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:17:40Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:17:50Z W! [outputs.event_hubs] Metric buffer overflow; 221 metrics have been dropped
2023-10-19T22:18:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:18:10Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:18:20Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:18:30Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:18:40Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:18:50Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:19:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:19:10Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:19:20Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:19:30Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:19:40Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:19:50Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:20:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:20:10Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:20:20Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:20:20Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:20:30Z W! [outputs.event_hubs] Metric buffer overflow; 226 metrics have been dropped
2023-10-19T22:20:40Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:20:50Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:21:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:21:00Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:21:10Z W! [outputs.event_hubs] Metric buffer overflow; 225 metrics have been dropped
2023-10-19T22:21:20Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:21:30Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:21:40Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:21:50Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:22:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:22:10Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:22:20Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:22:30Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:22:40Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:22:40Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:22:50Z W! [outputs.event_hubs] Metric buffer overflow; 220 metrics have been dropped
2023-10-19T22:23:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:23:10Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:23:20Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:23:30Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:23:40Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:23:50Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:24:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:24:10Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:24:20Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:24:30Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:24:40Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:24:50Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:24:50Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:25:00Z W! [outputs.event_hubs] Metric buffer overflow; 217 metrics have been dropped
2023-10-19T22:25:10Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:25:20Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:25:30Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:25:40Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:25:50Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:26:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:26:00Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:26:10Z W! [outputs.event_hubs] Metric buffer overflow; 217 metrics have been dropped
2023-10-19T22:26:20Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:26:30Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:26:40Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:26:50Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:27:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:27:10Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:27:20Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:27:30Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:27:40Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:27:40Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:27:50Z W! [outputs.event_hubs] Metric buffer overflow; 219 metrics have been dropped
2023-10-19T22:28:00Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:28:10Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:28:20Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:28:20Z W! [outputs.event_hubs] Metric buffer overflow; 6 metrics have been dropped
2023-10-19T22:28:30Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:28:40Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:28:50Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:28:50Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:29:00Z W! [outputs.event_hubs] Metric buffer overflow; 217 metrics have been dropped
2023-10-19T22:29:10Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:29:20Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:29:30Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:29:40Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:29:50Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:30:00Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:30:11Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:30:21Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:30:31Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:30:41Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:30:51Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:31:01Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:31:11Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:31:21Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:31:31Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:31:31Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:31:41Z W! [outputs.event_hubs] Metric buffer overflow; 295 metrics have been dropped
2023-10-19T22:31:51Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:32:01Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:32:11Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:32:21Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:32:31Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:32:41Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:32:51Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:33:01Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:33:11Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:33:21Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:33:31Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:33:41Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:33:51Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:34:01Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:34:11Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:34:21Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:34:31Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:34:41Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:34:51Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:35:01Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:35:11Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:35:21Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:35:31Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:35:41Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:35:51Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:36:01Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:36:11Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:36:21Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:36:31Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:36:41Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:36:51Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:37:01Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:37:11Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:37:21Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:37:31Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:37:41Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:37:41Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:37:51Z W! [outputs.event_hubs] Metric buffer overflow; 217 metrics have been dropped
2023-10-19T22:38:01Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:38:11Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:38:21Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:38:31Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:38:41Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:38:51Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:39:01Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:39:11Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:39:21Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:39:31Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:39:41Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:39:51Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:40:01Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:40:11Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:40:21Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:40:31Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:40:41Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:40:51Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:41:01Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:41:11Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:41:21Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:41:31Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:41:41Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:41:51Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:42:01Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:42:11Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:42:21Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:42:31Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:42:41Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:42:51Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:43:01Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:43:11Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:43:21Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:43:31Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:43:41Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:43:51Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:44:01Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:44:11Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:44:21Z E! [agent] Error writing to outputs.event_hubs: *Error{Condition: amqp:link:transfer-limit-exceeded, Description: credit limit exceeded for sending link [Redacted], Info: map[]}
2023-10-19T22:44:31Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval
2023-10-19T22:44:34Z I! [agent] Hang on, flushing any cached metrics before shutdown
2023-10-19T22:44:42Z I! Starting Telegraf 1.29.0-34ee5106 brought to you by InfluxData the makers of InfluxDB
2023-10-19T22:44:42Z I! Available plugins: 240 inputs, 9 aggregators, 29 processors, 24 parsers, 60 outputs, 5 secret-stores
2023-10-19T22:44:42Z I! Loaded inputs: influxdb_v2_listener
2023-10-19T22:44:42Z I! Loaded aggregators: 
2023-10-19T22:44:42Z I! Loaded processors: 
2023-10-19T22:44:42Z I! Loaded secretstores: 
2023-10-19T22:44:42Z I! Loaded outputs: event_hubs (2x)
2023-10-19T22:44:42Z I! Tags enabled: host=nulogik-pnp2-dcmvgn
2023-10-19T22:44:42Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"nulogik-pnp2-dcmvgn", Flush Interval:10s
2023-10-19T22:44:42Z I! [inputs.influxdb_v2_listener] Started HTTP listener service on 127.0.0.1:47596
2023-10-20T07:06:46Z W! [outputs.event_hubs] Link is closed, reconnecting and retrying
2023-10-20T09:30:23Z W! [outputs.event_hubs] Link is closed, reconnecting and retrying
2023-10-23T16:40:14Z W! [agent] ["outputs.event_hubs"] did not complete within its flush interval

System info

Telegraf 1.29.0~34ee5106, Windows 11 Pro 22H2, NSSM 2.24

Docker

Telegraf is run through NSSM v2.24

Steps to reproduce

Seems like a transient error on EventHub. Eleven telegraf are all publishing simultaneously on the same queue and only one had the issue.

Expected behavior

After a write error the amqp link should be closed and re-opened

Actual behavior

The link stays up and following flush always return the same error.

Additional info

No response

powersj commented 8 months ago

next steps: Migrate away from the azure-event-hubs-go library and use the azeventhubs library and verify if the error condition still exists.