powersj
commented
1 month ago Stop Telegraf
How are you stopping telegraf?
How reproducible is this? Was this a one-time thing or can you easily reproduce this?
data_format = "grok"
Can you provide 1-2 example lines please?
watch_method = "poll"
This is generally only required on Windows fwiw. It shouldn't matter, but you might consider switching to inotify
2024-07-19T07:28:21Z D! [inputs.tail] Recording offset 1559760 for "/data/syslog/syslog.log"
The message I would expect after this is something like the following line, which may indicate some deadlock or something hanging:
2024-07-19T07:28:21Z D! [inputs.tail] Tail removed for "/data/syslog/syslog.log"
Unfortunately, I have not been able to reproduce locally when running from the CLI and using Ctrl-C to stop Telegraf:
[agent]
debug = true
omit_hostname = true
metric_batch_size = 2
[[inputs.tail]]
files = ["metrics"]
watch_method = "poll"
data_format = "influx"
[[outputs.http]]
url = "http://127.0.0.1:8000/"
data_format = "influx"
insecure_skip_verify = true❯ ./telegraf --config config.toml
2024-07-19T13:45:12Z I! Loading config: config.toml
2024-07-19T13:45:12Z I! Starting Telegraf 1.32.0-a6270b71 brought to you by InfluxData the makers of InfluxDB
2024-07-19T13:45:12Z I! Available plugins: 234 inputs, 9 aggregators, 32 processors, 26 parsers, 61 outputs, 6 secret-stores
2024-07-19T13:45:12Z I! Loaded inputs: tail
2024-07-19T13:45:12Z I! Loaded aggregators:
2024-07-19T13:45:12Z I! Loaded processors:
2024-07-19T13:45:12Z I! Loaded secretstores:
2024-07-19T13:45:12Z I! Loaded outputs: http
2024-07-19T13:45:12Z I! Tags enabled:
2024-07-19T13:45:12Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"", Flush Interval:10s
2024-07-19T13:45:12Z D! [agent] Initializing plugins
2024-07-19T13:45:12Z D! [agent] Connecting outputs
2024-07-19T13:45:12Z D! [agent] Attempting connection to [outputs.http]
2024-07-19T13:45:12Z D! [agent] Successfully connected to outputs.http
2024-07-19T13:45:12Z D! [agent] Starting service inputs
2024-07-19T13:45:12Z D! [inputs.tail] Tail added for "metrics"
2024-07-19T13:45:17Z D! [outputs.http] Wrote batch of 2 metrics in 842.758µs
2024-07-19T13:45:17Z D! [outputs.http] Buffer fullness: 8 / 10000 metrics
2024-07-19T13:45:17Z D! [outputs.http] Wrote batch of 2 metrics in 415.694µs
2024-07-19T13:45:17Z D! [outputs.http] Buffer fullness: 6 / 10000 metrics
2024-07-19T13:45:22Z D! [outputs.http] Buffer fullness: 6 / 10000 metrics
2024-07-19T13:45:22Z E! [agent] Error writing to outputs.http: Post "http://127.0.0.1:8000/": dial tcp 127.0.0.1:8000: connect: connection refused
2024-07-19T13:45:27Z D! [outputs.http] Buffer fullness: 16 / 10000 metrics
2024-07-19T13:45:27Z E! [agent] Error writing to outputs.http: Post "http://127.0.0.1:8000/": dial tcp 127.0.0.1:8000: connect: connection refused
2024-07-19T13:45:27Z D! [outputs.http] Buffer fullness: 16 / 10000 metrics
2024-07-19T13:45:27Z E! [agent] Error writing to outputs.http: Post "http://127.0.0.1:8000/": dial tcp 127.0.0.1:8000: connect: connection refused
^C2024-07-19T13:45:29Z D! [agent] Stopping service inputs
2024-07-19T13:45:29Z D! [inputs.tail] Recording offset 930 for "metrics"
2024-07-19T13:45:29Z D! [inputs.tail] Tail removed for "metrics"
2024-07-19T13:45:29Z D! [agent] Input channel closed
2024-07-19T13:45:29Z I! [agent] Hang on, flushing any cached metrics before shutdown
2024-07-19T13:45:29Z D! [outputs.http] Buffer fullness: 16 / 10000 metrics
2024-07-19T13:45:29Z E! [agent] Error writing to outputs.http: Post "http://127.0.0.1:8000/": dial tcp 127.0.0.1:8000: connect: connection refused
2024-07-19T13:45:29Z I! [agent] Stopping running outputs
2024-07-19T13:45:29Z D! [agent] Stopped Successfully
ts-wong
Relevant telegraf.conf
Logs from Telegraf
System info
Telegraf 1.31.1 (git: HEAD@99fcdc28), Linux
Docker
No response
Steps to reproduce
scripts/init.sh(specifically, my system usesstart-stop-daemon)scripts/init.sh(it sendsSIGTERMto the process)Expected behavior
Telegraf should exit.
Actual behavior
Telegraf will not exit.
Additional info
No response