SIGHUP not discovering files in --config-directory

[gazpachoking]

As of telegraf 1.25.2, sending a SIGHUP no longer discovers new config files that were placed into a --config-directory. On 1.25.1 when adding files to this folder, then sending a SIGHUP, they were properly discovered and loaded.

Relevant telegraf.conf

Logs from Telegraf

Broken behavior from 1.25.2

2023-02-27T18:26:37Z I! Starting Telegraf 1.25.2
2023-02-27T18:26:37.075069732Z 2023-02-27T18:26:37Z I! Available plugins: 228 inputs, 9 aggregators, 26 processors, 21 parsers, 57 outputs, 2 secret-stores
2023-02-27T18:26:37.075074688Z 2023-02-27T18:26:37Z I! Loaded inputs: http mock
2023-02-27T18:26:37.075101233Z 2023-02-27T18:26:37Z I! Loaded aggregators: 
2023-02-27T18:26:37.075104782Z 2023-02-27T18:26:37Z I! Loaded processors: starlark (2x)
2023-02-27T18:26:37.075107133Z 2023-02-27T18:26:37Z I! Loaded secretstores: 
2023-02-27T18:26:37.075109410Z 2023-02-27T18:26:37Z I! Loaded outputs: influxdb_v2
2023-02-27T18:26:37.075111809Z 2023-02-27T18:26:37Z I! Tags enabled: 
2023-02-27T18:26:37.075114162Z 2023-02-27T18:26:37Z I! [agent] Config: Interval:30s, Quiet:false, Hostname:"", Flush Interval:10s
2023-02-27T18:27:04.058793132Z 2023-02-27T18:27:04Z I! Reloading Telegraf config
2023-02-27T18:27:04.058867649Z 2023-02-27T18:27:04Z I! [agent] Hang on, flushing any cached metrics before shutdown
2023-02-27T18:27:04.065975665Z 2023-02-27T18:27:04Z I! [agent] Stopping running outputs
2023-02-27T18:27:04.066180622Z 2023-02-27T18:27:04Z I! Starting Telegraf 1.25.2
2023-02-27T18:27:04.066211792Z 2023-02-27T18:27:04Z I! Available plugins: 228 inputs, 9 aggregators, 26 processors, 21 parsers, 57 outputs, 2 secret-stores
2023-02-27T18:27:04.066228407Z 2023-02-27T18:27:04Z I! Loaded inputs: http mock
2023-02-27T18:27:04.066233060Z 2023-02-27T18:27:04Z I! Loaded aggregators: 
2023-02-27T18:27:04.066240361Z 2023-02-27T18:27:04Z I! Loaded processors: starlark (2x)
2023-02-27T18:27:04.066243236Z 2023-02-27T18:27:04Z I! Loaded secretstores: 
2023-02-27T18:27:04.066246177Z 2023-02-27T18:27:04Z I! Loaded outputs: influxdb_v2
2023-02-27T18:27:04.066249207Z 2023-02-27T18:27:04Z I! Tags enabled: 
2023-02-27T18:27:04.066251827Z 2023-02-27T18:27:04Z I! [agent] Config: Interval:30s, Quiet:false, Hostname:"", Flush Interval:10s

Working properly on 1.25.1. Notice how "loaded inputs" changes here, but not above.

023-02-27T18:22:14Z I! Starting Telegraf 1.25.1
2023-02-27T18:22:14.776850274Z 2023-02-27T18:22:14Z I! Available plugins: 228 inputs, 9 aggregators, 26 processors, 21 parsers, 57 outputs, 2 secret-stores
2023-02-27T18:22:14.776860560Z 2023-02-27T18:22:14Z I! Loaded inputs: http mock
2023-02-27T18:22:14.776862765Z 2023-02-27T18:22:14Z I! Loaded aggregators: 
2023-02-27T18:22:14.776864618Z 2023-02-27T18:22:14Z I! Loaded processors: starlark (2x)
2023-02-27T18:22:14.776866347Z 2023-02-27T18:22:14Z I! Loaded secretstores: 
2023-02-27T18:22:14.776868702Z 2023-02-27T18:22:14Z I! Loaded outputs: influxdb_v2
2023-02-27T18:22:14.776870575Z 2023-02-27T18:22:14Z I! Tags enabled: 
2023-02-27T18:22:14.776872309Z 2023-02-27T18:22:14Z I! [agent] Config: Interval:30s, Quiet:false, Hostname:"", Flush Interval:10s
2023-02-27T18:22:39.174749132Z 2023-02-27T18:22:39Z I! Reloading Telegraf config
2023-02-27T18:22:39.174864062Z 2023-02-27T18:22:39Z I! [agent] Hang on, flushing any cached metrics before shutdown
2023-02-27T18:22:39.174946077Z 2023-02-27T18:22:39Z I! [agent] Stopping running outputs
2023-02-27T18:22:39.253110147Z 2023-02-27T18:22:39Z I! Starting Telegraf 1.25.1
2023-02-27T18:22:39.253167137Z 2023-02-27T18:22:39Z I! Available plugins: 228 inputs, 9 aggregators, 26 processors, 21 parsers, 57 outputs, 2 secret-stores
2023-02-27T18:22:39.253173173Z 2023-02-27T18:22:39Z I! Loaded inputs: http (5x) mock snmp (76x)
2023-02-27T18:22:39.253176043Z 2023-02-27T18:22:39Z I! Loaded aggregators: 
2023-02-27T18:22:39.253179562Z 2023-02-27T18:22:39Z I! Loaded processors: starlark (54x)
2023-02-27T18:22:39.253181968Z 2023-02-27T18:22:39Z I! Loaded secretstores: 
2023-02-27T18:22:39.253184705Z 2023-02-27T18:22:39Z I! Loaded outputs: influxdb_v2
2023-02-27T18:22:39.253188072Z 2023-02-27T18:22:39Z I! Tags enabled: 
2023-02-27T18:22:39.253191068Z 2023-02-27T18:22:39Z I! [agent] Config: Interval:30s, Quiet:false, Hostname:"", Flush Interval:10s

System info

Telegraf 1.25.2, Docker

Docker

No response

Steps to reproduce

1. Start telegraf with --config-directory 2 Add config files to the config directory
2. Send SIGHUP
3. Notice that only the files that were originally in the config directory got reloaded. ...

Expected behavior

All files in the config directory get loaded.

Actual behavior

Only files that existed when telegraf starts up were reloaded.

Additional info

No response

[gazpachoking]

Thinking this might be related to #12607

[powersj]

You are right, and it was reported in https://github.com/influxdata/telegraf/issues/12718 and fixed in https://github.com/influxdata/telegraf/pull/12721. That fix will go out today in v1.25.3 shortly.

[gazpachoking]

Oops, didn't see that one. Thank you!

[shriramb380]

is this fix https://github.com/influxdata/telegraf/pull/12607 going out in 1.25.3

[powersj]

You can always go look at the change log: https://github.com/influxdata/telegraf/blob/master/CHANGELOG.md

12607 went in v1.25.2

12721, which fixes this issue, went in v1.25.3

[shriramb380]

@powersj Thanks, do we know when will 1.25.3 be released or if there is a nightly build which will we can provide to SCB?

[powersj]

1.25.3 was released yesterday https://github.com/influxdata/telegraf/releases

[shriramb380]

Thanks a lot @powersj was not able to see in releases.