Telegraf does not get omit_hostname setting from config directory

RealHousailei commented 2 years ago

Relevent telegraf.conf

[agent]
  omit_hostname = true

[[outputs.influxdb]]
  urls = ["http://localhost:8086"]
  database = "trades"

Logs from Telegraf

2022-01-06T03:21:01Z I! Starting Telegraf 1.21.1 2022-01-06T03:21:01Z I! Loaded inputs: influxdb_v2_listener 2022-01-06T03:21:01Z I! Loaded aggregators: 2022-01-06T03:21:01Z I! Loaded processors: 2022-01-06T03:21:01Z I! Loaded outputs: influxdb 2022-01-06T03:21:01Z I! Tags enabled: host=VM-3-32-centos 2022-01-06T03:21:01Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"VM-3-32-centos", Flush Interval:10s

System info

Telegraf 1.21.1, CentOS7

Docker

No response

Steps to reproduce

Set follow in telegraf.d/telegraf.conf .
```
[agent]
omit_hostname = true
```

[[outputs.influxdb]] urls = ["http://localhost:8086"] database = "trades"


2. Set follow content in `telegraf.conf`.

[[inputs.influxdb_v2_listener]] service_address = ":8080"


3. Run `telegraf --config telegraf.conf --config-directory telegraf.d`.

    Log :

2022-01-06T03:21:01Z I! Starting Telegraf 1.21.1 2022-01-06T03:21:01Z I! Loaded inputs: influxdb_v2_listener 2022-01-06T03:21:01Z I! Loaded aggregators: 2022-01-06T03:21:01Z I! Loaded processors: 2022-01-06T03:21:01Z I! Loaded outputs: influxdb 2022-01-06T03:21:01Z I! Tags enabled: host=VM-3-32-centos 2022-01-06T03:21:01Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"VM-3-32-centos", Flush Interval:10s 2022-01-06T03:21:01Z I! [inputs.influxdb_v2_listener] Started HTTP listener service on :8080 ...

Take attention to `Tags enabled: host=VM-3-32-centos`.
    It set the "host" tag in the telegraf agent.

4. If you add follow into `telegraf.conf`, it will run as expected.

[agent] omit_hostname = true

2022-01-06T03:35:25Z I! Starting Telegraf 1.21.1 2022-01-06T03:35:25Z I! Loaded inputs: influxdb_v2_listener 2022-01-06T03:35:25Z I! Loaded aggregators: 2022-01-06T03:35:25Z I! Loaded processors: 2022-01-06T03:35:25Z I! Loaded outputs: influxdb 2022-01-06T03:35:25Z I! Tags enabled: 2022-01-06T03:35:25Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"", Flush Interval:10s ...



### Expected behavior

Do no set the "host" tag in the telegraf agent.

### Actual behavior

Telegraf does not get `omit_hostname = true` from config directory, and set the "host" tag in the telegraf agent.

### Additional info

_No response_

popey commented 2 years ago

Thanks for the issue report. I've confirmed on 1.21.1 and 1.21.2. Do you know if this has ever worked correctly on a previous release of Telegraf, or is this a new deployment?

RealHousailei commented 2 years ago

I try it on 1.19.0 and 1.20.0, the same happened.

salarali commented 1 year ago

Is there any progress on this? I get the same behavior in the current v.1.27.4

powersj commented 1 year ago

Hi,

Telegraf loads each file individually. In this bug report, this means that the telegraf.conf file is loaded first and the plugins and settings in that file are loaded. Then the config directory is loaded and any files there are loaded. In a recent version we also added output to telegraf to tell you what file is loaded and when.

Once a plugin is loaded and running we do not go back and make any updates to it. This is why the [agent] section, which is read later and not first, has settings that do not apply to the plugin that was read first.

While we could combine all the files into one variable and load that entire variable all at once, we also lose the ability to tell users what line is causing an issue. I do not see us changing this ordering.

The workaround is to ensure that the agent settings are in the very first file so that they are read first and applied to all other plugins.

influxdata / telegraf