Bug in custom_builder scanning of parsers

daniel-falk commented 3 months ago

Relevant telegraf.conf

[[inputs.mqtt_consumer]]
  name_override = "qr_mqtt_message"
  servers = ["tcp://mosquitto:1883"]
  topics = [
    "<REDACTED>"
  ]

  qos = 2
  persistent_session = false
  client_id = "telegraf_qr_code"

  data_format = "json_v2"

  [[inputs.mqtt_consumer.json_v2]]
    [[inputs.mqtt_consumer.json_v2.object]]
      path = "message.data"
      tags = ["data"]

[[inputs.mqtt_consumer]]
  name_override = "raw_mqtt_message"
  servers = ["tcp://mosquitto:1883"]

  # Capture the content as a string since we do not know the format of it...
  data_format = "value"
  data_type = "string"

  # Capture all topics and store the topic as a tag with name "topic"...
  topics = ["#"]
  topic_tag = "topic"

  qos = 2
  persistent_session = false
  client_id = "telegraf_generic"

[[outputs.influxdb_v2]]
  urls = ["http://influxdb:8086"]
  token = "${INFLUX_TOKEN}"
  organization = "test"
  bucket = "test_bucket"

Logs from Telegraf

System info

Linux danielfa-ThinkPad-T590 6.5.0-41-generic #41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun 3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Docker

No response

Steps to reproduce

Clone the telegraf repo and checkout master or v.1.31.1
Run make build_tools
Run ./tools/custom_builder/custom_builder --config <path-to-config>
Run ./telegraf --config /tmp/t.conf

Expected behavior

Telegraf should be built with the correct plugins and run.

Actual behavior

One of the parsers are not built / found be the custom_builder. This seems to be since there are two inputs using the same plugin, and the scanner seems to only use one of them.

When running the custom build, only the parser from the second usage of the plugin is captured:

./tools/custom_builder/custom_builder --config /tmp/t.conf 
2024/07/16 11:55:21 Importing configuration file(s)...
2024/07/16 11:55:21 Found 1 configuration files...
-------------------------------------------------------------------------------
Enabled plugins:
-------------------------------------------------------------------------------
aggregators (0):
-------------------------------------------------------------------------------
inputs (1):
  mqtt_consumer                   plugins/inputs/mqtt_consumer
-------------------------------------------------------------------------------
outputs (1):
  influxdb_v2                     plugins/outputs/influxdb_v2
-------------------------------------------------------------------------------
parsers (1):
  value                           plugins/parsers/value
-------------------------------------------------------------------------------
processors (0):
-------------------------------------------------------------------------------
secretstores (0):
-------------------------------------------------------------------------------
serializers (0):
-------------------------------------------------------------------------------
2024/07/16 11:55:21 Running build...
CGO_ENABLED=0 go build -tags "custom,inputs.mqtt_consumer,outputs.influxdb_v2,parsers.value" -ldflags " -X github.com/influxdata/telegraf/internal.Commit=99fcdc28 -X github.com/influxdata/telegraf/internal.Branch=HEAD -X github.com/influxdata/telegraf/internal.Version=1.31.1" ./cmd/telegraf

Note that json_v2 parser was not captured.

When running telegraf using the same config we get an error:

2024-07-16T09:56:42Z I! Loading config: /tmp/t.conf
2024-07-16T09:56:42Z E! error loading config file /tmp/t.conf: error parsing mqtt_consumer, adding parser failed: undefined but requested parser: json_v2

If I split the config file into two config files where each one is using one of the mqtt inputs, then both parsers are correctly detected:

./tools/custom_builder/custom_builder --config /tmp/t.conf --config /tmp/t2.conf 
2024/07/16 11:59:32 Importing configuration file(s)...
2024/07/16 11:59:32 Found 2 configuration files...
-------------------------------------------------------------------------------
Enabled plugins:
-------------------------------------------------------------------------------
aggregators (0):
-------------------------------------------------------------------------------
inputs (1):
  mqtt_consumer                   plugins/inputs/mqtt_consumer
-------------------------------------------------------------------------------
outputs (1):
  influxdb_v2                     plugins/outputs/influxdb_v2
-------------------------------------------------------------------------------
parsers (2):
  json_v2                         plugins/parsers/json_v2
  value                           plugins/parsers/value
-------------------------------------------------------------------------------
processors (0):
-------------------------------------------------------------------------------
secretstores (0):
-------------------------------------------------------------------------------
serializers (0):
-------------------------------------------------------------------------------
2024/07/16 11:59:32 Running build...
CGO_ENABLED=0 go build -tags "custom,inputs.mqtt_consumer,outputs.influxdb_v2,parsers.json_v2,parsers.value" -ldflags " -X github.com/influxdata/telegraf/internal.Commit=99fcdc28 -X github.com/influxdata/telegraf/internal.Branch=HEAD -X github.com/influxdata/telegraf/internal.Version=1.31.1" ./cmd/telegraf

Additional info

No response

srebhan commented 3 months ago

@daniel-falk please test the binary in PR #15630, available as soon as CI finished the tests, and let me know if this fixes the issue!

daniel-falk commented 3 months ago

@srebhan Thanks, that fixed it! :rocket:

Tested OK on:

commit c7bb5b40067e2018070bff1cb470517874451192 (HEAD -> custom_builder_issue_15627)
Author: Sven Rebhan <srebhan@influxdata.com>
Date:   Tue Jul 16 18:26:46 2024 +0200

    fix(tools.custom_builder): Handle multiple instance of the same plugin correctly

influxdata / telegraf