Add ability to Merge/Stack Telegraf configurations in UI

[samhld]

Use case

Installing multiple templates from which each Telegraf configuration is needed in the same data source.

Example:

User has a K8s-centric infrastructure. They want to run Telegraf as a daemonset and want to use Templates to get started. They use Redis, Nginx, HAproxy, Kafka, MySQL, etc., and want to take advantage of the templated Telegraf configurations that exist for each of these services.

Problem:

Complete Telegraf configurations are in each Template. User's ability to combine these configurations is hindered.

Their options become:

• Running Telegraf with multiple local configurations
  • telegraf.d/ currently only works with local configs and the idea behind the Telegraf/2.0 duality is managing and obtaining configuration from a central InfluxDB repository. Even locally, this method shares the other issues.
• Running multiple Telegrafs on the same data source
  • IMO, this is clunky and will be an oft-rejected pattern for users
• Copy/pasting relevant diffs of multiple configs into one
  • Too manual and degrades the overall InfluxDB Templates messaging of time-to-awesome and one-command utility
• Copy/paste parts of multiple config files into one

Separate problem (addressed in proposed solution):

The most part of authoring a template is including even one Telegraf configuration. It requires manually entering boilerplate YAML and pasting your TOML into that section. From there, the author must make sure all lines are properly indented. I think it's safe to this isn't the experience we want for users of any piece of InfluxData software.

Proposal

Editable configurations in UI. Ultimately, the envisioned UX of this to have some kind of UI for moving configuration blocks between configuration files. This would mean the UI understands the components of the config files and exposes them as objects that can be moved, copied, picked from a dropdown or checked/selected in a list, etc.

The idea here is to benefit both the Template author and consumer personas. If the Template author can create and edit configuration in the UI, they can export (generate this portion of the template) right from their workspace.

Some possible additions to this (needs more discussion and validation)

• Templates repo enforces a lack of agent and output configuration. The UI would supply a default Agent for all configurations to share. This could be copied and then edited for custom Agent configurations( adding jitter, changing intervals and batching, enabling debug, etc.). The output plugin would also be provided/generated. We would then leave the option of using non-InfuxDB outputs to the template user if needed (outside of template).

Workflow change for author:

• Author can now create and modify configuration from the UI and export from there which skips a couple steps!
• Author is now encouraged to include a namepass field in any Processor/Aggregator plugins they use. This is because they will only expect the input data from the Template for which they were written. When combined with inputs from other Templates, we'd like to avoid the necessary overhead of each of those plugins dealing with more data than they need to.
• (only if Agent/Output exclusion is enforced) Do not include Agent or Output configurations

Workflow change for consumer:

• Instead of copy/pasting piecemealed configuration and

Alternatives proposed

1. Telegraf would support intelligently merging configurations based on diffs. There is an assumption that the Agent configuration blocks are
   • Deemed out of scope
2. Telegraf would support running with multiple remote configurations (like telegraf.d/ but for centrally hosted configs in, say Influx2)
   • Could have added a flag like --config-url to pull multiple configs.
   • This does not fully eliminate challenges with config overlap, conflicts, etc.

[wd3mo]

I would love to see this happening. The most important thing in my opinion is that the configurations must be editable from the UI.

[wd3mo]

In the meantime I found out that it is possible to create/update telegraf configurations via CLI.

[samhld]

@wd3mo good news is I believe you can now edit and save the config in the Cloud UI. I'll have to check OSS but you can try it out with a free Cloud account.

[sjwang90]

@wd3mo Check this portion of the video of editing the Telegraf config that's already in the UI https://youtu.be/f7ws9pfLmJc?t=96