Filebeat: External configuration files for outputs

[getkub]

Describe the enhancement: We are trying to modularize all of the configurations as a standard. As per the https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-configuration-reloading.html link . I couldn't find a precise way of doing this for outputs for filebeat, so my guess work was to

# Put the below config into auditbeat.yml
filebeat.config.outputs:
  enabled: true
  path: ouputs.d/*.yml

And created an ouputs.d directory and put a my_ouputs.yml file in outputs.d. But this doesn't work. As per community,

Currently Filebeat only allows external configuration files for modules and inputs. Moving your output configuration to an external configuration file is not supported.

Describe a specific use case for the enhancement or feature: As per the details in the community forum, https://discuss.elastic.co/t/how-to-load-external-configuration-file/228757 an enhancement request would be truly useful to modularize the config files to send to various destinations.

Filebeat (or really any Beat) can only have a single output. By contrast, Filebeat can have multiple inputs and/or modules. So moving those to external configuration files might make more sense than for a single output. But this is more just an opinion; I think if you can make a strong enough case for this in your enhancement request issue and there's interest from other users in the community as well, we'd definitely prioritize it!

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[ycombinator]

Hi @getkub, we discussed this within the team today. Modularizing output configurations in the same way as modules or Filebeat inputs is not something we plan to add. This goes back to the idea that while there can be multiple modules and inputs, there can be only one output.

Beats, however, do allow users to specify multiple configuration files via the CLI. Perhaps you could use this to define the output configuration separately in it's own configuration file? For example:

/filebeat -c filebeat.main.yml -c filebeat.out.yml

[getkub]

Hi @getkub, we discussed this within the team today. Modularizing output configurations in the same way as modules or Filebeat inputs is not something we plan to add. This goes back to the idea that while there can be multiple modules and inputs, there can be only one output.

Beats, however, do allow users to specify multiple configuration files via the CLI. Perhaps you could use this to define the output configuration separately in it's own configuration file? For example:

/filebeat -c filebeat.main.yml -c filebeat.out.yml

Thanks for the update. I will let you know a scenario in Enterprise organisations where it is required. (I come from large Splunk estate and hence we and many customers use this model). So you want to write/automate modular apps/configurations and needs to push out certain configurations to certain clients.

For example We need configurations to collect

• ONLY Apache logs from "web-servers" + normal OS logs
• ONLY application logs from "payment servers" + normal OS logs

In this case, what we do is we create "modular apps" for web-servers, os_logs , payment-servers and based on the pattern of what server, we send combination of such "modular apps" and output it to relevant site/cluster or S3 etc. So the configurations are all in individual files (i.e modular) and depending on the client where it goes, all are loaded based on pattern. This way automation and isolation of configurations are achieved at same time.

This is lacking when you want to embed all the outputs into single yml (or have to automate into single yml file which makes it confusing)

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[archon810]

The current refusal to implement is disappointing. For example, for a config that only differs in a few variables (say between dev, staging, and prod), we right now have to duplicate the whole file. Most other software allows includes to have the main config and only different bits in other includes (apache, nginx, mysql come to mind, for example).

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[getkub]

We still need this feature in filebeat or elastic agents, so the modules can be configured externally

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

[archon810]

Bump.