search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.17k stars 4.92k forks source link

Support processors in the CheckPoint module #25613

Closed hazcod closed 3 years ago

hazcod commented 3 years ago

Describe the enhancement: The checkpoint module in x-pack currently does not support adding additional processors.

Describe a specific use case for the enhancement or feature: This does not allow us to drop certain events for example.

botelastic[bot] commented 3 years ago

This issue doesn't have a Team:<team> label.

fredtj commented 3 years ago

to add processors you could modify the ingest pipeline in elastic.

jsoriano commented 3 years ago

@hazcod could you give more details about the issue you found? what was the used configuration?

In principle it should be possible to use processors at the input level, on any module, or in the main configuration file.

hazcod commented 3 years ago

Some modules expose a var.processors variable, but I also see e.g.

- module: checkpoint
  input:
    processors: ...
jsoriano commented 3 years ago

@hazcod processors may be configured only in the main configuration file, or at the input level (see processors docs), not at the module level.

But overriding the processors in a module input is a bit tricky, specially for modules that already define their own processors, and don't include additional options for them.

Look at this topic in discuss for a similar conversation: https://discuss.elastic.co/t/system-module-error-when-defining-syslog-input-processors/221865

You can try to define them under the firewall fileset, like this:

- module: checkpoint
  firewall:
    input:
      processors: ...

But you would need to take into account the processors already defined in the module, what can be error-prone, and unreliable after upgrades. I would suggest you to define them in the main configuration file, with conditions to match only this module.

I don't think that something like processors at the module level can be supported, because this could conflict with existing configurations. There is an open issue about trying to improve this in any case: https://github.com/elastic/beats/issues/18925

I am going to close this issue because I think that there are alternatives to do what you want to do, and there is an issue already open to try to find better solutions.

Please open a topic in https://discuss.elastic.co/ for further questions.