Test single filter in daemon mode?

nicolasreich commented 3 years ago

Description

With LFV 1.x and with LFV 2.0 in standalone mode, it's possible to test a single filter, without input or output. If I have those files:

mutate.conf

filter {
  mutate { 
    rename => { "before" => "after" }
  }
}

mutate.test.yaml

---
codec: json_lines
ignore:
  - "@timestamp"
  - host
testcases:
  - input:
      - >
        {
          "before": "hello"
        }
    expected:
      - after: "hello"

I can run logstash-filter-verifier standalone mutate.test.yaml mutate.conf and the test runs fine. I haven't found a way to do this simply with the daemon mode.

Motivation

If a pipeline is made of several files, each containing one or more files, I'd like to be able to test them independently. Moreover, I'd like to be able to do this in daemon mode to spare some time.

A simple workaround would be to have a dummy input and output in a folder, a pipeline.yml file pointing to this folder, and copy each file to test in this folder one after the other. It would work fine, but it's a bit reimplementing something that's already in LFV.

Possible Drawbacks

In case it's not already implemented and I just missed it, it might be difficult to implement; and I do realize that now the priority is LFV 2.0 with its current set of features, not extending it, but I wanted to share this before forgetting.

breml commented 3 years ago

@nicolasreich Currently, there is no way to achieve this in daemon mode. We have a flag --logstash-config, which accepts a pure logstash config (without pipeline.yml), but this is not enough to achieve what you are asking for.

To make this work, LFV would need to:

detect, that the provided config does not contain input and output plugins
detect, that the test case does not state an input plugin
Implicitly add the missing input and output plugins
Implicitly add the name of the implicit input plugin as input_plugin in the test case
Generate the implicit pipelines.yml (this we already have)

It is certainly doable, but I am not yet sure, what the best way for the implementation would be. We could add more flags and logic to the existing daemon run command or we could create a new command for this specific use case (that is, a command, that behaves similar to v1/standalone, but uses the daemon for the execution of the test).

nicolasreich commented 3 years ago

@breml thanks for the feedback, I now understand the implications better.

Is the underlying issue basically that v1/standalone removes all inputs and outputs anyway, while daemon mode keeps them and runs the tests with them?

I'm under the impression that having daemon mode automatically detect whether there is a full pipeline or just a filter would add quite some complexity and potential for bugs. On the other hand, adding a flag, maybe to be used in conjunction with --logstash-config, such as --filters-only or whatever, could be a reasonably straightforward way to implement this: when this flag is present, LFV would remove all inputs/outputs, and do the 3 last points of your bullet list.

For now, we'll implement the workaround I mentioned in the first message, I think it should be much easier to implement than adding the feature to LFV, and should work just fine.

And again, thanks to you and @magnusbaeck for all the amazing work on LFV!

magnusbaeck / logstash-filter-verifier