Add option to ignore a specific filter plugin

[endersonmaia]

I have a situation where I use input-jdbc to get some events, and than I use filter-jdbc_streaming to enhance the message.

In this case I'd like to test the filter without the need to connect to the database.

One option would be to ignore a plugin used in the filter, and I would provide the event already with the filter-jdbc_streaming enhancements.

[magnusbaeck]

This would require LFV to be able to parse Logstash configuration files which is quite a bit of work. Wouldn't it be a reasonable workaround to just adapt your filenames and/or filename patterns to avoid including the configuration file containing the jdbc_streaming filter? For example, I invoke LFV with *-filter.conf so that LFV only picks up my configuration files with filters and avoid those with inputs and outputs. Faced with your problem I'd perhaps call the file with the jdbc_streaming filter something-externalfilter.conf and write test case files that feed Logstash with pre-populated events corresponding to what the jdbc_streaming filter would've done.

[endersonmaia]

Wouldn't it be a reasonable workaround to just adapt your filenames and/or filename patterns to avoid including the configuration file containing the jdbc_streaming filter?

That's what I ended up doing.

[breml]

I am just hitting the exact same problem with the logstash-filter-elasticsearch plugin. Some days ago, I updated https://github.com/breml/logstash-config to better support altering of parsed Logstash configurations (before, some important fields have been private, which made it in fact impossible to update an existing config).

If we can come up with a proposal on how we would config this in LFV, then this could be added now quite easily.

[magnusbaeck]

How about a testcase file configuration option that contains a list of plugin ids (i.e. the optional id option of a plugin entry in a Logstash configuration file, not the name of the plugin) that should be ignored?

[breml]

A quick and dirty version of the code to remove filters by ID:

import (
    config "github.com/breml/logstash-config"
    "github.com/breml/logstash-config/ast"
)

func removeFiltersByID(f *os.File, filename string, ids ...string) error {
    icfg, err := config.ParseReader(filename, f)
    if err != nil {
        return err
    }
    cfg := icfg.(ast.Config)

    for i := 0; i < len(cfg.Filter); i++ {
    plugin:
        for j := 0; j < len(cfg.Filter[i].BranchOrPlugins); j++ {
            plugin, ok := cfg.Filter[i].BranchOrPlugins[j].(ast.Plugin)
            if !ok {
                continue
            }
            for _, attr := range plugin.Attributes {
                if attr.Name() == "id" {
                    sattr := attr.(ast.StringAttribute)
                    for _, id := range ids {
                        if sattr.Value() == id {
                            cfg.Filter[i].BranchOrPlugins = append(cfg.Filter[i].BranchOrPlugins[:j], cfg.Filter[i].BranchOrPlugins[j+1:]...)
                            j--
                            continue plugin
                        }
                    }
                }
            }
        }
    }
    fmt.Println(cfg)
    return nil
}