logstash-plugins / logstash-filter-mutate

Apache License 2.0
16 stars 75 forks source link

Using comments in the add_field section causes syntax error #137

Open Kvrepo opened 5 years ago

Kvrepo commented 5 years ago

In Logstash 6.3.0 I've noticed the following phenomenon:

I have configuration A and configuration B. I run them with the following command:

/usr/share/logstash/bin/logstash -f /home/user/A.cf --path.data="/srv"

/usr/share/logstash/bin/logstash -f /home/user/B.cf --path.data="/srv"

Configuration A:

input {
    stdin {}
}

filter {

    mutate {

        add_field => {
            # Comment 1
            "some_field_1" => "some value"

            # Comment 2
            "some_field_2" => "some value"
        }

    }

}

output {
    stdout {}
}

Configuration A yields the following error:

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-05-31 12:58:29.124 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-05-31 12:58:30.131 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.3.0"}
[ERROR] 2019-05-31 12:58:31.253 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 14, column 13 (byte 188) after filter {\n\n    mutate {\n\n        add_field => {\n            # Comment 1\n            \"some_field_1\" => \"some value\"\n            \n            # Comment 2\n            ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]}

Configuration B:

input {
    stdin {}
}

filter {

    mutate {

        add_field => {
            # Comment 1
            "some_field_1" => "some value"

            "some_field_2" => "some value"
        }

    }

}

output {
    stdout {}
}

Configuration B yields no errors:

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-05-31 12:59:49.383 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-05-31 12:59:50.726 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.3.0"}
[INFO ] 2019-05-31 12:59:59.327 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2019-05-31 12:59:59.594 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x395ce82 run>"}
The stdin plugin is now waiting for input:
[INFO ] 2019-05-31 12:59:59.730 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2019-05-31 13:00:00.123 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9603}
Test
{
      "@timestamp" => 2019-05-31T11:00:04.690Z,
    "some_field_1" => "some value",
    "some_field_2" => "some value",
         "message" => "Test",
        "@version" => "1",
            "host" => "some.host.com"
}

The only difference between the two configurations is this part:

A:

        add_field => {
            # Comment 1
            "some_field_1" => "some value"

            # Comment 2
            "some_field_2" => "some value"
        }

B:

        add_field => {
            # Comment 1
            "some_field_1" => "some value"

            "some_field_2" => "some value"
        }

Am I right to assume that you can only use comments at the "beginning" of an add_field section, but not in the middle? Is it a bug or a feature?

Of course the obvious solution to this issue is avoiding these kinds of comments, but I wanted to gather some plus information about this before moving onward.

Udate: We have examined the issue further and come to the conclusion that this issue is present in every plugin which has a hash as a parameter (split, translate etc.).

voilaviola commented 4 years ago

This problem persists in version 7.8.1 with rename plugin. The following configurations will trigger a syntax error :

mutate {
    rename => {
        "column1" => "field_name" # Comment
    }
}

And

mutate {
    rename => {
        "column1" => "field_name" 
        # Comment
    }
}