remove_field not working

[Haitianisgood]

ENV：

logstash 6.2 elasticsearch 6.2

my config：

input {
  kafka {
        bootstrap_servers => "kafka0:19092,kafka1:19093,kafka2:19094"
        topics => ["app1","app2"]
        codec => "json"
        group_id => "Test"
        consumer_threads => 2
  }
}

filter {
  grok {
       remove_field => ["beat.name","beat.version","@version","@timestamp"]
  }

}

output {
  elasticsearch {
    codec => plain{ charset => "UTF-8" }
    hosts => "http://es1:9200"
    index => "%{[fields][log_topic]}-%{+YYYY.MM.dd}"
  }
}

I want to remove filed ""beat.name","beat.version","@version","@timestamp"" Why it is not working?

[jsvd]

if you only need to remove fields, you don't need grok, use the mutate filter instead.

Operations like add_field, remove_field, add_tag etc, only work when it's considered the filter was applied successfully. For grok, that means matching the grok pattern to the event data. Since none is specified (this plugin should raise an error in this situation, but it's a different issue), then it's not considered a successful match. Because of this, the event is tagged as _grokparsefailure and operations like remove_field aren't executed:

% bin/logstash -e 'filter { grok { remove_field => ["beat.name","beat.version","@version","@timestamp"] } }' 
[2018-06-15T11:10:47,434][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x22e115cc run>"}
The stdin plugin is now waiting for input:
tete
{
          "host" => "Joaos-MacBook-Pro-5.local",
       "message" => "tete",
          "tags" => [
        [0] "_grokparsefailure"
    ],
          "type" => "stdin",
      "@version" => "1",
    "@timestamp" => 2018-06-15T10:10:53.882Z
}

But with the right match:

% bin/logstash -e 'filter { grok { match => { "message" => "%{NUMBER}" } remove_field => ["beat.name","beat.version","@version","@timestamp"] } }'
[2018-06-15T11:16:52,966][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x344feedf run>"}
The stdin plugin is now waiting for input:
1
{
       "host" => "Joaos-MacBook-Pro-5.local",
    "message" => "1",
       "type" => "stdin"
}

you can see remove_field works

[Haitianisgood]

Thanks you very much! As your suggest,I use mutate filter it worked!

filter {

  mutate {
       remove_field => ["[beat][name]","[beat][version]","@version","customer_time","offset"]
  }

}