Logstash-filter-verifier seems not work

[disaster37]

Hi,

I use logstash:5.2.2 as a base docker container. I have added logstash-filter-verifier 1.2.0_linux_amd64. But when I run a easy sample (just on test with only one filter), I get:

2017/03/23 17:04:20 Reading test case file: /home/dev/logstash/test/1_10_beeware_test.json (/home/dev/logstash/test/1_10_beeware_test.json)
Use Unix domain sockets.
2017/03/23 17:04:20 Prepared configuration file directory /tmp/268199778 with these files: [1_10_beeware_filter.conf]
2017/03/23 17:04:20 Starting "/usr/share/logstash/bin/logstash" with args ["-w" "1" "--debug" "-e" "input { unix { mode => \"client\" path => \"/tmp/929250645/socket\" codec => \"line\" add_field => { \"[@metadata][__lfv_testcase]\" => \"0\" \"type\" => \"beeware\" } } } output { if [@metadata][__lfv_testcase] == \"0\" { file { path => \"/tmp/486562480\" codec => \"json_lines\" } } }" "-f" "/tmp/268199778" "-l" "/tmp/406888015"].
2017/03/23 17:04:42 Waiting for child with pid 166 to terminate.

And it seems frozen, no more messages ans the process never end.

ie: I have updated input-unix plugin do 3.0.3 and it's the same without use --sockets.

[magnusbaeck]

LFV hasn't been fully verified with Logstash 5 so there might an incompatibility there, but regardless I wouldn't expect it to hang like that. Can you share 1_10_beeware_filter.conf?

I use logstash:5.2.2 as a base docker container.

I'm not sure what you mean by this. Are you running everything in a container? Based on which image? How are you starting the container? Does it work if you don't run it inside a container?

[disaster37]

Ho yeh no problem:

1_10_beeware_filter.conf

filter {
    if "beeware" == [type]  {

        # Parse log
        ##################################################

        grok {
            match => [
                "message",
                "^%{WORD:host} %{IP:ip} \[%{DATA:action}\] \w+/\[\w+\] : %{DATA:[@metadata][rawDate]} - %{DATA:rule} from %{IP:ipSrc} on %{URIHOST:domain}?%{URIPATH:url}$"
            ]
              tag_on_failure => [ "_grokparsefailure_beeware", "malformed" ]
            overwrite => [ "host" ]
        }

        # if src parsed, format date and rename some fields
        if "_grokparsefailure_beeware" not in [tags] {

            # Parse date
            date {
                match => [ "[@metadata][rawDate]", "YYYY/MM/dd HH:mm:ss" ]
                tag_on_failure => [ "_dateparsefailure_beeware", "malformed" ]
            }

            # Geopi
            geoip {
              source => "ipSrc"
            }

            # Put data on lower case
            mutate {
                lowercase => [ "action", "domain" ]
            }

        }
    }
}

And my test:

{
  "fields": {
    "type": "beeware"
  },
  "input": [
    "bw050700cti3700 10.221.197.15 [ALERT] bw050700cti3700/[0df5a4580efe11e7830013191822300b] : 2017/03/22 13:49:57 - LOG_ALERT_NONE SUBTYPE_CUSTOM_MESSAGE Block and log all requests from 184.105.247.195 on 62.62.128.175/\n"
  ],
  "expected": [
    {
      "@timestamp": "2015-10-06T20:55:29.000Z",
      "host": "bw050700cti3700",
      "ip": "10.221.197.15",
      "action": "alert",
      "rule": "LOG_ALERT_NONE SUBTYPE_CUSTOM_MESSAGE Block and log all requests",
      "ipSrc": "184.105.247.195",
      "domain": "62.62.128.175",
      "url": "/",
      "message": "bw050700cti3700 10.221.197.15 [ALERT] bw050700cti3700/[0df5a4580efe11e7830013191822300b] : 2017/03/22 13:49:57 - LOG_ALERT_NONE SUBTYPE_CUSTOM_MESSAGE Block and log all requests from 184.105.247.195 on 62.62.128.175/"
    }
  ]
}

[breml]

@disaster37 There is a small issue with your config. Your input line should not end with \n as a newline is appended by LFV to every event (because the line codec is used).

Expect for this small problem I could run your config and testcase with logstash 5.2.1 and logstash-input-unix v3.0.3 (with and without --sockets). But I did not use docker containers for this.

Maybe there is a problem with the temp-dir within the container, as this is needed in both cases to store config, logs and results. Maybe it makes sense to first get the tests with LFV run without container to narrow down the problem.

[disaster37]

I have tested with this new setting, but i have the same issue.

{
  "fields": {
    "type": "beeware"
  },
  "input": [
    "bw050700cti3700 10.221.197.15 [ALERT] bw050700cti3700/[0df5a4580efe11e7830013191822300b] : 2017/03/22 13:49:57 - LOG_ALERT_NONE SUBTYPE_CUSTOM_MESSAGE Block and log all requests from 184.105.247.195 on 62.62.128.175/"
  ],
  "expected": [
    {
      "@timestamp": "2015-10-06T20:55:29.000Z",
      "host": "bw050700cti3700",
      "ip": "10.221.197.15",
      "action": "alert",
      "rule": "LOG_ALERT_NONE SUBTYPE_CUSTOM_MESSAGE Block and log all requests",
      "ipSrc": "184.105.247.195",
      "domain": "62.62.128.175",
      "url": "/",
      "message": "bw050700cti3700 10.221.197.15 [ALERT] bw050700cti3700/[0df5a4580efe11e7830013191822300b] : 2017/03/22 13:49:57 - LOG_ALERT_NONE SUBTYPE_CUSTOM_MESSAGE Block and log all requests from 184.105.247.195 on 62.62.128.175/"
    }
  ]
}

[breml]

Did you test without docker container?

[disaster37]

I haven't server on my hand to install from scratch logstash. But I look on /tmp/378603864 (the output logstash) that my input event has been processed by Logstash:

cat /tmp/378603864

{"geoip":{"timezone":"America/Los_Angeles","ip":"184.105.247.195","latitude":37.5497,"continent_code":"NA","city_name":"Fremont","country_code2":"US","country_name":"United States","dma_code":807,"country_code3":"US","region_name":"California","location":[-121.9621,37.5497],"postal_code":"94539","longitude":-121.9621,"region_code":"CA"},"ipSrc":"184.105.247.195","ip":"10.221.197.15","rule":"LOG_ALERT_NONE SUBTYPE_CUSTOM_MESSAGE Block and log all requests","message":"bw050700cti3700 10.221.197.15 [ALERT] bw050700cti3700/[0df5a4580efe11e7830013191822300b] : 2017/03/22 13:49:57 - LOG_ALERT_NONE SUBTYPE_CUSTOM_MESSAGE Block and log all requests from 184.105.247.195 on 62.62.128.175/","type":"beeware","url":"/","path":"/tmp/193843357/socket","@timestamp":"2017-03-22T13:49:57.000Z","domain":"62.62.128.175","@version":"1","host":"bw050700cti3700","action":"alert"}

When I look the code, I think logstash-filter-verifier stay that logstash exist ? right ? I but I don't understand why Logstash will exist because of it continue to stay new event ...

[disaster37]

Hum i think the problem is link with the x-pack plugins ...

[disaster37]

I finnaly found the problem. Need to add xpack.monitoring.enabled: false on logstash.yml when you install x-pack plugin.

Thx all for your help ;)