magnusbaeck / logstash-filter-verifier

Apache License 2.0
192 stars 27 forks source link

Escaping strings with double-quotes and backslashes #58

Closed matejzero closed 6 years ago

matejzero commented 6 years ago

Hello,

I have an apache log looking like this:

[04/Jun/2018:07:51:29 +0200] test.example.com:80 10.10.10.2 - GET "/noindex/css/fonts/Semibold/OpenSans-Semibold.ttf"

When I write my test cases, I escape my line to make it JSON valid and at the end, my json file looks like so:

{
  "codec": "line",
  "fields": {
    "type": "access",
    "appid": "httpd",
    "host": "testhost",
    "syslog_format": "false"
  },
  "ignore": [
    "logstash_node"
  ],
  "testcases": [
    {
      "description": "http request",
      "input": [
        "[04\/Jun\/2018:07:51:29 +0200] test.example.com:80 10.10.10.2 - GET \"\/noindex\/css\/fonts\/Semibold\/OpenSans-Semibold.ttf\""
      ],
      "expected": [
        {

        }
      ]
    }
  ]
}

The problem is that when LFV sends the line to the Logstash, it leaves backslashes before double-quotes around request URL and tests fails (since the original log files don't have backslashed quotes).

GROK pattern for log: \[%{HTTPDATE:timestamp}\] %{DATA:vhost}:%{POSINT:port} %{IP:clientip} %{ARNESHTTPDUSER:user} %{WORD:method} \"%{DATA:urlpath}\" \"%{DATA:urlquery}\" %{NUMBER:response} %{NUMBER:servetime} (?:%{NUMBER:bytes}|-) \"%{DATA:referrer}\" %{QS:useragent}

Output: "message": "[04/Jun/2018:07:51:29 +0200] test.example.com:80 10.10.10.2 - GET \"/noindex/css/fonts/Semibold/OpenSans-Semibold.ttf\"",

Am I doing this wrong?

matejzero commented 6 years ago

I replaced double quotes with singles inside the log file and that was that:) Quick and easy.