magnusbaeck / logstash-filter-verifier

Apache License 2.0
192 stars 27 forks source link

Overriding @timestamp in tests #63

Closed ghost closed 5 years ago

ghost commented 5 years ago

Hi, I've started looking at the Logstash filter verifier, and I was wondering if someone could point me in the direction of what I'm doing wrong.

I have a date filter in my Logstash configuration file that looks like this:

date {
    match => [ "timestamp", "ISO8601" ]
    timezone => "UTC" 
    target => "@timestamp"
 }

If I pass in a log entry that looks like this: 2018-10-30 14:51:48,116

I parse it correctly (shown is the Rubydebug output): "@timestamp" => 2018-10-30T14:51:48.116Z,

So this all works as I would expect.

Now, when I try to run the logstash filter verifier, I have created a conf file that looks like this:

{
  "fields": {
    <snip> 
  },
  "input": [
    "2018-10-26 09:25:42,219 <restOfMyMessage>"
  ],
  "ignore": ["host"],
  "expected": [
    {
      "@timestamp": "2018-10-26 09:25:42.219",
      <snip> 
    }
  ]
}

When I run the filter verifier, I get this output:

Running tests in test.json...
Comparing message 1 of 1 from test.json...
--- /tmp/836774239/test.json/1/expected  2018-11-01 08:45:19.800311458 -0400
+++ /tmp/836774239/test.json/1/actual    2018-11-01 08:45:19.798311434 -0400
@@ -1,7 +1,5 @@
{
-  "@timestamp": "2018-10-26 09:25:42.219",
+  "@timestamp": "2018-11-01T12:45:17.467Z",
  <snip> 
 }
Testcase failed, continuing with the rest: 1 message(s) did not match the expectations.
one or more testcases failed

It feels like I'm making a very simple mistake here, but I can't figure out why the date for the same configuration file and input would be different when outputting to rubydebug vs. using the logstash-filter-verifier.

I'm using the latest logstash-filter-verifier (1.5.0) and Logstash is 6.4.1. This behavior occurs regardless of whether or not I use the --logstash-version command line parameter.

Could you please advise? Thank you in advance.

ghost commented 5 years ago

This was due to an error before the date filter was being picked up, and it wasn't obvious using rubydebug.