search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.41k stars 1.07k forks source link

Incorrect "Date" conversion if timezone set to "CET" #2648

Closed jensschulze closed 8 years ago

jensschulze commented 8 years ago

If a given datetime has no timezone info, the timezone "CET" and the timezone "Berlin" in the "Date" converter lead to differing results.

Expected Behavior

Selecting "CET" should result in the same behavior as selecting "Berlin"

Current Behavior

I want to copy the timestamps from a vanilla nginx error log to the "timestamp" in Graylog using a regex extractor with a standard "Date" converter. If I select "CET" as the timezone in the converter, it has no effect: The datetime is treated as if it was UTC already, and every entry is 2 hours ahead. If I select "Berlin" (which is actually CET) the datetime is treated correctly.

Steps to Reproduce

2016/08/09 12:34:56 [error] 21695#0: *6507776 directory index of "/var/www/example/" is forbidden, client: 123.123.123.123, server: www.example.com, request: "GET /example/ HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/test.html"
        {
            "title": "timestamp",
            "type": "REGEX",
            "cursor_strategy": "COPY",
            "target_field": "timestamp",
            "source_field": "message",
            "configuration": {
                "regex_value": "^(\\d+/\\d+/\\d+\\s\\d+:\\d+:\\d+)"
            },
            "converters": [{
                "type": "DATE",
                "configuration": {
                    "date_format": "yyyy/MM/dd HH:mm:ss",
                    "time_zone": "Etc/CET"
                }
            }],
            "condition_type": "NONE",
            "condition_value": "",
            "order": 0
        }
        {
            "title": "timestamp",
            "type": "REGEX",
            "cursor_strategy": "COPY",
            "target_field": "timestamp",
            "source_field": "message",
            "configuration": {
                "regex_value": "^(\\d+/\\d+/\\d+\\s\\d+:\\d+:\\d+)"
            },
            "converters": [{
                "type": "DATE",
                "configuration": {
                    "date_format": "yyyy/MM/dd HH:mm:ss",
                    "time_zone": "Europe/Berlin"
                }
            }],
            "condition_type": "NONE",
            "condition_value": "",
            "order": 0
        }

Context

Your Environment

Nginx logs (access/error) are being sent via nxlog/Graylog Collector Sidecar

joschi commented 8 years ago

@jensschulze

Selecting "CET" should result in the same behavior as selecting "Berlin"

Germany is currently running on Central European Summertime (CEST) and not Central European Time (CET).

If you want to have the current timezone being used in Germany, use "Europe/Berlin" and not "CET".

jensschulze commented 8 years ago

@joschi But CET is one hour ahead of UTC. The converter treated CET as if it was UTC.

joschi commented 8 years ago

@jensschulze Make sure that there is no type (or leading/trailing whitespace) around the timezone.

And like I said, use "Europe/Berlin" if you want to be as specific as possible.

Also see a6a5daea3b5d69d7a7b65fb1f3d1cf33d3f987eb for a specific test case for your issue.

jensschulze commented 8 years ago

@joschi First of all I am using "Europe/Berlin" now so atm the issue does not affect me anymore. I could not run your test yet (but I think you need a functional test here, a unit test will not suffice. Pls see below). But there are two reasons I am still unhappy with this: First of all the server timezone is "CET", not "Europe/Berlin". This may sound picky, but I'd like to use the same timezone identifier even if the results are 100% the same. Secondly, I think there is in fact a bug, and I am quite fond I found it.

Hope this helps!

jensschulze commented 8 years ago

@joschi Sorry for not investigating completely. I did now, and actually the Javascript Application running in the browser is the culprit. It sends "Etc/CET" where "CET" would be correct. I do not know where the JS App gets its timezone mapping from, but the mapping "show CET => send Etc/CET" is definitely wrong. In the Web Interface I changed the Date Converter of one of my extractors to "CET":

PUT /system/inputs/57aaff20adbe1d00014cbcae/extractors/dc779d35-ae80-42c6-a0b2-2c7b68869279 HTTP/1.1
Host: www.graylogtest.dev:12900
Connection: keep-alive
Content-Length: 315
Accept: application/json
Origin: http://www.graylogtest.dev:9000
X-Requested-With: XMLHttpRequest
Authorization: Basic XXXXXXXXXXXXXXXXXXXXX
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36
Content-Type: application/json
DNT: 1
Referer: http://www.graylogtest.dev:9000/system/inputs/749e211a-556d-41a9-af45-02be9b029c19/57aaff20adbe1d00014cbcae/extractors/dc779d35-ae80-42c6-a0b2-2c7b68869279/edit
Accept-Encoding: gzip, deflate, sdch
Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4,es;q=0.2

{"title":"_event_received_time","cut_or_copy":"copy","source_field":"EventReceivedTime","target_field":"_event_received_time","extractor_type":"copy_input","extractor_config":{},"converters":{"date":{"date_format":"yyyy-MM-dd HH:mm:ss","time_zone":"Etc/CET"}},"condition_type":"none","condition_value":"","order":0}

The other "CET" Date converters are not affected in the Content Pack JSON, but this one is now "Etc/CET", and as excpected it does not work as desired anymore.

joschi commented 8 years ago

@jensschulze Thanks for the follow up. We'll see what we can do.

joschi commented 8 years ago

This issue has been addressed in #2395 and will be fixed in Graylog 2.1.0.