andrewvc
commented
7 years ago Hi @s33butler . I think it's a reasonable request that if you've got a lot of date fields you should be able to do this in one date filter invocation.
You are correct, by the way, that adding the UTC offset should just fix the problem.
That being said, I think it'd be nice if you let you match against N different source fields and automatically copied them to similarly named destinations.
I was asked to submit an issue by andrewvc for my use case... https://discuss.elastic.co/t/issues-with-dates-times-that-dont-have-a-timezone-logstash-kibana/66008/5
The basic issue is when I try and replace @timestamp with another date, the times on all other dates are 5 hours off in Kibana because they are stored without a timezone and Kibana assumes they must be UTC.
More detail using sample data... csv columns => [ "receive_time","generated_time","start_time"] line would look like ( dates in EST / UTC -5 / -0500) = 2016/11/07 13:09:44,2016/11/07 13:09:39,2016/11/07 13:09:40
Since they are firewall syslog entries, I would prefer that @timestamp use "generated_time" for auditing purposes in case there is an issue with syslog and/or older logs need to be imported.
date { locale => "en" timezone => "America/New_York" match => [ "generated_time", "yyyy/MM/dd HH:mm:ss" ] tag_on_failure => ["_dateparsefailure"] }
Here is where it breaks down... The timestamp is converted to ISO8601(UTC) using the new date (2016/11/07 13:09:39) ...
"@timestamp" => "2016-11-07T18:09:39.000Z", The other three dates are left as is... "receive_time" => "2016/11/07 13:09:44", "generated_time" => "2016/11/07 13:09:39", "start_time" => "2016/11/07 13:09:40",
Because Kibana setting uses the Browser's timezone offset (EST / UTC -5 in my case) all the other times are assumed UTC and then lose 5 hours. 2016/11/07 13:09:44 >> 2016/11/07 08:09:44
I have read so many posts now I don't know what the best solution is. I imagine if I can just add the -0500 timezone to my times then all should be good?
Having 5 dates in my csv, I was able to solve my problem with 6 separate date filters per Andrew_Cholakian1's suggestion by just overwriting the fields.
Perhaps there is a more efficient way as it gets real messy if you have syslogs from different timezones. Say I have firewalls in both UTC -5 and UTC -8, now I need 12 date filters as it is basically (6 x each timezone) for my scenario.
Thank you for your time and assistance. Scott