search

driskell / log-courier

The Log Courier Suite is a set of lightweight tools created to ship and process log files speedily and securely, with low resource usage, to Elasticsearch or Logstash instances.
Other
419 stars 107 forks source link

Use timezone ids compatible with logstash date filter #345

Closed dkirrane closed 3 years ago

dkirrane commented 8 years ago

The timezone created by log courier add timezone field: true doesn't match the Joda timezone ids supported by the Logstash date filter plugin.

Can log-courier produce the same Joda compatible timezone Ids so that it can be used to the date filter plugin? (Joda ids)

Logstash setup using timezone from log courier event:

        date {
            match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss,SSS" ]
            locale => "en"
            timezone => "%{timezone}"
        }

The logstash error I get when I use the timezone produced by log-courier i.e. -0600 MDT:

{:timestamp=>"2016-09-12T11:26:34.487000+0000", :message=>"Failed parsing date from field", :field=>"log_timestamp", :value=>"2016-09-10 20:13:50,769", :exception=>"wrong number of arguments (2 for 1)", :config_parsers=>"YYYY-MM-dd HH:mm:ss,SSS", :config_locale=>"en", :level=>:warn}
driskell commented 8 years ago

You are right, for use in date filter it needs to be a different format.

I think best thing is for https://github.com/driskell/log-courier/blob/master/lc-lib/harvester/harvester.go#L105 to be extended so we have timezoneName too and an additional option, either timezone format to modify existing field, or add timezone name field or something to add another field with correct content - which should be able to come from time.Local.String() - that'll return Europe/London from what I can see.

I'll look at this later if I can but PR are always welcome.

driskell commented 3 years ago

There is now an add timezone name field option that will add a new timezone_name field with the string, such as UTC or Europe/London which all should be compatible.