[Filebeat][Fortinet] Add the ability to set a default timezone in fortinet config

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash

https://www.elastic.co/products/beats

Other

96 stars 4.92k forks source link

[Filebeat][Fortinet] Add the ability to set a default timezone in fortinet config #20300

Closed marc-gr closed 3 years ago

marc-gr commented 4 years ago

Describe the enhancement:

Some fortinet logs do not have a tz field to set the timezone from, now we are defaulting to UTC for this, but would be desirable to set a fallback default timezone if this happens.

Describe a specific use case for the enhancement or feature:

When we have a log like

<189>date=2020-07-20 time=08:29:08 devname="name" devid="id" logid="0000000013" type="traffic" subtype="forward" level="notice" vd="root" eventtime=1595226548 srcip=1.1.1.1 srcport=53384 srcintf="src-1243" srcintfrole="dmz" dstip=192.168.1.1 dstport=80 dstintf="port25" dstintfrole="lan" poluuid="aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaa" sessionid=3022129855 proto=6 action="close" policyid=213 policytype="policy" service="HTTP" dstcountry="Reserved" srccountry="Reserved" trandisp="noop" duration=3 sentbyte=2942 rcvdbyte=12589 sentpkt=21 rcvdpkt=28 policyname="name" appcat="unscanned"

we want to be able to define which timezone it is other than UTC

cc\ @P1llus

elasticmachine commented 4 years ago

Pinging @elastic/siem (Team:SIEM)

jamiehynds commented 4 years ago

This issue came up recently on Discuss, with a user providing a workaround: https://discuss.elastic.co/t/filebeat-using-fortinet-module-tz-issue/251518/10

elasticmachine commented 3 years ago

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

111andre111 commented 3 years ago

This is a possible workaround but what we really want is related to these 2 issues:

Fix the problem that there is a prepopulated timezone applied when there shouldn't be one.

https://github.com/elastic/beats/pull/20273

Additionally we want it to automatically set a default timezone when no timezone is available in the message in our issue here.

Important to mention here is that this can happen only with older Fortinet Firmwares. Here was a discussion around this topic:

https://github.com/elastic/beats/issues/19010#issuecomment-643247437

jsoriano commented 3 years ago

For consistency, it could be interesting to do what we do in other modules with similar problematics:

Pipelines have two date processors: one to handle events with event.timezone and another one without event.timezone (as here in the nginx module).
Default event.timezone is set with add_locale (as here), and can be also set, or overriden by the user by setting event.timezone with the add_fields processor.
Add docs to the module documentation: https://www.elastic.co/guide/en/beats/filebeat/7.10/filebeat-module-nginx.html#_time_zone_support_10, we have a shared asciidoc for that that can be included in other docs.

nicpenning commented 3 years ago

Just a note, you can use eventtime as a UTC time that matches the date time fields. Why not use that time?

In your example above:

date=2020-07-20 time=08:29:08

eventtime=1595226548 == Monday, July 20, 2020 6:29:08 AM

Thoughts?

zez3 commented 3 years ago

Would this be something related? https://github.com/elastic/beats/issues/11273

nicpenning commented 3 years ago

Would this be something related? https://github.com/elastic/beats/issues/11273

No, I don't think so.