search

enotspe / fortinet-2-elasticsearch

Fortinet products logs to Elasticsearch
Apache License 2.0
89 stars 39 forks source link

Remove additional useless fields #20

Closed Whysmerhill closed 4 years ago

Whysmerhill commented 4 years ago

Hello,

I have found some additional fields from our fortinet logs which seems useless and can be removed as the N/A one in 40-fortigate_2_ecs :

if [srccountry]=="Reserved" { mutate { remove_field => ["srccountry"] } } if [dstcountry]=="Reserved" { mutate { remove_field => ["dstcountry"] } } if [dstdevcategory]=="None" { mutate { remove_field => ["dstdevcategory"] } } I don't think the reserved value tell us anything meaningful so do you think those fields should be removed too ?

Best regards.

Cyb3rSn0rlax commented 4 years ago

I guess we can remove srccountry and dstcountry anyway since geoip filter is used to enrich events

enotspe commented 4 years ago

what I have seen is that sometimes the geoip fails to map the IP, but Fortinet does provide a country. Some other times coutry mapped by Elastic does not match de country mapped by Fortinet. So that is why I haven´t removed those fields, but I agree those are "removable" fields.

Anyway, the intention of this project is to be as broad as posiible, so I would prefer not to remove them from the parsers here, and just let everybody do their own adjustments according to its own needs. Another strategy could be to remove all none ecs fields and just keep event.original, or to remove event.original and keep the rest, or to rename instead of copy all the translations to avoid duplicate values, etc. All are valid strategies.