Open helmut72 opened 1 year ago
Probably a grep filter first to drop it? https://docs.fluentbit.io/manual/pipeline/filters/grep
input --> filter (grep) --> filter (geoip) --> output
The grep filter prevents the record carrying on if a key is missing:https://docs.fluentbit.io/manual/pipeline/filters/grep#excluding-records-missing-invalid-fields`
The grep filter prevents the record carrying on if a key is missing
But I want and need the other log lines as it is. Just without empty geoip keys, because it doesn't make sense for log lines that doesn't contain the key configured in lookup_key
.
For example a mailserver (postfix) doesn't log in every line ip addresses. But all other log lines are required too. It also saves CPU cycles, if a log line isn't parsed by geoip2 filter, when the key configured in lookup_key
doesn't exist.
Have also tested re-tagging log lines which contains a remote_addr key as a workaround. Only this new tag will be processed by geoip2 filter. This works, but then the output is unordered from input.
I think it's a bug, because geoip2 filter process a "ghost" key, that doesn't exist.
Having this same issue with a similar configuration in version 2.1.10
My workaround is checking later, if there is a remote_addr
key at all. If not, geoip lookup must have failed and therefore it's save to delete all geo* keys for this log line.
# workaround: remove all geo_* tags if there wasn't remote_addr
[FILTER]
name modify
match app1
condition Key_does_not_exist remote_addr
remove geo_isocode
remove geo_city
remove geo_latitude
remove geo_longitude
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale
label.
Still think this behavior is a bug.
Still think this behavior is a bug.
+1
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale
label.
Still an issue
Bug Report
Describe the bug
geoip2 always process logs, regardless if
lookup_key
exist or not. It always adds empty/nil geoip fields to log lines.To Reproduce
Add geoip2 filter and send a log without 'lookup_key'. Example config:
Output:
Expected behavior
Ignore the log line if
lookup_key
doesn't exist. Are there really reasons to process the log, if nolookup_key
exist?Expected output:
Your Environment
fluent-bit 2.1.8
Thank you.