modify conditions doesn't process longitude and latitude content from geoip2 filter

[helmut72]

Bug Report

Describe the bug

modify conditions doesn't process longitude and latitude output from geoip2 filter correctly, if output of geoip2 filter contains content in latitude or longitude.

To Reproduce

Example configuration:

[INPUT]
    Name   dummy
    Tag    dummy
    Dummy  {"any_key": "any_value", "remote_addr": "8.8.8.8"}

[FILTER]
    Name       geoip2
    Match      dummy
    database   GeoLite2-City.mmdb
    Lookup_key remote_addr
    Record     isocode   remote_addr %{country.iso_code}
    Record     city      remote_addr %{city.names.en}
    Record     latitude  remote_addr %{location.latitude}
    Record     longitude remote_addr %{location.longitude}

[FILTER]
    name      modify
    match     dummy
    condition Key_value_does_not_match isocode .
    remove    isocode

[FILTER]
    name      modify
    match     dummy
    condition Key_value_does_not_match city .
    remove    city

[FILTER]
    name      modify
    match     dummy
    condition Key_value_does_not_match latitude .
    remove    latitude

[FILTER]
    name      modify
    match     dummy
    condition Key_value_does_not_match longitude .
    remove    longitude

[OUTPUT]
    Name   stdout
    Match  dummy

Without modify filters:

[0] dummy: [[1691879021.252772367, {}], {"any_key"=>"any_value", "remote_addr"=>"8.8.8.8", "isocode"=>"US", "city"=>nil, "latitude"=>37.751000, "longitude"=>-97.822000}]

With modify filters:

[0] dummy: [[1691878881.252598045, {}], {"any_key"=>"any_value", "remote_addr"=>"8.8.8.8", "isocode"=>"US"}]

Longitude and latitude were deleted, but they have content.

Expected behavior

Only city should be deleted because it's empty. Longitude and latitude keys won't be deleted if they have also content like isocode. Expected output:

[0] dummy: [[1691879021.252772367, {}], {"any_key"=>"any_value", "remote_addr"=>"8.8.8.8", "isocode"=>"US", "latitude"=>37.751000, "longitude"=>-97.822000}]

Your Environment

fluent-bit 2.1.8

Additional context

I want to do a workaround for 7825 and delete all geo keys, if they are null/nil. Because modify conditions Key_value_equals or Key_value_matches doesn't support nil, even if Ruby regex tester support empty content (another bug?), I did the opposite and use Key_value_does_not_match condition as a workaround.

Thank you

[patrick-stephens]

I wonder if it is because they are not strings? I recall seeing some other issues a while back down to numbers vs strings in the values.

If you have a simple LUA filter that turns them to strings then does modify work?

I would also suggest just moving the chain of modify filters into a single LUA filter anyway - you'll save the cost of pack/unpack between filters and it can cope with null/nil/type errors more specifically for your use case.

[helmut72]

I wonder if it is because they are not strings?

I guess that too. But Key_value_does_not_match does not mention strings only, just REGEXP:VALUE. Numbers are valid in Json values. That's the reason why I think it's a bug.

Even if Lua helps to solve my problem, it's the 3rd workaround ;-)

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.