search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.37k stars 1.06k forks source link

Messages not routing properly into stream #6129

Closed wazlecracker closed 5 years ago

wazlecracker commented 5 years ago

Messages coming from a Checkpoint firewall aren't properly routing into a stream used for an alert. When loading a message that should have triggered an alert and testing the rules, the stream indicates the message would be routed. Yet they still aren't. I've deleted and recreated the stream along with using the pipeline decorator in search.

Using a pipeline to process messages:

rule "Checkpoint key value conversion"
when 
    from_input("5b04ba37f3b473049282a35e")

then
    set_fields
            (
            fields: key_value
                    (
                    value: to_string($message.message),
                    delimiters: ";",
                    kv_delimiters: ":=",
                    allow_dup_keys: true,
                    trim_key_chars: "_\"[]\\",
                    trim_value_chars: "_\"[]\\"

                    )
            );

end

Have spoken via email with engineer Wayne about this back and forth a few times but I assume he's a busy man and haven't gotten an update in over a month.

Expected Behavior

Proper message routing into a stream

Current Behavior

Messages not being routed into a stream.

Possible Solution

Wayne from Graylog thinks it's possible there's a whitespace character at the root of the issue.

Steps to Reproduce (for bugs)

Here is a sample message from Checkpoint that should, but doesn't, route into the stream <134>1 2019-07-08T13:08:03Z BTF001SA001 CheckPoint 1338 - [action:"Accept"; flags:"16384"; ifdir:"inbound"; ifname:"eth3-01"; logid:"352"; loguid:"{0x5d231603,0x0,0x30a640a,0x4044}"; origin:"10.100.10.3"; originsicname:"CN=BTF-ECP-CPGW02,O=BTF-ECP-CPMA01..zckc5i"; sequencenum:"9"; time:"1562591283"; version:"5"; __policy_id_tag:"product=VPN-1 & FireWall-1[db_tag={F0FF8108-7A1F-AE46-8596-2CD7F6857DD3};mgmt=BTF-ECP-CPMA01;date=1562186812;policy_name=Standard\]"; aggregated_log_count:"1"; connection_count:"1"; creation_time:"1562580483"; dst:"174.140.141.196"; duration:"10800"; hll_key:"11167844646117334161"; inzone:"External"; last_hit_time:"1562580483"; layer_name:"Network"; layer_name:"Application Control"; layer_uuid:"3e43aa22-789f-4915-b98f-60682b6c8019"; layer_uuid:"b5d965fb-85e8-4ba8-b465-d0d84139bad6"; match_id:"22"; match_id:"16777229"; parent_rule:"0"; parent_rule:"0"; rule_action:"Accept"; rule_action:"Accept"; rule_name:"External to CP VPN"; rule_name:"Cleanup rule"; rule_uid:"6c58f420-b977-44a0-a311-6c2ec8e245a2"; rule_uid:"ba1e7d79-b19d-444e-8da2-e980630b8d8c"; outzone:"Local"; product:"VPN-1 & FireWall-1"; proto:"6"; service:"443"; service_id:"https"; src:"178.128.229.22"; update_count:"2"; ]

Here are the rules of the stream: Field rule_action must match exactly Accept Field inzone must match exactly External Field src_country_code must be present Field src_country_code must not match regular expression US|CA

Context

Need to be alerted if a foreign connection attempt gets past geoblocks currently in place.

Your Environment

jalogisch commented 5 years ago

What is the processing order in System > Configuration? If the Processing Pipeline is after Message Filter Chain you can't route with Stream rules on fields that are created in the Pipeline.

Use the stream routing of processing pipelines - that is even more flexibel.

We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our discussion forum or join the #graylog channel on freenode IRC.

Thank you!

wazlecracker commented 5 years ago

Here is the order. Sorry, should have stated this was already checked. 1 | Pipeline Processor | active 2 | Message Filter Chain | active 3 | AWS Instance Name Lookup | active 4 | GeoIP Resolver | active

I was told by a graylog engineer during a call that this is a bug.