search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
23 stars 435 forks source link

[checkpoint] parsing errors #2723

Closed jamiehynds closed 1 year ago

jamiehynds commented 2 years ago

Our Checkpoint integration experiences errors when indexing the events below. Log samples available on request.

1.

# field name cannot be an empty string
<134>1 2021-10-12T20:43:19Z NPMDCM00001 CheckPoint 17476 - [flags:131072"; ifdir:inbound"; loguid:{0x6165f367,0x12,0x9213960a,0x223cc444}"; origin:1.1.1.1"; sequencenum:1"; time:1634071399"; version:5"; default_device_message:<133>NPFDCM00002 xpand[114581\]: admin localhost t +installer:packages:Check_Point_R80_30_JUMBO_HF_Bundle_T237_sk153152_Security_Gateway_3_10_FULL.tgz:description <b>Package Hotfix Bundle R80_30_JUMBO_HF_MAIN contains:<br>&#8226; CPFC R80.30 R80_30_GOGO_JHF_MAIN - Installing...<br><span style='color:blue'>&#8226; Check Point Security Gateway R80.30 R80_30_GOGO_JHF_MAIN</span><br><span style='color:blue'>&#8226; Performance Pack R80.30 R80_30_GOGO_JHF_MAIN</span><br><span style='color:blue'>&#8226; SecurePlatform R80.30 R80_30_GOGO_JHF_MAIN</span><br><span style='color:blue'>&#8226; Mobile Access R80.30 R80_30_GOGO_JHF_MAIN</span><br><span style='color:blue'>&#8226; Deployment Installer</span><br><span style='color:blue'>&#8226; Auto Updater</span><br><br>Note: After the package is installed, the machine reboots.</b> "; facility:local use 0"; product:Syslog"; syslog_severity:Notice"]
          "checkpoint" : {
            ...
            "action_reason" : "Updatable Objects are used in the policy but gateway package is missing (see sk121877).",
          ...
          "error" : {
            "message" : "field [observer.ingress.interface.name] already exists"
          },

The kv processor is splitting fields based on the character :, but in this case, the field is incorrectly splitted by : in the HTML syntax (e.g. style='color:blue') of the description field. And the trailing . in the wrongly splitted key CPFC R80.30 R80_30_GOGO_JHF_MAIN - Installing... was reflected as a JSON path separator. It seems that an additional process is needed before the kv processor.

  1. # failed to parse field [checkpoint.action_reason] of type [long] in document
<134>1 2021-12-08T22:49:59Z NPLDCA00101 CheckPoint 16760 - [action:"Drop"; contextnum:"1"; flags:"283924"; ifdir:"inbound"; ifname:"bond1.2997"; logid:"6"; loguid:"{0x342e51d2,0x18859aaf,0xe739b528,0x219981fe}"; origin:"10.150.18.1"; originsicname:"CN=NPFDCA00002_VS-Internal,O=NPMDCM00001.domain.com.r44cbg"; sequencenum:"219"; time:"1639003799"; version:"5"; __nsons:"0"; __p_dport:"0"; __policy_id_tag:"product=VPN-1 & FireWall-1[db_tag={BD8FEDFC-C097-4D42-B787-FD8117C26BEE};mgmt=NPMDCM00001;date=1639001067;policy_name=VS-Internal_policy\]"; __pos:"7"; action_reason:"Updatable Objects are used in the policy but gateway package is missing (see sk121877)."; bytes:"1364"; client_inbound_bytes:"1312"; client_inbound_interface:"bond1.2997"; client_inbound_packets:"9"; client_outbound_bytes:"52"; client_outbound_packets:"2"; context_num:"4294967295"; dst:"51.105.71.136"; elapsed:"35"; inzone:"Internal"; lastupdatetime:"1639003799"; layer_name:"Network"; layer_name:"App & URL Filter"; layer_uuid:"e22a0b5b-74e0-42d9-9500-06d1285f219c"; layer_uuid:"3d5ca2ae-a176-4198-a14a-6df6d458b636"; match_id:"1948"; match_id:"33554427"; parent_rule:"0"; parent_rule:"0"; rule_action:"Accept"; rule_action:"Accept"; rule_name:"CPNotEnoughDataForRuleMatch"; rule_uid:"1e7567a9-a6fb-4e3c-82f5-f139928e582c"; outzone:"External"; packets:"11"; product:"VPN-1 & FireWall-1"; proto:"6"; reason:"Connection terminated before the Security Gateway was able to make a decision: Insufficient data passed. To learn more see sk113479."; s_port:"64807"; segment_time:"1639003759"; server_inbound_bytes:"52"; server_inbound_packets:"1"; server_outbound_bytes:"1312"; server_outbound_interface:"wrp320"; server_outbound_packets:"18"; service:"443"; service_id:"https"; src:"1.1.1.1"; start_time:"1639003759"]
          "checkpoint" : {
            ...
            "action_reason" : "Updatable Objects are used in the policy but gateway package is missing (see sk121877).",
          ...
          "error" : {
            "message" : "field [observer.ingress.interface.name] already exists"
          },

The pipeline has a processor that renaming checkpoint.action_reason to checkpoint.action_reason_msg if the action_reason is a string. However, this processor was not properly running due to the pipeline has interrupted due to field [observer.ingress.interface.name] already exists error. We may need to add ignore_failure: true for each processor.

  1. # failed to parse field [checkpoint.subs_exp] of type [date] in document
<134>1 2021-12-08T23:39:34Z NPLDCA00101 CheckPoint 16760 - [flags:"166216"; ifdir:"outbound"; loguid:"{0x61b14238,0x1006d,0xcd12960a,0x20bf4178}"; origin:"1.1.1.1"; originsicname:"CN=NPFDCA00002,O=NPMDCM00001.domain.com.r44cbg"; sequencenum:"2"; time:"1639006774"; version:"5"; contract_name:"Anti Virus Basic Metadata"; log_id:"4"; product:"New Anti Virus"; severity:"4"; special_properties:"0"; subs_exp:"Tue Jan  1 00:00:00 2030"; subscription_stat:"valid"; subscription_stat_desc:"Contract is up to date."]
          "checkpoint" : {
            ...
            "subs_exp" : "Tue Jan  1 00:00:00 2030"

subs_exp is not listed in our field definition. We may need to add this.

elasticmachine commented 2 years ago

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

efd6 commented 2 years ago

I saw a similar kind of issue when fixing a problem with the Sophos XG integration (https://github.com/elastic/integrations/pull/2320#pullrequestreview-833536993). Both that case and here have a syntax that regular languages like kv are not powerful enough to express. I would be possible to write a stack-based parser to handle them, but ISTM that this is something that might want more generalised support in the ingest processors; when I looked while fixing the Sophos issue I could not find anything that would do the work.

P1llus commented 2 years ago

We should simply rewrite the regex for the KV processor, it should be sufficient, in terms of the interface issues I will double check where it can be set twice.

Will add the lines provided by the customer to our test cases and update the ingest pipeline