[Palo Alto Cortex XDR]: The alerts logs throws "field [message] already exists"

hp0620 commented 3 days ago

Integration Name

Palo Alto Cortex XDR [panw_cortex_xdr]

Dataset Name

panw_cortex_xdr.alerts

Integration Version

1.29.0

Agent Version

8.15.3

Agent Output Type

logstash

Elasticsearch Version

8.15.3

OS Version and Architecture

Debian amd64 20.04.6 LTS (Focal Fossa)

Software/API Version

No response

Error Message

There are no errors in the log and the rest of the data gets ingested. However, in the ingested document, we can see, indicating the field was not correctly ingested, even though the message field was indeed ingested.

          "error": {
            "message": [
              "field [message] already exists"
            ]
          },
            "kind": "pipeline_error",
            "action": "BLOCKED_9",

Event Original

This is from the sample event - https://github.com/elastic/integrations/blob/main/packages/panw_cortex_xdr/data_stream/alerts/_dev/test/pipeline/test-panw-xdr.log-expected.json

"""{"external_id":"396239671","severity":"high","matching_status":"UNMATCHABLE","end_match_attempt_ts":null,"local_insert_ts":1626347122923,"bioc_indicator":null,"matching_service_rule_id":null,"attempt_counter":null,"bioc_category_enum_key":null,"is_whitelisted":false,"starred":false,"deduplicate_tokens":"af4e477c1e284c3f9b1fff340fddb4d0,57f0d1f4096a45bdb4cd8d4b8a626f15","filter_rule_id":null,"mitre_technique_id_and_name":null,"mitre_tactic_id_and_name":null,"agent_version":null,"agent_device_domain":null,"agent_fqdn":null,"agent_os_type":"NO_HOST","agent_os_sub_type":null,"agent_data_collection_status":null,"mac":"4c:ae:a3:8e:c8:6a","events":{"agent_install_type":"NA","agent_host_boot_time":null,"event_sub_type":null,"module_id":null,"association_strength":10,"dst_association_strength":10,"story_id":"MzYzOTQ0MDE1MDI4OTE3NDUyNA==","event_id":"MzYzOTQ0MDE1MDI4OTE3NDUyNA==","event_type":"Network Connections","event_timestamp":1626346867000,"actor_process_instance_id":null,"actor_process_image_path":null,"actor_process_image_name":null,"actor_process_command_line":null,"actor_process_signature_status":"N/A","actor_process_signature_vendor":null,"actor_process_image_sha256":null,"actor_process_image_md5":null,"actor_process_causality_id":null,"actor_causality_id":null,"actor_process_os_pid":null,"actor_thread_thread_id":null,"causality_actor_process_image_name":null,"causality_actor_process_command_line":null,"causality_actor_process_image_path":null,"causality_actor_process_signature_vendor":null,"causality_actor_process_signature_status":"N/A","causality_actor_causality_id":null,"causality_actor_process_execution_time":null,"causality_actor_process_image_md5":null,"causality_actor_process_image_sha256":null,"action_file_path":null,"action_file_name":null,"action_file_md5":null,"action_file_sha256":null,"action_file_macro_sha256":null,"action_registry_data":null,"action_registry_key_name":null,"action_registry_value_name":null,"action_registry_full_key":null,"action_local_ip":"10.10.10.10","action_local_port":58642,"action_remote_ip":"175.16.199.1","action_remote_port":443,"action_external_hostname":"175.16.199.1","action_country":"DK","action_process_instance_id":null,"action_process_causality_id":null,"action_process_image_name":null,"action_process_image_sha256":null,"action_process_image_command_line":null,"action_process_signature_status":"N/A","action_process_signature_vendor":null,"os_actor_effective_username":null,"os_actor_process_instance_id":null,"os_actor_process_image_path":null,"os_actor_process_image_name":null,"os_actor_process_command_line":null,"os_actor_process_signature_status":"N/A","os_actor_process_signature_vendor":null,"os_actor_process_image_sha256":null,"os_actor_process_causality_id":null,"os_actor_causality_id":null,"os_actor_process_os_pid":null,"os_actor_thread_thread_id":null,"fw_app_id":"web-browsing","fw_interface_from":"INTERNET","fw_interface_to":"INTERNET","fw_rule":"INTERNET_INTERNET_GlobalProtect-443","fw_rule_id":null,"fw_device_name":"FW-DEVICE_NAME","fw_serial_number":"12352345","fw_url_domain":"9.9.9.9","fw_email_subject":null,"fw_email_sender":null,"fw_email_recipient":null,"fw_app_subcategory":"internet-utility","fw_app_category":"general-internet","fw_app_technology":"browser-based","fw_vsys":"vsys1","fw_xff":null,"fw_misc":"7.7.7.7/cgi-bin/config.exp","fw_is_phishing":"No","dst_agent_id":"6.6.6.6","dst_causality_actor_process_execution_time":null,"dns_query_name":null,"dst_action_external_hostname":null,"dst_action_country":"US","dst_action_external_port":null,"contains_featured_host":"NO","contains_featured_user":"NO","contains_featured_ip":"NO","image_name":null,"container_id":null,"cluster_name":null,"user_name":null},"alert_id":"2879211","detection_timestamp":1626346849000,"name":"Cisco RV320/RV325 Router Unauthenticated Configuration Export Vulnerability","category":"Vulnerability","endpoint_id":"192.168.2.2","description":"Info-Leak (7.7.7.7/cgi-bin/config.exp)","host_ip":["192.168.2.2"],"host_name":null,"mac_addresses":["ab:ae:f5:5d:c8:6a"],"source":"PAN NGFW","action":"BLOCKED_9","action_pretty":"Prevented (Terminated The Session And Sent a TCP Reset To Both Sides Of The Connection)"}"""

What did you do?

Ingested a document using a single pipeline, to reproduce a behavior a user reported:

PUT test_xdr/_doc/1111?pipeline=logs-panw_cortex_xdr.alerts-1.29.0

What did you see?

Was able to reproduce the same error that the user reported:

          "error": {
            "message": [
              "field [message] already exists"
            ]
          },
            "kind": "pipeline_error",
            "action": "BLOCKED_9",

What did you expect to see?

No field [message] already exists error

Anything else?

The same message above was not detected from the integration data set.

So checking the pipeline - logs-panw_cortex_xdr.alerts-1.29.0, , it looks like we are renaming the field,

  - rename:
      field: panw_cortex.xdr.name
      target_field: message
      ignore_missing: true

, but as you can see in the sample data above, the message field is already there - https://github.com/elastic/integrations/blob/main/packages/panw_cortex_xdr/data_stream/alerts/_dev/test/pipeline/test-panw-xdr.log-expected.json#L51

            "message": "Cisco RV320/RV325 Router Unauthenticated Configuration Export Vulnerability",

, so either the rename needs to send the panw_cortex.xdr.name to a different field or maybe delete the rename part?

elasticmachine commented 3 days ago

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

hp0620 commented 3 days ago

This bug reported is created by Support as a result of the SFDC case - 01779437, instead of creating an SDH to Integration Team.

efd6 commented 3 days ago

This is likely caused by logstash ECS compatibility mode (discussed offline).

hp0620 commented 2 days ago

I just realized that I used the wrong input file when I tested, which of course led to a behavior that's not the true representation of the ingestion.

Feeding a different file - https://github.com/elastic/integrations/blob/main/packages/panw_cortex_xdr/data_stream/alerts/_dev/test/pipeline/test-panw-xdr.log, I did not reproduce the issue the user was reporting. For this reason, I am closing this one.

Thank you, Dan for the quick response and apologies for the wrong alert!

elastic / integrations