jamiehynds commented 2 years ago
Description

Palo Alto Networks Cortex Data Lake stores the context-rich enhanced network logs generated by our security products, including PANW next-generation firewalls, Prisma Access, and Cortex XDR. Most Cortex apps use the Cortex Data Lake to access, analyze, and report on your network data.
Architecture

Cortex Data Lake supports log forwarding via syslog. Relevant docs available here
Integration release checklist

This checklist is intended for integrations maintainers to ensure consistency when creating or updating a Package, Module or Dataset for an Integration.
All changes

[ ] Change follows the contributing guidelines
[ ] Supported versions of the monitoring target are documented
[ ] Supported operating systems are documented (if applicable)
[ ] Integration or System tests exist
[ ] Documentation exists
[ ] Fields follow ECS and naming conventions
[ ] At least a manual test with ES / Kibana / Agent has been performed.
[ ] Required Kibana version set to:
New Package

[ ] Screenshot of the "Add Integration" page on Fleet added
Dashboards changes

[ ] Dashboards exists
[ ] Screenshots added or updated
[ ] Datastream filters added to visualizations
Log dataset changes

[ ] Pipeline tests exist (if applicable)
[ ] Generated output for at least 1 log file exists
[ ] Sample event (sample_event.json) exists
elasticmachine commented 2 years ago
Pinging @elastic/security-external-integrations (Team:Security-External Integrations)
LaZyDK commented 2 years ago
I have this pipeline in production. It can handle TRAFFIC and THREAT logs at this time.
PUT _ingest/pipeline/logs-panw.datalake-0.1.0
{
  "description": "Pipeline for Palo Alto Networks Datalake Logs",
  "processors": [
    {
      "set": {
        "field": "event.ingested",
        "value": "{{_ingest.timestamp}}"
      }
    },
    {
      "set": {
        "field": "ecs.version",
        "value": "8.0.0"
      }
    },
    {
      "set": {
        "field": "observer.vendor",
        "value": "Palo Alto Networks"
      }
    },
    {
      "set": {
        "field": "observer.product",
        "value": "PAN-OS"
      }
    },
    {
      "set": {
        "field": "observer.type",
        "value": "firewall"
      }
    },
    {
      "set": {
        "field": "event.original",
        "value": "{{message}}"
      }
    },
    {
      "gsub": {
        "field": "message",
        "pattern": "[\r\n]",
        "replacement": "",
        "ignore_missing": true
      }
    },
    {
      "grok": {
        "field": "message",
        "patterns": [
          "- panwlogs - %{DATA:event.created},%{DATA:observer.serial_number},%{DATA:panw.panos.type},(?:%{DATA:panw.panos.sub_type})?,%{DATA},%{DATA:_temp_.generated_time},%{GREEDYDATA:message}$"
        ]
      }
    },
    {
      "csv": {
        "field": "message",
        "target_fields": [
          "source.ip",
          "destination.ip",
          "source.nat.ip",
          "destination.nat.ip",
          "panw.panos.ruleset",
          "source.user.name",
          "destination.user.name",
          "network.application",
          "panw.panos.related_vsys",
          "observer.ingress.zone",
          "observer.egress.zone",
          "observer.ingress.interface.name",
          "observer.egress.interface.name",
          "panw.panos.log_profile",
          "panw.panos.flow_id",
          "panw.panos.repeat_count",
          "source.port",
          "destination.port",
          "source.nat.port",
          "destination.nat.port",
          "network.transport",
          "panw.panos.action",
          "network.bytes",
          "source.bytes",
          "destination.bytes",
          "network.packets",
          "event.start",
          "event.duration",
          "panw.panos.url_category",
          "panw.panos.sequence_number",
          "_temp_.srcloc",
          "_temp_.dstloc",
          "source.packets",
          "destination.packets",
          "event.reason",
          "panw.panos.dg_hier_level_1",
          "panw.panos.dg_hier_level_2",
          "panw.panos.dg_hier_level_3",
          "panw.panos.dg_hier_level_4",
          "panw.panos.vsys_name",
          "observer.hostname",
          "panw.panos.action_source",
          "panw.panos.source_uuid",
          "panw.panos.dest_uuid",
          "panw.panos.tunnelid_imsi",
          "panw.panos.monitor_tag_imei",
          "panw.panos.parent_session_id",
          "panw.panos.parent_start_time",
          "panw.panos.tunnel",
          "panw.panos.ep_assoc_id",
          "panw.panos.chunks_total",
          "panw.panos.chunks_sent",
          "panw.panos.chunks_received",
          "panw.panos.rule_matched_uuid",
          "panw.panos.http2_connection",
          "panw.panos.link_change_count",
          "panw.panos.policy_id",
          "panw.panos.link_switches",
          "panw.panos.sdwan_cluster",
          "panw.panos.sdwan_device_type",
          "panw.panos.sdwan_cluster_type",
          "panw.panos.sdwan_site",
          "panw.panos.dynusergroup_name",
          "panw.panos.xff_ip",
          "panw.panos.source_device_category",
          "panw.panos.source_device_profile",
          "panw.panos.source_device_modee",
          "panw.panos.source_device_vendor",
          "panw.panos.source_device_osfamily",
          "panw.panos.source_device_osversion",
          "panw.panos.source_device_host",
          "panw.panos.source_device_mac",
          "panw.panos.dest_device_category",
          "panw.panos.dest_device_profile",
          "panw.panos.dest_device_model",
          "panw.panos.dest_device_vendor",
          "panw.panos.dest_device_osfamily",
          "panw.panos.dest_device_osversion",
          "panw.panos.dest_device_host",
          "panw.panos.dest_device_mac",
          "panw.panos.container_id",
          "panw.panos.pod_namespace",
          "panw.panos.pod_name",
          "panw.panos.source_edl",
          "panw.panos.dest_edl",
          "panw.panos.host_id",
          "panw.panos.endpoint_serial_number",
          "panw.panos.source_dynamic_address_group",
          "panw.panos.dest_dynamic_address_group",
          "panw.panos.ha_session_owner",
          "panw.panos.time_generated_high_res",
          "panw.panos.nssai_network_slice_type",
          "panw.panos.nssai_network_slice_differentiator"
        ],
        "if": "ctx?.panw?.panos?.type == 'TRAFFIC'"
      }
    },
    {
      "csv": {
        "field": "message",
        "target_fields": [
          "source.ip",
          "destination.ip",
          "source.nat.ip",
          "destination.nat.ip",
          "panw.panos.ruleset",
          "source.user.name",
          "destination.user.name",
          "network.application",
          "panw.panos.related_vsys",
          "observer.ingress.zone",
          "observer.egress.zone",
          "observer.ingress.interface.name",
          "observer.egress.interface.name",
          "panw.panos.log_profile",
          "panw.panos.flow_id",
          "panw.panos.repeat_count",
          "source.port",
          "destination.port",
          "source.nat.port",
          "destination.nat.port",
          "network.transport",
          "panw.panos.action",
          "url.original",
          "panw.panos.threat.name",
          "panw.panos.threat.severity",
          "_temp_.direction",
          "panw.panos.sequence_number",
          "_temp_.srcloc",
          "_temp_.dstloc",
          "panw.panos.network.pcap_id",
          "file.hash.sha256",
          "panw.panos.cloud",
          "panw.panos.url_idx",
          "panw.panos.file_type",
          "email.sender",
          "email.subject",
          "email.recepient",
          "panw.panos.report_id",
          "panw.panos.dg_hier_level_1",
          "panw.panos.dg_hier_level_2",
          "panw.panos.dg_hier_level_3",
          "panw.panos.dg_hier_level_4",
          "panw.panos.vsys_name",
          "observer.hostname",
          "panw.panos.source_uuid",
          "panw.panos.dest_uuid",
          "panw.panos.tunnelid_imsi",
          "panw.panos.monitor_tag_imei",
          "panw.panos.parent_session_id",
          "panw.panos.parent_start_time",
          "panw.panos.tunnel",
          "panw.panos.threat.category",
          "panw.panos.content_version",
          "panw.panos.sig_flags",
          "panw.panos.rule_matched_uuid",
          "panw.panos.http2_connection",
          "panw.panos.dynusergroup_name",
          "panw.panos.xff_ip",
          "panw.panos.source_device_category",
          "panw.panos.source_device_profile",
          "panw.panos.source_device_model",
          "panw.panos.source_device_vendor",
          "panw.panos.source_device_osfamily",
          "panw.panos.source_device_osversion",
          "panw.panos.source_device_host",
          "panw.panos.source_device_mac",
          "panw.panos.dest_device_category",
          "panw.panos.dest_device_profile",
          "panw.panos.dest_device_model",
          "panw.panos.dest_device_vendor",
          "panw.panos.dest_device_osfamily",
          "panw.panos.dest_device_osversion",
          "panw.panos.dest_device_host",
          "panw.panos.dest_device_mac"
        ],
        "if": "ctx?.panw?.panos?.type == 'THREAT'"
      }
    },
    {
      "set": {
        "field": "event.timezone",
        "value": "{{_conf.tz_offset}}",
        "if": "ctx?._conf?.tz_offset != null && ctx?._conf?.tz_offset != 'local'"
      }
    },
    {
      "date": {
        "if": "ctx?.event?.timezone != null",
        "field": "_temp_.generated_time",
        "formats": [
          "yyyy/MM/dd HH:mm:ss"
        ],
        "timezone": "{{ event.timezone }}",
        "on_failure": [
          {
            "append": {
              "field": "error.message",
              "value": "{{ _ingest.on_failure_message }}"
            }
          }
        ]
      }
    },
    {
      "date": {
        "if": "ctx?.event?.timezone != null && ctx?.event?.created != null ",
        "field": "event.created",
        "target_field": "event.created",
        "formats": [
          "yyyy/MM/dd HH:mm:ss"
        ],
        "timezone": "{{ event.timezone }}",
        "on_failure": [
          {
            "append": {
              "field": "error.message",
              "value": "{{ _ingest.on_failure_message }}"
            }
          }
        ]
      }
    },
    {
      "date": {
        "if": "ctx?.event?.timezone != null && ctx?.event?.start != null",
        "field": "event.start",
        "target_field": "event.start",
        "timezone": "{{ event.timezone }}",
        "formats": [
          "yyyy/MM/dd HH:mm:ss"
        ],
        "on_failure": [
          {
            "append": {
              "field": "error.message",
              "value": "{{ _ingest.on_failure_message }}"
            }
          }
        ]
      }
    },
    {
      "date": {
        "if": "ctx?.event?.timezone != null && ctx?.panw?.panos?.parent_session?.start_time != null",
        "field": "panw.panos.parent_session.start_time",
        "target_field": "panw.panos.parent_session.start_time",
        "timezone": "{{ event.timezone }}",
        "formats": [
          "yyyy/MM/dd HH:mm:ss"
        ],
        "on_failure": [
          {
            "append": {
              "field": "error.message",
              "value": "{{ _ingest.on_failure_message }}"
            }
          }
        ]
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "source.bytes"
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "source.packets"
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "source.port"
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "destination.bytes"
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "destination.packets"
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "destination.port"
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "network.bytes"
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "network.packets"
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "event.duration"
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "_temp_.labels"
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "panw.panos.sequence_number"
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "source.nat.port"
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "destination.nat.port"
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "panw.panos.repeat_count"
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "panw.panos.scp.chunks"
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "panw.panos.scp.chunks_sent"
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "panw.panos.scp.chunks_received"
      }
    },
    {
      "remove": {
        "if": "ctx?.panw?.panos?.network?.pcap_id == \"0\"",
        "field": [
          "panw.panos.network.pcap_id"
        ]
      }
    },
    {
      "script": {
        "lang": "painless",
        "if": "ctx?._temp_?.labels != null && ctx._temp_.labels != 0",
        "params": {
          "pcap_included": 2147483648,
          "ipv6_session": 33554432,
          "ssl_decrypted": 16777216,
          "url_filter_denied": 8388608,
          "nat_translated": 4194304,
          "captive_portal": 2097152,
          "x_forwarded_for": 524288,
          "http_proxy": 262144,
          "container_page": 32768,
          "temporary_match": 8192,
          "symmetric_return": 2048
        },
        "source": "def labels = ctx?.labels; if (labels == null) {\n  labels = new HashMap();\n  ctx['labels'] = labels;\n} long value = ctx._temp_.labels; for (entry in params.entrySet()) {\n  def flag = entry.getValue();\n  if (flag instanceof String) {\n      flag = Long.decode(flag);\n  }\n  if ((value & flag) != 0) {\n      labels[entry.getKey()] = true;\n  }\n}\n"
      }
    },
    {
      "script": {
        "lang": "painless",
        "if": "ctx?.event?.duration != null",
        "params": {
          "NANOS_IN_A_SECOND": 1000000000
        },
        "source": "long nanos = ctx['event']['duration'] * params.NANOS_IN_A_SECOND; ctx['event']['duration'] = nanos; def start = ctx.event?.start; if (start != null) {\n  ctx.event['end'] = ZonedDateTime.parse(start).plusNanos(nanos);\n}\n"
      }
    },
    {
      "set": {
        "field": "network.direction",
        "value": "inbound",
        "if": "ctx?.panw?.panos?.type == \"TRAFFIC\" && ctx?._temp_?.external_zones != null && ctx?._temp_?.internal_zones != null && ctx?.observer?.ingress?.zone != null && ctx?.observer?.egress?.zone != null && ctx._temp_.external_zones.contains(ctx.observer.ingress.zone) && ctx._temp_.internal_zones.contains(ctx.observer.egress.zone)\n"
      }
    },
    {
      "set": {
        "field": "network.direction",
        "value": "outbound",
        "if": "ctx?.panw?.panos?.type == \"TRAFFIC\" && ctx?._temp_?.external_zones != null && ctx?._temp_?.internal_zones != null && ctx?.observer?.ingress?.zone != null && ctx?.observer?.egress?.zone != null && ctx._temp_.external_zones.contains(ctx.observer.egress.zone) && ctx._temp_.internal_zones.contains(ctx.observer.ingress.zone)\n"
      }
    },
    {
      "set": {
        "field": "network.direction",
        "value": "internal",
        "if": "ctx?.panw?.panos?.type == \"TRAFFIC\" && ctx?._temp_?.internal_zones != null && ctx?.observer?.ingress?.zone != null && ctx?.observer?.egress?.zone != null && ctx._temp_.internal_zones.contains(ctx.observer.egress.zone) && ctx._temp_.internal_zones.contains(ctx.observer.ingress.zone)\n"
      }
    },
    {
      "set": {
        "field": "network.direction",
        "value": "external",
        "if": "ctx?.panw?.panos?.type == \"TRAFFIC\" && ctx?._temp_?.external_zones != null && ctx?.observer?.ingress?.zone != null && ctx?.observer?.egress?.zone != null && ctx._temp_.external_zones.contains(ctx.observer.egress.zone) && ctx._temp_.external_zones.contains(ctx.observer.ingress.zone)\n"
      }
    },
    {
      "set": {
        "field": "network.direction",
        "value": "unknown",
        "if": "ctx?.panw?.panos?.type == \"TRAFFIC\" && ctx?._temp_?.external_zones != null && ctx?._temp_?.internal_zones != null && (\n  (\n    !ctx._temp_.external_zones.contains(ctx?.observer?.egress?.zone) &&\n    !ctx._temp_.internal_zones.contains(ctx?.observer?.egress?.zone)\n  ) ||\n  (\n    !ctx._temp_.external_zones.contains(ctx?.observer?.ingress?.zone) &&\n    !ctx._temp_.internal_zones.contains(ctx?.observer?.ingress?.zone)\n  )\n)\n"
      }
    },
    {
      "set": {
        "field": "network.direction",
        "value": "inbound",
        "if": "ctx?.panw?.panos?.type == \"THREAT\" && (ctx?._temp_?.direction == \"0\" || ctx?._temp_?.direction == \"client-to-server\")"
      }
    },
    {
      "set": {
        "field": "network.direction",
        "value": "outbound",
        "if": "ctx?.panw?.panos?.type == \"THREAT\" && (ctx?._temp_?.direction == \"1\" || ctx?._temp_?.direction == \"server-to-client\")"
      }
    },
    {
      "set": {
        "field": "network.direction",
        "value": "unknown",
        "if": "ctx?.panw?.panos?.type == \"THREAT\" && ctx?.network?.direction == null"
      }
    },
    {
      "set": {
        "field": "network.type",
        "value": "ipv4",
        "if": "ctx?.panw?.panos?.type == \"TRAFFIC\" && ctx?.labels?.ipv6_session == null"
      }
    },
    {
      "set": {
        "field": "network.type",
        "value": "ipv6",
        "if": "ctx?.panw?.panos?.type == \"TRAFFIC\" && ctx?.labels?.ipv6_session != null"
      }
    },
    {
      "set": {
        "field": "event.kind",
        "value": "event",
        "if": "ctx?.panw?.panos?.type == \"TRAFFIC\""
      }
    },
    {
      "append": {
        "field": "event.category",
        "value": [
          "network_traffic",
          "network"
        ],
        "if": "ctx?.panw?.panos?.type == \"TRAFFIC\""
      }
    },
    {
      "set": {
        "field": "event.kind",
        "value": "alert",
        "if": "ctx?.panw?.panos?.type == \"THREAT\""
      }
    },
    {
      "append": {
        "field": "event.category",
        "value": [
          "security_threat",
          "intrusion_detection",
          "network"
        ],
        "if": "ctx?.panw?.panos?.type == \"THREAT\""
      }
    },
    {
      "append": {
        "field": "event.type",
        "value": "allowed",
        "if": "ctx?.panw?.panos?.action != null && ['alert', 'allow', 'continue'].contains(ctx.panw.panos.action)"
      }
    },
    {
      "append": {
        "field": "event.type",
        "value": "denied",
        "if": "ctx?.panw?.panos?.action != null && ['deny', 'drop', 'reset-client', 'reset-server', 'reset-both', 'block-url', 'block-ip', 'random-drop', 'sinkhole', 'block'].contains(ctx.panw.panos.action)"
      }
    },
    {
      "set": {
        "field": "event.outcome",
        "value": "success"
      }
    },
    {
      "set": {
        "field": "event.action",
        "value": "flow_started",
        "if": "ctx?.panw?.panos?.sub_type == \"start\""
      }
    },
    {
      "append": {
        "field": "event.type",
        "value": [
          "start",
          "connection"
        ],
        "if": "ctx?.panw?.panos?.sub_type == \"start\""
      }
    },
    {
      "set": {
        "field": "event.action",
        "value": "flow_terminated",
        "if": "ctx?.panw?.panos?.sub_type == \"end\""
      }
    },
    {
      "append": {
        "field": "event.type",
        "value": [
          "end",
          "connection"
        ],
        "if": "ctx?.panw?.panos?.sub_type == \"end\""
      }
    },
    {
      "set": {
        "field": "event.action",
        "value": "flow_dropped",
        "if": "ctx?.panw?.panos?.sub_type == \"drop\""
      }
    },
    {
      "append": {
        "field": "event.type",
        "value": [
          "denied",
          "connection"
        ],
        "if": "ctx?.panw?.panos?.sub_type == \"drop\""
      }
    },
    {
      "set": {
        "field": "event.action",
        "value": "flow_denied",
        "if": "ctx?.panw?.panos?.sub_type == \"deny\""
      }
    },
    {
      "append": {
        "field": "event.type",
        "value": [
          "denied",
          "connection"
        ],
        "if": "ctx?.panw?.panos?.sub_type == \"deny\""
      }
    },
    {
      "set": {
        "field": "event.action",
        "value": "data_match",
        "if": "ctx?.panw?.panos?.sub_type == \"data\""
      }
    },
    {
      "set": {
        "field": "event.action",
        "value": "file_match",
        "if": "ctx?.panw?.panos?.sub_type == \"file\""
      }
    },
    {
      "set": {
        "field": "event.action",
        "value": "flood_detected",
        "if": "ctx?.panw?.panos?.sub_type == \"flood\""
      }
    },
    {
      "set": {
        "field": "event.action",
        "value": "packet_attack",
        "if": "ctx?.panw?.panos?.sub_type == \"packet\""
      }
    },
    {
      "set": {
        "field": "event.action",
        "value": "scan_detected",
        "if": "ctx?.panw?.panos?.sub_type == \"scan\""
      }
    },
    {
      "set": {
        "field": "event.action",
        "value": "spyware_detected",
        "if": "ctx?.panw?.panos?.sub_type == \"spyware\""
      }
    },
    {
      "set": {
        "field": "event.action",
        "value": "url_filtering",
        "if": "ctx?.panw?.panos?.sub_type == \"url\""
      }
    },
    {
      "set": {
        "field": "event.action",
        "value": "virus_detected",
        "if": "ctx?.panw?.panos?.sub_type == \"virus\""
      }
    },
    {
      "set": {
        "field": "event.action",
        "value": "exploit_detected",
        "if": "ctx?.panw?.panos?.sub_type == \"vulnerability\""
      }
    },
    {
      "set": {
        "field": "event.action",
        "value": "wildfire_verdict",
        "if": "ctx?.panw?.panos?.sub_type == \"wildfire\""
      }
    },
    {
      "set": {
        "field": "event.action",
        "value": "wildfire_virus_detected",
        "if": "ctx?.panw?.panos?.sub_type == \"wildfire-virus\""
      }
    },
    {
      "set": {
        "field": "event.severity",
        "if": "ctx?.log?.level == \"critical\"",
        "value": 1
      }
    },
    {
      "set": {
        "field": "event.severity",
        "if": "ctx?.log?.level == \"high\"",
        "value": 2
      }
    },
    {
      "set": {
        "field": "event.severity",
        "if": "ctx?.log?.level == \"medium\"",
        "value": 3
      }
    },
    {
      "set": {
        "field": "event.severity",
        "if": "ctx?.log?.level == \"low\"",
        "value": 4
      }
    },
    {
      "set": {
        "field": "event.severity",
        "if": "ctx?.log?.level == \"informational\"",
        "value": 5
      }
    },
    {
      "set": {
        "field": "panw.panos.action",
        "value": "drop-icmp",
        "if": "ctx?.panw?.panos?.action == \"drop icmp\" || ctx?.panw?.panos?.action == \"drop ICMP\""
      }
    },
    {
      "set": {
        "field": "panw.panos.action",
        "value": "reset-both",
        "if": "ctx?.panw?.panos?.action == \"reset both\""
      }
    },
    {
      "set": {
        "field": "panw.panos.action",
        "value": "reset-client",
        "if": "ctx?.panw?.panos?.action == \"reset client\""
      }
    },
    {
      "set": {
        "field": "panw.panos.action",
        "value": "reset-server",
        "if": "ctx?.panw?.panos?.action == \"reset server\""
      }
    },
    {
      "set": {
        "field": "panw.panos.destination.nat.ip",
        "copy_from": "destination.nat.ip",
        "if": "ctx?.destination?.nat?.ip != null"
      }
    },
    {
      "set": {
        "field": "panw.panos.source.nat.ip",
        "copy_from": "source.nat.ip",
        "if": "ctx?.source?.nat?.ip != null"
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "destination.nat.port",
        "target_field": "panw.panos.destination.nat.port"
      }
    },
    {
      "convert": {
        "type": "long",
        "ignore_missing": true,
        "field": "source.nat.port",
        "target_field": "panw.panos.source.nat.port"
      }
    },
    {
      "append": {
        "if": "ctx?.source?.ip != null",
        "field": "related.ip",
        "allow_duplicates": false,
        "value": [
          "{{source.ip}}"
        ]
      }
    },
    {
      "append": {
        "if": "ctx?.destination?.ip != null",
        "field": "related.ip",
        "allow_duplicates": false,
        "value": [
          "{{destination.ip}}"
        ]
      }
    },
    {
      "append": {
        "if": "ctx?.source?.nat?.ip != null",
        "field": "related.ip",
        "allow_duplicates": false,
        "value": [
          "{{source.nat.ip}}"
        ]
      }
    },
    {
      "append": {
        "if": "ctx?.destination?.nat?.ip != null",
        "field": "related.ip",
        "allow_duplicates": false,
        "value": [
          "{{destination.nat.ip}}"
        ]
      }
    },
    {
      "geoip": {
        "if": "ctx?.source?.ip != null",
        "field": "source.ip",
        "target_field": "source.geo"
      }
    },
    {
      "geoip": {
        "if": "ctx?.destination?.ip != null",
        "field": "destination.ip",
        "target_field": "destination.geo"
      }
    },
    {
      "geoip": {
        "database_file": "GeoLite2-ASN.mmdb",
        "field": "source.ip",
        "target_field": "source.as",
        "properties": [
          "asn",
          "organization_name"
        ],
        "ignore_missing": true
      }
    },
    {
      "geoip": {
        "database_file": "GeoLite2-ASN.mmdb",
        "field": "destination.ip",
        "target_field": "destination.as",
        "properties": [
          "asn",
          "organization_name"
        ],
        "ignore_missing": true
      }
    },
    {
      "rename": {
        "field": "source.as.asn",
        "target_field": "source.as.number",
        "ignore_missing": true
      }
    },
    {
      "rename": {
        "field": "source.as.organization_name",
        "target_field": "source.as.organization.name",
        "ignore_missing": true
      }
    },
    {
      "rename": {
        "field": "destination.as.asn",
        "target_field": "destination.as.number",
        "ignore_missing": true
      }
    },
    {
      "rename": {
        "field": "destination.as.organization_name",
        "target_field": "destination.as.organization.name",
        "ignore_missing": true
      }
    },
    {
      "rename": {
        "if": "ctx.source?.geo?.name == null",
        "field": "_temp_.srcloc",
        "target_field": "source.geo.name",
        "ignore_missing": true
      }
    },
    {
      "rename": {
        "if": "ctx.destination?.geo?.name == null",
        "field": "_temp_.dstloc",
        "target_field": "destination.geo.name",
        "ignore_missing": true
      }
    },
    {
      "convert": {
        "field": "source.port",
        "type": "integer",
        "if": "ctx?.source?.port != null"
      }
    },
    {
      "convert": {
        "field": "destination.port",
        "type": "integer",
        "if": "ctx?.destination?.port != null"
      }
    },
    {
      "convert": {
        "field": "source.nat.port",
        "type": "integer",
        "if": "ctx?.source?.nat?.port != null"
      }
    },
    {
      "convert": {
        "field": "destination.nat.port",
        "type": "integer",
        "if": "ctx?.destination?.nat?.port != null"
      }
    },
    {
      "community_id": {
        "target_field": "network.community_id",
        "if": "ctx?.source?.port != null && ctx?.source?.port != 0 && ctx?.destination?.port != null && ctx?.destination?.port != 0"
      }
    },
    {
      "community_id": {
        "target_field": "panw.panos.network.nat.community_id",
        "source_ip": "source.nat.ip",
        "source_port": "source.nat.port",
        "destination_ip": "destination.nat.ip",
        "destination_port": "destination.nat.port",
        "if": "ctx?.source?.nat?.port != null && ctx?.source?.nat?.port != 0 && ctx?.destination?.nat?.port != null && ctx?.destination?.nat?.port != 0"
      }
    },
    {
      "append": {
        "if": "ctx?.panw?.panos?.network?.nat?.community_id != null && ctx.panw.panos.network.nat.community_id != ctx?.network?.community_id",
        "field": "network.community_id",
        "value": [
          "{{panw.panos.network.nat.community_id}}"
        ]
      }
    },
    {
      "grok": {
        "if": "ctx?.panw?.panos?.threat?.name != null",
        "field": "panw.panos.threat.name",
        "ignore_failure": true,
        "patterns": [
          "%{GREEDYDATA:panw.panos.threat.name}\\(\\s*%{GREEDYDATA:panw.panos.threat.id}\\s*\\)"
        ]
      }
    },
    {
      "set": {
        "field": "panw.panos.threat.name",
        "value": "URL-filtering",
        "if": "ctx?.panw?.panos?.threat?.id == \"9999\""
      }
    },
    {
      "set": {
        "field": "rule.name",
        "value": "{{panw.panos.ruleset}}",
        "ignore_empty_value": true
      }
    },
    {
      "append": {
        "field": "related.user",
        "allow_duplicates": false,
        "value": "{{client.user.name}}",
        "if": "ctx?.client?.user?.name != null"
      }
    },
    {
      "append": {
        "field": "related.user",
        "allow_duplicates": false,
        "value": "{{source.user.name}}",
        "if": "ctx?.source?.user?.name != null"
      }
    },
    {
      "append": {
        "field": "related.user",
        "allow_duplicates": false,
        "value": "{{server.user.name}}",
        "if": "ctx?.server?.user?.name != null"
      }
    },
    {
      "append": {
        "field": "related.user",
        "allow_duplicates": false,
        "value": "{{destination.user.name}}",
        "if": "ctx?.destination?.user?.name != null"
      }
    },
    {
      "append": {
        "field": "related.hash",
        "allow_duplicates": false,
        "value": "{{panw.panos.file.hash}}",
        "if": "ctx?.panw?.panos?.file?.hash != null"
      }
    },
    {
      "append": {
        "field": "related.hosts",
        "value": "{{observer.hostname}}",
        "if": "ctx?.observer?.hostname != null && ctx.observer?.hostname != ''",
        "allow_duplicates": false
      }
    },
    {
      "remove": {
        "field": [
          "_temp_",
          "_conf"
        ],
        "ignore_missing": true
      }
    },
    {
      "remove": {
        "field": [
          "source.nat.ip",
          "source.nat.port"
        ],
        "if": "ctx?.source?.nat?.ip == \"0.0.0.0\" && ctx?.source?.nat?.port == 0"
      }
    },
    {
      "remove": {
        "field": [
          "destination.nat.ip",
          "destination.nat.port"
        ],
        "if": "ctx?.destination?.nat?.ip == \"0.0.0.0\" && ctx?.destination?.nat?.port == 0"
      }
    },
    {
      "remove": {
        "field": "event.original",
        "if": "ctx?.tags == null || !(ctx.tags.contains('preserve_original_event'))",
        "ignore_failure": true,
        "ignore_missing": true
      }
    }
  ],
  "on_failure": [
    {
      "append": {
        "field": "error.message",
        "value": "{{ _ingest.on_failure_message }} {{ _ingest.on_failure_processor_type }}"
      }
    },
    {
      "remove": {
        "field": [
          "_temp_",
          "_conf",
          "message"
        ],
        "ignore_missing": true
      }
    }
  ]
}
jamiehynds commented 1 year ago
Hey @LaZyDK - we're revisiting Cortex Data Lake at the moment. @NateUT99 has been trying to use our PANW integration with logs from Data Lake, but has noticed some oddities.
Curious if you've ever manage to use our integration with logs coming in via Data Lake, or do you still use your own pipeline above?
LaZyDK commented 1 year ago
I used the pipeline above but haven't used it for some time now. I could not use the Elastic integration at the time.
NateUT99 commented 1 year ago
@LaZyDK I seem to be able to parse most of the logs; however, I do know that Palo Alto professional services is doing some sort of "translation" on their side to make them look more like standard PAN-OS logs. This works -- for the most part -- but as @jamiehynds said I do see some oddities (ex. the "is decrypted" flag appears to be flipped... yes means no).
It would be nice if we could get an "official" integration that worked for the logs from CDL -- original (preferred) or "translated" -- and I am happy to provide samples of all of the log types if needed.
elastic / integrations

Palo Alto Cortex Data Lake #2818
