[PANW] source.user.email contain information that are not email addresses

[herrBez]

According to the documentation the PANW integration (in particular, the Threat dataset) populates the (non-ECS?) fields source.user.email and destination.user.email, which should be two EMail addresses.

The information added in these two fields are the "Sender" and "Recipient" of the Palo Alto's Threat Log Field Documentation, which are ambiguously documented as:

Specifies the name of the sender of an email.

and

Specifies the name of the receiver of an email.

respectively.

A customer noted that the field is (sometimes?) filled with information that are not EMail Addresses. Some examples are reported in the following:

EXAMPLE 1

EXAMPLE 2 source.user.email contains a valid URL

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

@herrBez Are you able to obtain a sanitized copy of the complete documents that show this behaviour, preferably with event.original if possible.

[herrBez]

Hi @efd6, sorry for the late response. I hope that the message field is sufficient.

Sample Event

```json { "_index": ".ds-filebeat-8.4.3-2022.11.16-000549", "_id": "T7wugYQBQP-Pwsn4ANBr", "_version": 1, "_score": 0, "_source": { "agent": { "name": "ELK02", "id": "7c96e4df-bd16-4127-9fb4-be97dee87234", "type": "filebeat", "ephemeral_id": "4662fa1e-df4d-4949-8cdf-ddf4b829b4f4", "version": "8.4.3" }, "log": { "file": { "path": "/var/log/elk/syslog/fw/palo/TESTFW123a.log" }, "offset": 2114224085, "level": "informational" }, "destination": { "geo": { "name": "Netherlands" }, "port": 443, "ip": "1.1.1.1", "user": { "email": "social-networking,low-risk" } }, "rule": { "name": "Rule Name", "uuid": "ab1vf234-3046-4352-b4a2-1a1aa1234a12" }, "source": { "geo": { "name": "10.0.0.0-10.255.255.255" }, "port": 54935, "ip": "10.10.10.10", "user": { "email": "social-networking,low-risk" } }, "message": "10.10.10.10,1.1.1.1,0.0.0.0,0.0.0.0,Rule Name,,,http-proxy,vsys1,LAN,Zscaler,ethernet1/1.205,tunnel.340,ABC-Log,2022/11/16 17:03:06,36430490,1,54935,443,0,0,0x2b000,tcp,alert,\"www.example.com/\",(9999),computer-and-internet-info,informational,client-to-server,7097204421617394645,0x8000000000000000,10.0.0.0-10.255.255.255,Netherlands,,,0,,\"social-networking,low-risk\",1,Windows ............................................................ Windows 10 .............................................................................. N ZTunnel/1.0,,\"social-networking,low-risk\",\"social-networking,low-risk\",\"social-networking,low-risk\",\"social-networking,low-risk\",\"social-networking,low-risk\",0,12,0,0,0,,TESTFW123a,\"social-networking,low-risk\",,,connect,0,,0,,N/A,unknown,AppThreat-0-0,0x0,0,4294967295,\"social-networking,low-risk\",\"computer-and-internet-info,low-risk\",ab1vf234-3046-4352-b4a2-1a1aa1234a12,0,,,,,,,,,,,,,,,,,,,,,,,,,,,\"computer-and-internet-info,low-risk\",\"computer-and-internet-info,low-risk\",0,2022-11-16T17:03:06.291+01:00,,\"computer-and-internet-info,low-risk\",,proxy,networking,browser-based,5,\"evasive-behavior,used-by-malware,able-to-transfer-file,has-known-vulnerability,tunnel-other-application,prone-to-misuse,pervasive-use\",,http-proxy,no,no", "panw": { "panos": { "payload_protocol_id": "4294967295", "subject": "social-networking,low-risk", "logged_time": "2022-11-16T17:03:06.000+01:00", "repeat_count": 1, "type": "THREAT", "url_idx": "1", "flow_id": "36430490", "wildfire": { "report_id": "0", "name": "social-networking,low-risk" }, "action": "alert", "ruleset": "Rule Name", "action_flags": "0x8000000000000000", "x_forwarded_for": "social-networking,low-risk", "content_version": "AppThreat-0-0", "http_headers": "social-networking,low-risk", "dst": { "dynamic_address_group": "computer-and-internet-info,low-risk" }, "http2_connection": "0", "url_category_list": "computer-and-internet-info,low-risk", "imsi": "0", "threat_category": "unknown", "log_profile": "ABC-Log", "sub_type": "url", "justification": "computer-and-internet-info,low-risk", "tunnel_type": "N/A", "device_group_hierarchy4": "0", "src": { "dynamic_address_group": "computer-and-internet-info,low-risk" }, "partial_hash": "0", "high_resolution_timestamp": "2022-11-16T17:03:06.291+01:00", "device_group_hierarchy1": "12", "url": { "category": "computer-and-internet-info" }, "device_group_hierarchy2": "0", "device_group_hierarchy3": "0", "virtual_sys": "vsys1", "sequence_number": "7097204421617394645", "application": { "risk_level": 5, "characteristics": "evasive-behavior,used-by-malware,able-to-transfer-file,has-known-vulnerability,tunnel-other-application,prone-to-misuse,pervasive-use", "tunneled": "http-proxy", "sub_category": "proxy", "is_sanctioned": "no", "is_saas": "no", "technology": "browser-based", "category": "networking" }, "parent_session": { "id": "0" }, "sctp": { "assoc_id": "0" }, "threat": { "name": "URL-filtering", "id": "9999" } } }, "url": { "original": "www.example.com/" }, "tags": [ "forwarded", "_geoip_database_unavailable_GeoLite2-City.mmdb", "_geoip_database_unavailable_GeoLite2-City.mmdb", "_geoip_database_unavailable_GeoLite2-City.mmdb", "_geoip_database_unavailable_GeoLite2-City.mmdb", "_geoip_database_unavailable_GeoLite2-ASN.mmdb", "_geoip_database_unavailable_GeoLite2-ASN.mmdb", "_geoip_database_unavailable_GeoLite2-ASN.mmdb", "_geoip_database_unavailable_GeoLite2-ASN.mmdb" ], "network": { "community_id": "1:POz+sVVVzZXASZTG9avmUorDzmg=", "application": "http-proxy", "transport": "tcp", "type": "ipv4", "direction": "inbound" }, "labels": { "container_page": true, "temporary_match": true, "client_server_policy_based_forwarding": true }, "input": { "type": "filestream" }, "observer": { "ingress": { "zone": "LAN", "interface": { "name": "ethernet1/1.205" } }, "product": "PAN-OS", "hostname": "TESTFW123a", "vendor": "Palo Alto Networks", "serial_number": "123456789123", "type": "firewall", "egress": { "zone": "Zscaler", "interface": { "name": "tunnel.340" } } }, "@timestamp": "2022-11-16T17:03:06.000+01:00", "ecs": { "version": "8.4.0" }, "related": { "hosts": [ "TESTFW123a" ], "ip": [ "10.10.10.10", "1.1.1.1", "0.0.0.0" ] }, "http": { "request": { "referrer": "social-networking,low-risk", "method": "connect" } }, "event": { "severity": 5, "timezone": "+01:00", "created": "2022-11-16T17:03:06.000+01:00", "kind": "alert", "module": "panw", "action": "url_filtering", "category": [ "intrusion_detection", "threat", "network" ], "type": [ "allowed" ], "dataset": "panw.panos", "outcome": "success" }, "user_agent": { "original": "Windows ............................................................ Windows 10 .............................................................................. N ZTunnel/1.0", "os": { "name": "Windows", "version": "10", "full": "Windows 10" }, "name": "Other", "device": { "name": "Other" } } }, "fields": { "destination.geo.name": [ "Netherlands" ], "event.category": [ "intrusion_detection", "threat", "network" ], "user_agent.original.text": [ "Windows ............................................................ Windows 10 .............................................................................. N ZTunnel/1.0" ], "observer.egress.interface.name": [ "tunnel.340" ], "source.user.email": [ "social-networking,low-risk" ], "panw.panos.action_flags": [ "0x8000000000000000" ], "panw.panos.partial_hash": [ "0" ], "panw.panos.repeat_count": [ 1 ], "observer.vendor": [ "Palo Alto Networks" ], "source.ip": [ "10.10.10.10" ], "panw.panos.type": [ "THREAT" ], "agent.name": [ "ELK02" ], "network.community_id": [ "1:POz+sVVVzZXASZTG9avmUorDzmg=" ], "panw.panos.high_resolution_timestamp": [ "2022-11-16T17:03:06.291+01:00" ], "event.outcome": [ "success" ], "user_agent.original": [ "Windows ............................................................ Windows 10 .............................................................................. N ZTunnel/1.0" ], "event.severity": [ 5 ], "input.type": [ "filestream" ], "agent.hostname": [ "ELK02" ], "tags": [ "forwarded", "_geoip_database_unavailable_GeoLite2-City.mmdb", "_geoip_database_unavailable_GeoLite2-City.mmdb", "_geoip_database_unavailable_GeoLite2-City.mmdb", "_geoip_database_unavailable_GeoLite2-City.mmdb", "_geoip_database_unavailable_GeoLite2-ASN.mmdb", "_geoip_database_unavailable_GeoLite2-ASN.mmdb", "_geoip_database_unavailable_GeoLite2-ASN.mmdb", "_geoip_database_unavailable_GeoLite2-ASN.mmdb" ], "panw.panos.application.risk_level": [ 5 ], "source.port": [ 54935 ], "agent.id": [ "7c96e4df-bd16-4127-9fb4-be97dee87234" ], "panw.panos.sequence_number": [ 7097204421617395000 ], "panw.panos.tunnel_type": [ "N/A" ], "panw.panos.log_profile": [ "ABC-Log" ], "observer.egress.zone": [ "Zscaler" ], "panw.panos.ruleset": [ "Rule Name" ], "panw.panos.url_idx": [ "1" ], "destination.port": [ 443 ], "panw.panos.application.is_saas": [ "no" ], "panw.panos.http_headers": [ "social-networking,low-risk" ], "user_agent.os.full": [ "Windows 10" ], "user_agent.os.name": [ "Windows" ], "labels.client_server_policy_based_forwarding": [ true ], "panw.panos.justification": [ "computer-and-internet-info,low-risk" ], "labels.temporary_match": [ true ], "agent.type": [ "filebeat" ], "panw.panos.logged_time": [ "2022-11-16T17:03:06.000+01:00" ], "related.ip": [ "10.10.10.10", "1.1.1.1", "0.0.0.0" ], "network.application": [ "http-proxy" ], "panw.panos.sub_type": [ "url" ], "panw.panos.src.dynamic_address_group": [ "computer-and-internet-info,low-risk" ], "panw.panos.subject": [ "social-networking,low-risk" ], "observer.product": [ "PAN-OS" ], "labels.container_page": [ true ], "panw.panos.x_forwarded_for": [ "social-networking,low-risk" ], "panw.panos.imsi": [ "0" ], "panw.panos.application.is_sanctioned": [ "no" ], "destination.ip": [ "1.1.1.1" ], "observer.hostname": [ "TESTFW123a" ], "panw.panos.http2_connection": [ "0" ], "panw.panos.url.category": [ "computer-and-internet-info" ], "traefik.access.user_agent.os_name": [ "Windows" ], "event.action": [ "url_filtering" ], "@timestamp": [ "2022-11-16T16:03:06.000Z" ], "log.file.path": [ "/var/log/elk/syslog/fw/palo/TESTFW123a.log" ], "source.geo.name": [ "10.0.0.0-10.255.255.255" ], "agent.ephemeral_id": [ "4662fa1e-df4d-4949-8cdf-ddf4b829b4f4" ], "user_agent.device.name": [ "Other" ], "panw.panos.sctp.assoc_id": [ "0" ], "panw.panos.url_category_list": [ "computer-and-internet-info,low-risk" ], "panw.panos.application.characteristics": [ "evasive-behavior,used-by-malware,able-to-transfer-file,has-known-vulnerability,tunnel-other-application,prone-to-misuse,pervasive-use" ], "panw.panos.parent_session.id": [ "0" ], "observer.ingress.interface.name": [ "ethernet1/1.205" ], "url.original.text": [ "www.example.com/" ], "user_agent.os.version": [ "10" ], "panw.panos.content_version": [ "AppThreat-0-0" ], "traefik.access.user_agent.name": [ "Other" ], "panw.panos.application.tunneled": [ "http-proxy" ], "panw.panos.device_group_hierarchy3": [ "0" ], "panw.panos.device_group_hierarchy4": [ "0" ], "panw.panos.device_group_hierarchy1": [ "12" ], "panw.panos.device_group_hierarchy2": [ "0" ], "http.request.method": [ "connect" ], "traefik.access.user_agent.original": [ "Windows ............................................................ Windows 10 .............................................................................. N ZTunnel/1.0" ], "panw.panos.application.category": [ "networking" ], "log.level": [ "informational" ], "panw.panos.application.sub_category": [ "proxy" ], "panw.panos.wildfire.name": [ "social-networking,low-risk" ], "event.kind": [ "alert" ], "rule.name": [ "Rule Name" ], "panw.panos.action": [ "alert" ], "panw.panos.application.technology": [ "browser-based" ], "user_agent.name": [ "Other" ], "log.offset": [ 2114224085 ], "observer.serial_number": [ "123456789123" ], "panw.panos.flow_id": [ "36430490" ], "observer.type": [ "firewall" ], "ecs.version": [ "8.4.0" ], "event.created": [ "2022-11-16T16:03:06.000Z" ], "agent.version": [ "8.4.3" ], "destination.user.email": [ "social-networking,low-risk" ], "related.hosts": [ "TESTFW123a" ], "panw.panos.wildfire.report_id": [ "0" ], "panw.panos.threat_category": [ "unknown" ], "user_agent.os.name.text": [ "Windows" ], "panw.panos.threat.name": [ "URL-filtering" ], "event.module": [ "panw" ], "observer.ingress.zone": [ "LAN" ], "panw.panos.virtual_sys": [ "vsys1" ], "network.direction": [ "inbound" ], "event.timezone": [ "+01:00" ], "network.type": [ "ipv4" ], "http.request.referrer": [ "social-networking,low-risk" ], "panw.panos.threat.id": [ "9999" ], "message": [ "10.10.10.10,1.1.1.1,0.0.0.0,0.0.0.0,Rule Name,,,http-proxy,vsys1,LAN,Zscaler,ethernet1/1.205,tunnel.340,ABC-Log,2022/11/16 17:03:06,36430490,1,54935,443,0,0,0x2b000,tcp,alert,\"www.example.com/\",(9999),computer-and-internet-info,informational,client-to-server,7097204421617394645,0x8000000000000000,10.0.0.0-10.255.255.255,Netherlands,,,0,,\"social-networking,low-risk\",1,Windows ............................................................ Windows 10 .............................................................................. N ZTunnel/1.0,,\"social-networking,low-risk\",\"social-networking,low-risk\",\"social-networking,low-risk\",\"social-networking,low-risk\",\"social-networking,low-risk\",0,12,0,0,0,,TESTFW123a,\"social-networking,low-risk\",,,connect,0,,0,,N/A,unknown,AppThreat-0-0,0x0,0,4294967295,\"social-networking,low-risk\",\"computer-and-internet-info,low-risk\",ab1vf234-3046-4352-b4a2-1a1aa1234a12,0,,,,,,,,,,,,,,,,,,,,,,,,,,,\"computer-and-internet-info,low-risk\",\"computer-and-internet-info,low-risk\",0,2022-11-16T17:03:06.291+01:00,,\"computer-and-internet-info,low-risk\",,proxy,networking,browser-based,5,\"evasive-behavior,used-by-malware,able-to-transfer-file,has-known-vulnerability,tunnel-other-application,prone-to-misuse,pervasive-use\",,http-proxy,no,no" ], "network.transport": [ "tcp" ], "rule.uuid": [ "ab1vf234-3046-4352-b4a2-1a1aa1234a12" ], "user_agent.os.full.text": [ "Windows 10" ], "panw.panos.payload_protocol_id": [ "4294967295" ], "url.original": [ "www.example.com/" ], "event.type": [ "allowed" ], "panw.panos.dst.dynamic_address_group": [ "computer-and-internet-info,low-risk" ], "event.dataset": [ "panw.panos" ] } } ```

[efd6]

Thanks @herrBez. The pipeline looks like it is doing what is intended from the PANW documentation for the Threat log data there. The docs describe the relevant fields as "Sender (sender)" -> "Specifies the name of the sender of an email." and "Recipient (recipient)" -> "Specifies the name of the receiver of an email.", but these fields in the messages have the text "social-networking,low-risk" (which also appears in many other fields in ways that don't really make sense). Do we know where this field value is coming from?

[botelastic[bot]]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!