Added PanOS 11 syslog standard fields; repaired broken field extracts & name collisions

[jwiley80]

Description

Added PanOS 11 syslog standard fields; repaired broken field extracts & name collisions

How Has This Been Tested?

Tested in Splunk against large-scale existing pan:* data flows

Types of changes

• Breaking change 1: In default/transforms.conf, [extract_userid] previously omitted the "src_user" field early in the message, causing all fields following it to parse incorrectly. This change includes the "src_user" field correctly, and causes all fields after it to parse correctly. The may cause breakage for some users, dashboards, or other use cases dependent the currently incorrect field assignments.

• Breaking change 2: In default/transforms.conf, [extract_config] previously included the "devicegroup_level3" and "devicegroup_level4" fields that do not exist in the log data. All fields following these extracts have been parsing incorrectly. This change correctly excludes the "devicegroup_level3" and "devicegroup_level4" fields to match the data correctly, and causes all fields after that point to parse correctly. The may cause breakage for some users, dashboards, or other use cases dependent the currently incorrect field assignments.

• Bug fixes:

In props.conf

1. pan:config - field aliases and evals added for CIM mapping and compatibility
2. pan:globalprotect - field aliases and evals added for CIM mapping and compatibility; aliased "time_generated" to "generated_time" for consistency with other pan sourcetype naming
3. pan:hipmatch - field aliases and evals added for CIM mapping and compatibility
4. pan:system - field aliases and evals added for CIM mapping and compatibility, added description extracts
5. pan:userid - added Field Aliases to match corrected Transforms extracts

In transforms.conf

1. extract_traffic - updated to PanOS11 syslog fields in TechDocs; used "host_id", "host_serial", "nssai_sd" and "nssai_sst" instead of current "hostid", "serialnumber", "nsdsai_sd" and "nsdsai_sst" in TechDocs
2. extract_threat - updated to PanOS11 syslog fields in TechDocs; used "host_id" and "host_serial" instead of current "hostid" and "serialnumber" in TechDocs
3. extract_system - updated to PanOS11 syslog fields in TechDocs (only added "high_res_timestamp")
4. extract_hipmatch- updated to PanOS11 syslog fields in TechDocs; used "host_id" and "host_serial" instead of current "hostid" and "serialnumber" in TechDocs
5. extract_globalprotect - updated to PanOS11 syslog fields in TechDocs; used "host_id" and "host_serial" instead of current "hostid" and "serialnumber" in TechDocs. In extract_globalprotect, the old version uses 'serial_number' for this field, which collides with field 3, which is the 'dvc_serial', not the serial of the src/user asset being described in the log

Note:

1. The fields "host_id" and "host_serial" in extract_threat, extract_traffic, extract_globalprotect, and extract_hipmatch are extremely useful for asset correlation, and needs to be consistently named for analysis.
2. I changed field names from 'dst...' to 'dest...' for Splunk CIM compatibility. Both can be used with field aliasing if preferred, but dest is more consistent with the existing Splunk CIM.

[welcome-to-palo-alto-networks[bot]]

:tada: Thanks for opening this pull request! We really appreciate contributors like you! :raised_hands:

[btorresgil]

Thanks for the PR! Seeing a lot of good changes here. With the breaking changes we'd have to do a major release with comprehensive release note so give us some time to go through everything and plan.

@jwiley80 Can you remove any lines you commented out and do another commit/push? We'll see the lines are removed in the diff during review but having them still exist commented makes the diff harder to parse.

Thanks again!

[jwiley80]

Any progress on this? I'm not sure if you're waiting on something from me.