search

PaloAltoNetworks / Splunk-Apps

Palo Alto Networks App for Splunk leverages the data visibility provided by Palo Alto Networks next-generation firewalls and endpoint security with Splunk's extensive investigation and visualization capabilities to deliver an advanced security reporting and analysis tool.
https://pan.dev/splunk/docs/
ISC License
106 stars 49 forks source link

Logs not being properly parsed when shipped from Panorama to Splunk #305

Open NMCPaul opened 1 year ago

NMCPaul commented 1 year ago

Describe the bug

Field extractions are not working properly on logs sent from Palo Alto Panorama v10.2.4-h4 to Splunk v9.0.2 with the Palo Alto app/add-on v8.1.0.

Expected behavior

Field extractions and transforms properly parse the data from the logs.

Current behavior

Field extraction are not working correctly. An example of this would be the word “deny” being assign to the transport field rather than the protocol of tcp, or udp, etc. Looking at the transforms.conf file of the add-on, I see the first field in many of the [extract_*] stanzas are defined as “future_use1” as shown below.

FIELDS="future_use1","receive_time","serial_number","log_type","log_subtype","version","generated_time","src_ip","dest_ip","src_translated_ip","dest_translated_ip","rule","src_user","dest_user","app","vsys","src_zone","dest_zone","src_interface","dest_interface","log_forwarding_profile","future_use3","session_id","repeat_count","src_port","dest_port","src_translated_port","dest_translated_port","session_flags","transport","action","misc","threat","raw_category","severity","direction","sequence_number","action_flags","src_location","dest_location","future_use4","content_type","pcap_id","file_hash","cloud_address","url_index","user_agent","file_type","xff","referrer","sender","subject","recipient","report_id","devicegroup_level1","devicegroup_level2","devicegroup_level3","devicegroup_level4","vsys_name","dvc_name","future_use5","src_vm","dest_vm","http_method","tunnel_id","tunnel_monitor_tag","tunnel_session_id","tunnel_start_time","tunnel_type","threat_category","content_version","future_use6"

Each event in the data I am receiving from my firewall begins with a timestamp which is the second field listed in the extraction. I removed “future_use1” from the globalprotect, threat, traffic, userid, correlation, hipmatch, system, and config extractions, and now they seem to be working better.

In conjunction with the above change, I also had to alter the regexes which set the sourcetype on my events. The default configuration looks like the following:

DEST_KEY = MetaData:Sourcetype
REGEX = ^[^,]+,[^,]+,[^,]+,TRAFFIC,
FORMAT = sourcetype::pan:traffic

Since my data starts with the ”receive_time” field, it is necessary to change the regex to the following for the appropriate sourcetypes to be assigned.

REGEX = ^[^,]+,[^,]+,TRAFFIC,

Possible solution

Modify the regexes as noted above.

Steps to reproduce

  1. The following Panorama entries (one of each type), when sent from Panorama to Splunk, do not get processed correctly if run through the un-modified plugins:

13:22:49,000702698924,USERID,login,1,2023/08/28 13:22:49,,207.74.235.202,ad\cook370,User-ID-Agent,0,1,2700,0,0,agent,,7270585355236084506,0x0,0,0,0,0,,Panorama,0,,2023/08/28 13:22:46,1,0x0,ad\cook370,,1969-12-31T19:00:00.000-05:00

13:22:51,016401010483,THREAT,file,2562,2023/08/28 13:22:43,198.108.197.113,192.122.186.130,0.0.0.0,0.0.0.0,Allow-Inside-to-Internet-AppDefault-1,,,netflix-base,vsys1,Inside,Outside,ethernet1/14,ethernet1/13,default,2023/08/28 13:22:43,1778831,1,45279,80,0,0,0x2000,tcp,alert,"AAAABSa4Ov8t2867atBdCPWOsUcZeeBkOFUFjD1jACIJVWYDHsoU85K0IqgHd0A",WebP Image File(52275),streaming-media,low,server-to-client,7270675961878125631,0x8000000000000000,Region-198.108.19x.x,United States,,,0,,,195,,,,,,,,0,94,83,0,0,,External-PA-3260-Primary,occ-0-883-3419.1.nflxso.net/dnm/api/v6/tx1O544a9T7n8Z_G12qaboulQQE/AAAABSa4Ov8t2867atBdCPWOsUcZeeBkOFUFjD1jACIJVWYDHsoU85K0IqgHd0A2r3AR40TG7Nu3iYLRXdropQx1yhuer-0YTdZaDem7tkk5Tg.webp?r=030,,,,0,,0,,N/A,N/A,AppThreat-8748-8241,0x0,0,4294967295,,,01adbd9d-cfa3-4bfc-b4d4-b50dace4a9cf,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0,2023-08-28T13:22:43.077-04:00,,,,photo-video,media,browser-based,3,"evasive-behavior,consume-big-bandwidth,has-known-vulnerability,pervasive-use",netflix,netflix-base,no,no,

13:24:12,016401010483,GLOBALPROTECT,0,2562,2023/08/28 13:24:03,vsys1,gateway-tunnel-latency,tunnel,,,claerhc,US,DELL3070,97.91.54.255,0.0.0.0,97.91.54.255,0.0.0.0,412d4483-2d72-4a84-ab9e-c6485e4d8682,7JWRL43,6.0.4,any,,1,,,"Pre-tunnel latency: 32ms, Post-tunnel latency: 31ms",success,,0,,0,GP-GW,7270675961866100325,0x8000000000000000,2023-08-28T13:24:03.947-04:00,,,,,,94,83,0,0,,External-PA-3260-Primary,1

13:24:12,016401010448,HIPMATCH,0,2562,2023/08/28 13:24:03,ad\claerhc,vsys1,DELL3070,Windows,172.25.3.61,OS-Check-Windows,1,object,,,7270613427142266946,0x8000000000000000,94,83,0,0,,External-PA-3260-Secondary,1,0.0.0.0,412d4483-2d72-4a84-ab9e-c6485e4d8682,7JWRL43,,2023-08-28T13:24:03.952-04:00

13:24:31,000702698924,SYSTEM,general,0,2023/08/28 13:24:31,,general,,0,0,general,informational,"Connection to Update server: updates.paloaltonetworks.com completed successfully, initiated by 10.10.0.135",7270585106127961531,0x0,0,0,0,0,,Panorama,0,0,2023-08-28T13:24:31.000-04:00

Screenshots

n/a

Context

Trying to use data from our PA firewalls in order to set up reports and dashboards.

Your Environment

NMCPaul commented 1 year ago

Adding on to this, I have the following questions:

welcome-to-palo-alto-networks[bot] commented 1 year ago

:tada: Thanks for opening your first issue here! Welcome to the community!

jeffwalzer commented 1 year ago

Any update on this?

NMCPaul commented 1 year ago

Haven't heard anything from anyone yet. Are you running into the same thing?

jeffwalzer commented 1 year ago

Somewhat similar - having issues with events related to pan:firewall_cloud - but seems like no one from Palo has been responding to much of anything

NMCPaul commented 1 year ago

Just curious - is there a better method to get support or an actual reply for this?

jeffwalzer commented 1 year ago

Not that I know of - very disappointing to not hear anything back at all

jwiley80 commented 1 year ago

I second that sentiment

Jeremy Wiley 678-852-5789

From: JeffW @.> Sent: Wednesday, October 4, 2023 9:01:20 AM To: PaloAltoNetworks/Splunk-Apps @.> Cc: Subscribed @.***> Subject: Re: [PaloAltoNetworks/Splunk-Apps] Logs not being properly parsed when shipped from Panorama to Splunk (Issue #305)

Not that I know of - very disappointing to not hear anything back at all — Reply to this email directly, view it on GitHub [github. com], or unsubscribe [github. com]. You are receiving this because you are subscribed to this thread. Message

Not that I know of - very disappointing to not hear anything back at all

— Reply to this email directly, view it on GitHub [github.com]https://urldefense.com/v3/__https://github.com/PaloAltoNetworks/Splunk-Apps/issues/305*issuecomment-1746833000__;Iw!!MLsdJ25-fIk!umoavp_d04bPvLp8N19vumA4ZYsaerUBl8ADgtwPNv0cuVsIqtqFbUepuNf0Gm2FMdn6ol0jLe7JPwV6mgDDPf7bHJY$, or unsubscribe [github.com]https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AU44XXPGSJVOAXNNKJK3J5TX5VM2BAVCNFSM6AAAAAA4DIBJU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBWHAZTGMBQGA__;!!MLsdJ25-fIk!umoavp_d04bPvLp8N19vumA4ZYsaerUBl8ADgtwPNv0cuVsIqtqFbUepuNf0Gm2FMdn6ol0jLe7JPwV6mgDD2AgQpYI$. You are receiving this because you are subscribed to this thread.Message ID: @.***>