SC4S does not recognized Palo Alto Version 11 logs

[Jaxjohnny]

Was the issue replicated by support? - Yes. Splunk support confirmed, SC43S recognized the new Palo Alto Panorama logs as syslog, and therefore placing the events in the nix:syslog sourcetype

What is the sc4s version ? trying to figure this out, we changed to 3.2.2 on two SC4S servers. The rest are at 2.2

Is there a pcap available? - yes, i have one attached to the splunk support case, and i can get another one. I can "follow tcp stream" and then sanitize it for you.

Is the issue related to the environment of the customer or Software related issue? - The issue happened after the Palo Alto upgrade to version 11. We have one solitary version 10 server, non-prod, that was not upgraded. It is still coming in and recognized by SC4S as palo

Is it related to Data loss, please explain ? - we thought we were losing data, but it was actually going to the nix:syslog index instead of the palo alto indexes

Protocol? Hardware specs? - TCP 601

Last chance index/Fallback index? -

Is the issue related to local customization? - no, Palo upgrade

Do we have all the default indexes created? - yes.

Describe the bug A clear and concise description of what the bug is.

the following keys are set in splunk_metadata.csv

nix_syslog,index,nix_log_general nix_syslog,sc4s_template,t_standard pan_panos_log,index,firewall_palo_log pan_panos_globalprotect,index,firewall_palo_globalprotect pan_panos_traffic,index,firewall_palo_traffic pan_panos_threat,index,firewall_palo_threat pan_panos_system,index,firewall_palo_system pan_panos_config,index,firewall_palo_config pan_panos_hipmatch,index,firewall_palo_hipmatch pan_panos_correlation,index,firewall_palo_corelation pan_panos,index,firewall_palo_decryption

Most of the data is going into the nix_log_general index.

[Jaxjohnny]

Thank you for placing all the questions in the issue request. That was very clever of you.

[Jaxjohnny]

the code upgrade was from 10.2.7-h3 to pan-os 11.1.2

[Jaxjohnny]

The Splunk support case we opened is 3450268 They recommended we open an issue with this forum.

[ikheifets-splunk]

hello, @Jaxjohnny! Most of the team on Easter holidays, we will take a look little bit later

[Jaxjohnny]

Thank you so much. Sent from my T-Mobile 5G Device -------- Original message --------From: Ilya @.> Date: 4/2/24 8:06 AM (GMT-05:00) To: splunk/splunk-connect-for-syslog @.> Cc: Jaxjohnny @.>, Mention @.> Subject: Re: [splunk/splunk-connect-for-syslog] SC4S does not recognized Palo Alto Version 11 logs (Issue #2382) hello, @Jaxjohnny! Most of the team on Easter holidays, we will take a look little bit later

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[Jaxjohnny]

Thank you.

[Jaxjohnny]

one note here, we do have a single palo alto firewall in their lab environment that was NOT upgraded. It is still working perfectly

[mstopa-splunk]

hi @Jaxjohnny I reproduced the problem:

echo "<14>1 2024-03-28T17:24:00+00:00 XXXX.xxx.xxx - - - -  1,2024/03/28 17:23:59,000000000000,USERID,login,2562,2024/03/28 17:23:58,vsys1,1.1.1.1,xxxxxxxxxxxxxxx.xxxxxx.xxxxxxx,XXXXXXXXXXXXX,0,1,36000,0,0,agent,,000000000000000000,0x8000000000000000,3866,4335,4336,0,,xx-xxxxxxxxxxx,1,,2024/03/28 17:23:57,1,0x0,xxxxxxxxxxxxxxx.xxxxxx.xxxxxxx,,2024-03-28T17:23:58.636+00:00,,client-to-server," > /dev/udp/0.0.0.0/514

In previous versions PAN panos format was like this:

echo "<14>1 2024-03-28T17:24:00+00:00 XXXX.xxx.xxx 1,2024/03/28 17:23:59,000000000000,USERID,login,2562,2024/03/28 17:23:58,vsys1,1.1.1.1,xxxxxxxxxxxxxxx.xxxxxx.xxxxxxx,XXXXXXXXXXXXX,0,1,36000,0,0,agent,,000000000000000000,0x8000000000000000,3866,4335,4336,0,,xx-xxxxxxxxxxx,1,,2024/03/28 17:23:57,1,0x0,xxxxxxxxxxxxxxx.xxxxxx.xxxxxxx,,2024-03-28T17:23:58.636+00:00,,client-to-server," > /dev/udp/0.0.0.0/514

In the v11 the vendor added skipped fields for APP-NAME PROCID MSGID and SDATA: <14>1 2024-03-28T17:24:00+00:00 XXXX.xxx.xxx 1, -> <14>1 2024-03-28T17:24:00+00:00 XXXX.xxx.xxx - - - - 1, their idea is to make their messages RFC5424 compliant

but there are two whitespaces between the last skipped field and the message, so their log format is not RFC5424 compliant and syslog-ng's syslog parser doesn't extract the message field correctly. This will work for you:

echo "<14>1 2024-03-28T17:24:00+00:00 XXXX.xxx.xxx - - - - 1,2024/03/28 17:23:59,000000000000,USERID,login,2562,2024/03/28 17:23:58,vsys1,1.1.1.1,xxxxxxxxxxxxxxx.xxxxxx.xxxxxxx,XXXXXXXXXXXXX,0,1,36000,0,0,agent,,000000000000000000,0x8000000000000000,3866,4335,4336,0,,xx-xxxxxxxxxxx,1,,2024/03/28 17:23:57,1,0x0,xxxxxxxxxxxxxxx.xxxxxx.xxxxxxx,,2024-03-28T17:23:58.636+00:00,,client-to-server," > /dev/udp/0.0.0.0/514

Please request fixing the log format on the vendor side

[mstopa-splunk]

@Jaxjohnny in the meantime try this postfilter for local, temporary solution:

block parser app-postfilter-pan_panos() {
 channel {
    rewrite {
        set("$(strip $MSG )", value("MSG"));
    };
    parser { app-syslog-pan_panos(); };
 };
};

application app-postfilter-pan_panos[sc4s-postfilter] {
    filter {
        "${PROGRAM}" eq ""
        and message('^ 1,[^,]+,[^,]+,[A-Z]+\,')
        ;
    };
    parser { app-postfilter-pan_panos(); };
};

[Jaxjohnny]

Thank you So if place this in the /opt/sc4s/local/config/app_parsers/syslog, will the splunk_metadata.csv then work? Or do we also need host match filters?

[Jaxjohnny]

Where do i add specifics for indexes and sourcetypes and hosts? Does this make the splunk_metadata work? Or should I remove those entries?

[Jaxjohnny]

i tried this parser, but the data is now going into the netfw index

block parser app-postfilter-pan_panos_traffic() {
 channel {
    rewrite {
        set("$(strip $MSG )", value("MSG"));

    };

    rewrite {
            r_set_splunk_dest_default(
                sourcetype('pan:traffic')
                index('ent_inf_firewall_palo_traffic')
                 );};

    parser { app-syslog-pan_panos(); };
 };
};

application app-postfilter-pan_panos_traffic[sc4s-postfilter] {
    filter {

       ((match("10.13.4.6" value("HOST") type(string)) or match("10.13.4.7" value("HOST") type(string)) or
        match("10.13.4.8" value("HOST") type(string)) or match("10.11.4.6" value("HOST") type(string)) or
        match("10.11.4.7" value("HOST") type(string)) or match("10.11.4.8" value("HOST") type(string)) or
        match("la4panlc01*" value("HOST") type(string) flags(ignore-case)) or match("la4panlc02*" value("HOST") type(string) flags(ignore-case)) or
        match("la4panlc03*" value("HOST") type(string) flags(ignore-case)) or match("at1panlc01*" value("HOST") type(string) flags(ignore-case)) or
        match("at1panlc02*" value("HOST") type(string) flags(ignore-case)) or match("at1panlc03*" value("HOST") type(string) flags(ignore-case))) and match("TRAFFIC" value("MESSAGE")))

        and

        ("${PROGRAM}" eq ""
        and message('^ 1,[^,]+,[^,]+,[A-Z]+\,'))
        ;
    };
    parser { app-postfilter-pan_panos_traffic(); };
};

[mstopa-splunk]

@Jaxjohnny my apologies, I forgot that the solution was supposed to work with splunk_metadata.csv.

postilfters are applied after the metadata csv, so let's switch to an earlier topic:

block parser app-syslog-strip-pan_panos() {
 channel {
    rewrite {
        set("$(strip $MSG )", value("MSG"));
    };
    parser { app-syslog-pan_panos(); };
 };
};

application app-syslog-strip-pan_panos[sc4s-syslog] {
    filter {
        "${PROGRAM}" eq ""
        and message('^ 1,[^,]+,[^,]+,[A-Z]+\,')
        ;
    };
    parser { app-syslog-strip-pan_panos(); };
};

After copying it to your local sc4s directory and restarting the service Palo Alto v11 logs should behave like v10 (please keep your current splunk_metadata.csv). Please try on your end and let me know.

[Jaxjohnny]

Thank you. That seems to have worked out of the box.

[Jaxjohnny]

Thank you so much