Netapp OnTap ems logs go to sc4s:fallback

DavidLopez-jr commented 2 weeks ago

Note: If your issue is not a bug or a feature request, please raise a support ticket through our support portal (Splunk.com > Support > Support Portal). This will help us resolve your issue more efficiently and provide you with better assistance. For more information on how to work with the Splunk Support, please refer to this guide.

**Was the issue replicated by support? No

**What is the sc4s version ? 3.26.1

**Which operating system (including its version) are you using for hosting SC4S?REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="8.9"

**Which runtime (Docker, Podman, Docker Swarm, BYOE, MicroK8s) are you using for SC4S? Podman

**Is there a pcap available? If so, would you prefer to attach it to this issue or send it to Splunk support? I do not have a pcap available but can provide tcpdump.

**Is the issue related to the environment of the customer or Software related issue? No

**Is it related to Data loss, please explain ?No Protocol? none Hardware specs? none

**Do we have all the default indexes created? No

Describe the bug A clear and concise description of what the bug is. Our SC4S host is collecting events from a NetApp OnTap host sending ems logs. The logs are being sent to sourcetype=sc4s:fallback. It appears the SC4S filter for the source is misconfigured. It appears the filtering/parser was based on audit logs.

Steps to reproduce the behavior:

Go to 'send netapp ems logs'
Click on 'perform splunk search'
Scroll down to 'view results'
See error

cwadhwani-splunk commented 1 week ago

Hi @DavidLopez-jr Could you please provide the PCAP file, so that we can look at the raw logs and try to reproduce the issue on our environment. Please create a support ticket and attach the pcap file/tcpdump, so that we can ge the raw logs to move forward.

DavidLopez-jr commented 1 week ago

I added the tcpdump to case # 3585166 Netapp OnTap ems logs go to sc4s:fallback #2610.

From: cwadhwani-splunk @.> Sent: Tuesday, October 8, 2024 04:19 AM To: splunk/splunk-connect-for-syslog @.> Cc: Lopez, David (US) @.>; Mention @.> Subject: EXTERNAL: Re: [splunk/splunk-connect-for-syslog] Netapp OnTap ems logs go to sc4s:fallback (Issue #2610)

Hi @DavidLopez-jrhttps://github.com/DavidLopez-jr Could you please provide the PCAP file, so that we can look at the raw logs and try to reproduce the issue on our environment. Please create a support ticket and attach the pcap file/tcpdump, so that we can ge the raw logs to move forward.

— Reply to this email directly, view it on GitHubhttps://github.com/splunk/splunk-connect-for-syslog/issues/2610#issuecomment-2399314069, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ANLGXA6PFC7BILWX2HRLTW3Z2OPQPAVCNFSM6AAAAABPKNFUQGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJZGMYTIMBWHE. You are receiving this because you were mentioned.Message ID: @.**@.>>

cwadhwani-splunk commented 5 days ago

Hi @DavidLopez-jr I checked the existing parser written for netapp:ontap (ontap:ems), the log that has been provided by you seems a bit different than the sample log that we have used for writting the parser. We will need a pcap file to confirm the format of the ems logs coming from the netapp ontap. I tried to go online and find a sample syslog log of ems type but could not find it, so a pcap file would be helpful. We can check and update our parser if required.

Also, looking at the tags that you have provided in the GitHub issue,

wireformat:rfc|wireformat:rfc3164|vps|.app.app-fallbackz-lastchance|.app.app-fix-invalid-program-z_bsdconvention|ns_vendor:netapp|ns_product:ontap|.source.s_NETAPP_ONTAP

it seems like you are using some additional env parameters and local parsers. We will need the env file and the local parsers (/opt/sc4s/local folder) to check what you are facing.

Please provide these details (PCAP, env file (redact the sensitive details), local parsers) on the support case that you opened. Support can help you to get these details.

cwadhwani-splunk commented 1 day ago

We reviewed the current parser for NetApp ONTAP within SC4S. The existing parser is designed for the RFC 3164 syslog format, the parser expects the date and time within the message itself. However, the logs you've shared are in RFC 5424 format, and it doesn't meet the parser's criteria. As a result, the current parser isn't applicable to the current syslog and sourcetype, and the field structures differ.

To address this, we can provide a postfilter for local usage that can correctly classify the shared logs. Note that this post-filter assumes that NetApp ONTAP logs are sent to a specific port as it is currently configured in your env file (SC4S_LISTEN_NETAPP_ONTAP_UDP_PORT=5090).

Alternatively, you could check if there's a way to configure NetApp ONTAP to send logs in RFC 3164 format. This would eliminate the need for an additional parser.

Here is the postfilter:

Add a file in: /opt/sc4s/local/config/app_parsers/app-postfilter-netapp_ontap.conf

block parser app-postfilter-netapp_ontap() {
    channel {
        rewrite {
            r_set_splunk_dest_default(
                index("infraops")
                sourcetype('ontap:ems')
                vendor("netapp")
                product("ontap")
                class('ems')
            );
        };
    };
};

application app-postfilter-netapp_ontap[sc4s-postfilter] {
    filter {
        match("netapp", value('.netsource.sc4s_vendor'), type(string))
        and filter(f_is_rfc5424);
    };  
    parser { app-postfilter-netapp_ontap(); };
};

Please make sure to restart your sc4s service after making these changes.

splunk / splunk-connect-for-syslog

Netapp OnTap ems logs go to sc4s:fallback #2610