syslog messages do not parse correctly when IPv4 addresses are in process name

[RaygunDuck]

Example of the issue This line parses fine:

<14>Mar 25 19:08:39 ServerName conman[1324]: INFO conman - lhvpn-10.194.1.45-nodes-86 is now running successfully This does not, and is sent to the sc4s:fallback sourcetype <29>Mar 25 19:08:39 ServerName 10.192.1.45.nodes-86[4035]: Options error: --ca fails with '/etc/config/lhvpn/10.192.1.45.nodes-86.ca': No such file or directory (errno=2) I have observed this from multiple servers and log types.

[ryanfaircloth]

This is incorrect and not allowed in BSD format syslog. What is the source product have you communicated the defect to the vendor?

[RaygunDuck]

I have not. However, despite this not strictly following the TAG requirements for rfc3164 I am able to receive and process the log with standard syslog-ng ose without any specialty configurations. I have only had issues when using SC4S. If the field extractions did not work properly that wouldn't be a problem, but the log is mangled by sc4c before sending to the index.

The single line log of:

<29>Mar 25 19:08:39 ServerName 10.192.1.45.nodes-86[4035]: Options error: --ca fails with '/etc/config/lhvpn/10.192.1.45.nodes-86.ca': No such file or directory (errno=2) becomes a two line, single event with a _raw of: PRI=29 MESSAGE=10.192.1.45.nodes-86[4035]: Options error: --ca fails with '/etc/config/lhvpn/10.192.1.45.nodes-86.ca': No such file or directory (errno=2)

[rjha-splunk]

sc4s fallback is for the sourcetypes which are not identified by the parsers and thats why it adds PRI and message.

[RaygunDuck]

Regardless, SC4S is still providing less functionality than syslog-ng writing out files that are picked up by a monitor input

[rjha-splunk]

All the supported vendors are documented , if something is breaking please feel free to share the sample sanitised log and we will try to fix it. the fallback details are documented here :https://splunk.github.io/splunk-connect-for-syslog/main/sources/#the-sc4s-fallback-sourcetype

[RaygunDuck]

I have provided this. What additional information do you require?

[rjha-splunk]

source product for example.

[RaygunDuck]

Source product is irrelevant. SC4S is failing to recognize BSD syslog messages that are handled without issue using syslog, rsyslog, and syslog-ng. Only SC4S is incapable of properly recognizing the log as a BSD formatted message.

And yes, it has been pointed out that it does not strictly follow the RFC, but in practice RFCs are not always followed exactly as they are not beholden to a certifying body. All other programs that are standard in the handling of RFC 3164 and RFC 5424 messages are capable of correctly processing the log.

From my interactions here splunk appears to be unwilling to provide functional parity. Could you explain why that is?

[rjha-splunk]

I understand your concern, we will keep this issue open and discuss internally.

sc4s was designed to work in the said way and it is working as designed, we will check what we can do here and keep posted.

[ryanfaircloth]

The log format is not bsd formatted. In a unconfigured syslog-ng instance no processing other than logging the raw message occurs sc4s performs complex processing on the structured data for multiple use cases using syslog-ng. I'm sure splunk would be happy to help as they have with hundreds of non compliant sources. They need some help from you or the vendor. They need enough message samples from a packet capture to see all the defects in the source and they need to know the vendor and product to document the support.

[RaygunDuck]

So, won't fix. Got it.

[ryanfaircloth]

To be very specific its not a fix because its not wrong. However sc4s is designed to handle these conditions by forcing a developer to make correct choices. If you interested in the enhancement you just to to answer the rather simple questions asked. Your interactions imply you are more interested in this not being resolved than seeing a resolution for non technical reasons. (Not a Splunker)

[RaygunDuck]

I have provided the required information in order to fix/enhance/(insert your preferred term here) however the responses that I continue to receive are difficult to interpret as anything other than being deliberately obtuse in order to avoid resolving the stated issue.

[rjha-splunk]

Apologies, our stand is still same we will check it internally and keep you posted.

[ryanfaircloth]

@RaygunDuck the handling outliers can be tricky you could help by being specific with the source vendor and product to allow proper documentation for dealing with this non standard format and testing for regression in future releases. "syslog" is defined in RFC5424 as a standard, your incorrectly relying on rfc3164 which is not a standards track document. More than one sample is useful in many cases when source products are not following the standard there is more than one defect and the defects are not consistent its common to need a packet capture to find all the potential issues. SC4S is extensible you can add the parser support yourself as well. Another valid choice is to refer the issue back to the origin and ask them to send standard syslog

[rjha-splunk]

As per the current guidelines if the problem is with metadata and nonstandard RFC formats, we suggest to write custom parsers , splunk_metadata.csv to correct it.