FIX: Update syslog ingest pipeline per #5251

[abesinger]

Update syslog ingest pipeline to support issues discussed in #5251.

In summary:

• parsing of rsyslog's RSYSLOG_ForwardFormat, with ISO8601/RFC3339 timestamps
• storing hostname, process name and pid in ECS compliant fields
• parsing of priority into facility and level.

[weslambert]

Thanks for putting this together. We have not been able to spend much time on the syslog parsing, as it's usually only been as issues arise or a need for custom parsing (at which point we branch off and perform the parsing in the new ingest pipeline for the custom logs), and it has not been a primary development effort considering we are switching much of the pipeline to native ECS pipelines and modules. When that happens, we may simply switch to the built-in parsing and modules provided by the Filebeat syslog input, as opposed to using raw UDP/TCP input w/ port 514 and our own pipeline, or the existing pipeline may be modified to accommodate upstream changes included with the syslog input.

Have you tested this across upgrades, and with data from pre-implementation and post-implementation, co-existing within the same search timeframe? Has there been any testing around mapping changes or anything of that nature to ensure there are no mapping conflicts?

[abesinger]

In terms of testing across upgrades, I started this on 2.3.60 and I've since gone through all the upgrades up to 2.3.91, and it continues to work. I'm not entirely sure of what you mean by "data from pre and post-implemenation"? And WRT mapping changes, do you mean ECS fields? FWIW, the additions I made a pretty straightforward, (a copy of the code is in #5251 if you want to see), and basically make the pipeline work properly for the default rsyslog configuration of most Linux distros. I don't have a dedicated testing environment right now. I'm doing this all on a semi-production system.

WRT to moving to filebeat, if that means having to install filebeat on all hosts sending syslog, don't. I've worked in enough large environments to know that requiring additional software get installed on all hosts on the network can be a non-starter. Even if you get it done on most, there will be hosts that can't get done for some reason, and you still need to collect logs from those hosts. And in terms of getting sysadmins to comply, it's a lot easier to get them to make a modification to their existing rsyslog config than it is to get them to install and support new software.

[weslambert]

When referring to pre and post, I'm just trying to get an understanding if you're maintaining data from prior to the pipeline changes, as well as data after the pipeline changes. If a pipeline changes, or the mappings associated with the fields manipulated by the pipeline change, there can be potential for field conflicts from the data type perspective. By having the data in there from both periods of time, we can see if there are potential issues that would arise similarly in someone else's deployment.

WRT to Filebeat, I am not referring to installing Filebeat on every node. When you send syslog to Security Onion, it is actually Filebeat listening on port 514 UDP/TCP that initially receives the data. The data is then forwarded from Filebeat to Logstash, to Redis, to Logstash, then the Elasticsearch ingest node pipeline on the search node (or Logstash search pipeline on the same node).

[abesinger]

Thanks for the clarification. BTW, I've put in the PR so that my mods can be easily reviewed.

WRT pre/post data... I don't specifically have pre/post parsed data, but I can see running a test using the same source data with and without the pipeline, and comparing the resulting Elastic documents. However, the only data I have to test with is rsyslog messages. The original pipeline matched two particular syslog formats, but neither are a common Linux format, and I don't have source data for either of those.

Also, I made these changes because rsyslog messages were simply not getting parsed properly with the original pipeline. So running a test with my data and comparing the results would really only show whether or not the changes work.

WRT potential changes to field mappings... Technically, The addition to the grok section sets some common fields when it matches an rsyslog message. But otherwise, my changes don't alter any existing fields, they just create some new ones, and copy some existing ones to new ones. I specifically wanted to avoid changing fields that another might be depending on.

[abesinger]

WRT filebeat, thanks for the clarification on that too. I'd suggest not just relying on the filebeat parsing, their modules have their own problems (for example, see my screed in #5553). Having a pipeline that ingests the output of the filebeat syslog module seems reasonable, though it might be tricky to implement since the filebeat model has modules push their own ingest pipelines.

[weslambert]

We can control to which pipelines events are sent from Filebeat or Logstash, or both, so we have a bit of flexibility there. As far as the Filebeat module pipelines, we load those ourselves, and have Logstash dynamically assign to a pipeline based off of the module, and we can always implement a final ingest pipeline or change things after-the-fact as needed. We can look at potentially including this after the .100 release, but I hesitate to make any changes at the moment, as we have a fair number of changes coming down the pike for the upcoming release that need to be tested. We'll likely need to add some field mappings as well if there are no fields, depending on if dynamic mapping is being used or not.

[abesinger]

Fair enough wrt waiting until after .100.

I'm not sure what you mean about "adding field mappings if there are no fields?"

[weslambert]

I haven't reviewed the code much yet, but your previous comment alluded to additional fields, so I wasn't sure if those would be included in the mapping or not (as I wasn't sure which fields there were):

But otherwise, my changes don't alter any existing fields, they just create some new ones, and copy some existing ones to new ones

Based on what's shown in the PR, it looks like those fields should already have mappings, so it shouldn't be an issue.

[dougburks]

Looks like this breaks our pfsense firewall log parsing, so we'll have to revert this for now and revisit later.

[abesinger]

@dougburks I'm happy to try to debug, but I'd need data. Can you post some log samples and maybe an example of what's breaking?

[dougburks]

Thanks @abesinger. It's most likely something trivial, but we need to get this release out the door and simply don't have any time to troubleshoot right now.

Here's a sanitized sample:

<134>Mar  1 17:21:30 filterlog[17123]: 70,,,1567509123,igb0.123,match,pass,in,4,0x0,,64,0,0,DF,6,tcp,64,192.168.123.123,74.125.123.123,65123,443,0,S,3379863123,,65535,,mss;nop;wscale;nop;nop;TS;sackOK;eol

If you'd like more samples and have time, you could always do a quick installation of pfSense in a VM.

[abesinger]

NP wrt your release taking priority. I'll see what I can do. Unfortunately spinning up a VM for this is not really feasible for me, but I may be able to figure this out without that. It may very well be trivial.

FYI, I understand well how messy this can get, but it's solveable. In the early 2000s I worked on parsing all of the different syslog header formats for the messages we were collecting at SDSC. I ended identifying something like 20 (or was it 40? it's been a minute ;-) different formats, and parsing properly depended not only on having regexes to match each different header, but the specific order of the regexes mattered.

[abesinger]

@dougburks FYI I think I see the problem and how to fix it, but I'm tied up with some other stuff right now. If you can leave this open I'll follow up

[dougburks]

Leaving open, thanks @abesinger !

[dougburks]

@abesinger Just as an FYI, @weslambert has this working now.

[abesinger]

Thanks for letting me know.

[weslambert]

https://github.com/Security-Onion-Solutions/securityonion/pull/7644