IIS log parsing fails for exchange iis logs: likely due to slash in username

[kgeil]

Hi @philhagen, thanks for creating and maintaining SOf-Elk. I have been a user since you introduced it at SANS CDI in 2016, and in my latest role, use it very frequently.

I have noticed over the last several months that IIS logs from exchange servers have a high parse failure rate. I created my own config file based on the csv plugin, and used that for a while, but after the recent Sof-Elk update, I decided to try the built-in functionality again. I still had the same problem, so I dug into the built in configuration files, and found that, for the set of logs I'm currently working with, if there's a slash in the cs-username field, parsing breaks. (log sample below). I fixed it for myself by changing line 43 in grok-patterns/for572_custom to HTTPDUSER %{EMAILADDRESS}|%{WORD}[\]%{USER}|%{USER}. Basically, all I did was change the regex section betweeen %{WORD} and %{USER}. The original value was: [/\]

It's currently working for my needs, but I'm not sure what will break as a result. If anyone knows of a better way, of course, let me know. Otherwise, I'll keep feeding it logs for a few months, and will be in touch.

Thanks again for all of the work on Sof-Elk. Log sample below:

2021-10-25 00:00:00 10.200.0.98 POST /EWS/Exchange.asmx &CorrelationID=<empty>;&cafeReqId=4dg211hd-zv91-88g1-a231-7a64cbe618fd; 443 STI\aagnew 10.200.0.162 AppleExchangeWebServices/814.100.1+Mail/3654.120.0.1.13 - 200 0 0 15

[philhagen]

Hey! Thanks for the report - I know this has lingered...end of year got very busy. I've been trying to recreate this in the grok debugger and been unsuccessful. the [/\\] portion is designed to capture either zero or one forward slashes or zero or one backslashes. In my test on the platform (latest version), your sample log parses fine, with STI\aagnew populating into the ident field via this grok statement.

The resulting record is here:

Could you re-try with the existing grok patterns? If this still does not parse, could you confirm which version of the VM/branch of the code you're using and attach a screenshot of the resulting record both before and after altering the grok pattern?

[kgeil]

@philhagen , thanks for getting in touch. I'll do some testing this week, and will be in touch.

I did start using a workaround: I created a directory just for filebeat to grab exchange IIS logs and add a tag. Then a config file: 1599-preprocess-exchangeiis.conf, which (conditionally on that tag) uses gsub to replace the slash with a pipe.

Thanks again, I'll be in touch soon. Kevin

[philhagen]

ok sounds good and thanks - again, I was able to parse your sample in the stock VM so want to be sure the problem still exists!

[kgeil]

@philhagen , thanks again for getting in touch. I installed a new instance of sof-elk, updated, and tested, and the iis logs in question parse beautifully.

Thanks again for all of your work on this project!

[philhagen]

Great news!! Thanks for closing the loop and glad it worked for you!