rsyslog / liblognorm

a fast samples-based log normalization library
http://www.liblognorm.com
GNU Lesser General Public License v2.1
99 stars 64 forks source link

Parse custom application log file using different rulebase? #303

Closed jbyers-suse closed 6 years ago

jbyers-suse commented 6 years ago

I wanted to verify the correct procedure to parse a second custom log file that I'm loading into the rsyslog.configuration using the imfile module.

The Apache logs are currently loaded and parsed as expected using the rulebase specified with the mmnormalize action statement. I'm trying to use a different rulebase to parse another custom application log file. Could I specify multiple rulebases in the same file for different log formats? Should I consider using a custom defined user type and/or include in the existing rulebase for the Apache logs?

Here is the relevant section of the rsyslog configuration

module(load="imfile") input(type="imfile" File="/var/log/apache2/*_log" Tag="apache:"

input(type="imfile" File="(path to custom application log file)" Tag="app:"

module(load="mmnormalize") action(type="mmnormalize" rulebase="/opt/apache.rb"

I'll provide the contents of the two rulebases that I configured if needed. The second rulebase is in the same directory and specified using a rule statement.

Thank you. I appreciate any advice/suggestions/feedback.

davidelang commented 6 years ago

mmnormalize handles large rulesets reall well, just combine the different rules into one ruleset, it's actually faster than having

if type=foo then mmnormalize (ruleset1) if type=bar them mmnormalize (ruleset2) etc

At a previous job I had a 1400 line mmnormalize rulebase. Rainer used it in benchmarking and it performed very well.

David Lang

jbyers-suse commented 6 years ago

Thank you.