search

hvisage / metalog

metalog is a syslog replacement that allows regular expresion matching
GNU General Public License v2.0
37 stars 11 forks source link

readme 

                           .:. METALOG .:.

       ------------------------ BLURB ------------------------

Metalog is a modern replacement for syslogd and klogd. The logged messages can be dispatched according to their facility, urgency, program name and/or Perl-compatible regular expressions. Log files can be automatically rotated when they exceed a certain size or age. External shell scripts (e.g., mail) can be launched when specific patterns are found.

Metalog is simple to configure, accepts unlimited number of rules and has (switchable) memory bufferization for maximal performance.

http://metalog.sourceforge.net/

    ------------------------ COMPILATION ------------------------

In order to compile Metalog, you should have the PCRE2 library on your system. That library is used to process Perl-compatible regular expressions.

libpcre2 comes with many distributions, but you can also download it from :

                         http://www.pcre.org/

Once installed, don't forget to run "ldconfig".

Then, compiling and installing Metalab isn't rocket science :

./configure make install-strip

   ------------------------ CONFIGURATION ------------------------

A configuration file should be installed. Its default location is /etc/metalog.conf (unless you tweaked --with-sysconfdir) . You can find a sample file in this directory, but it's certainly not perfect for your system and your needs. So read on.

A configuration file has the following syntax :

[default values]

:
...
:
... [...] To make it clear, here's an example : maxsize = 100000 maxtime = 86400 maxfiles = 5 Kernel messages : facility = "kern" logdir = "/var/log/kernel" Crond : program = "crond" logdir = "/var/log/crond" A section defines several things : - A title (useless for the software, it's just to make your configuration file look better) . - Filters : you can define facilities, program names, urgency levels and regular expressions. An incoming message will pass through all filters. If all conditions are matching, actions defined for the section are performed. - Actions : they are taken only when all previous conditions are met. Only two actions are currently possible : write the message to a log file, and/or launch an external script. Here's a list of values that can be independently assigned for each section : * minimum = : only record a message if its urgency is inferior or equal to . Level '0' is the most critical one, while level '7' is for debugging messages. 'minimum = 5' will strip all non-important messages. The default minimum level is 7 (ie. keep all messages) . Example : record only critical messages to /var/log/important : Critical messages : facility = "*" minimum = 1 logdir = "/var/log/important" * maximum = : don't log if the message level exceeds that value. By default, maximum is the largest possible level. * facility = : only record a message if the application that issued it uses syslog facility . * break = 0|1 (default=0) : if set to 1 and a section is matched, perform action, but don't consider any more possible section matches below this one in the config file. This is useful for creating a config where specific types of log messages are matched and dispatched, while a "catch-all" section at the bottom of the config file handles the default case. Using break=0 would cause messages to be potentially handled by multiple sections, causing message duplication, while using break=1 (default) on the sections above the "catch-all" would avoid duplication. Facility names are : "auth", "authpriv", "cron", "daemon", "ftp", "kern", "lpr", "mail", "news", "security", "syslog", "user", "uucp", "local0", "local1" ... "local7" . All kernel messages are logged with facility "kern". A section can have several "facility = ..." lines to match more than one facility. If is "*", it'll match all the facilities. Example : record all authentication messages to /var/log/auth : Authentication messages : facility = "auth" facility = "authpriv" logdir = "/var/log/auth" * program = : only record messages signed by a specific daemon or program. You can use this as a modern alternative to syslog facilities : use "*" as a facility, and set a program name with that directive. Kernel messages can be caught with 'program = "kernel"' . Example : record Pure-FTPd messages in a directory, and in.ftpd messages in another directory (although the facility is the same : FTP) : In.FTPd messages : facility = "*" program = "in.ftpd" logdir = "/var/log/in.ftpd" Pure-FTPd messages : facility = "*" program = "pure-ftpd" logdir = "/var/log/pure-ftpd" * program_regex = : similar to program, but not just a string equal match, but a regular expression. Example : this would match program fields with [postfix/smtp], [postfix/smtpd] and [postfix/cleanup] etc. All Postfix messages : facility = "*" program_regex = "postfix" logdir = /var/log/postfix * program_neg_regex = : log programs that do _not_ match the regex. * regex = : a message must match the regular expression to pass that filter. Multiple regexes are allowed for a single section. All Perl extensions are allowed, and matching is case insensitive. Example : record all authentication failures to /var/log/pwdfail : Password failures : regex = "(password|login|authentication)\s+(fail|invalid)" regex = "(failed|invalid)\s+(password|login|authentication)" regex = "ILLEGAL ROOT LOGIN" logdir = "/var/log/pwdfail" * neg_regex = : this is the opposite of the previous directive. Logging will occur if the regex doesn't match. It can be useful to remove the crap, and it can be freely mixed with "regex" directives. "regex" and "neg_regex" directives are scanned in order. Example : facility = "mail" neg_regex= "starting daemon" logdir = "/var/log/mail" * maxsize = : automatically rotate log files when their size have exceeded that size. Messages are never truncated and no message can be lost during a rotation. Every section can have a different maxsize value. Example : rotate logs files when they are more than 1,000,000 bytes long : maxsize = 1000000 * maxfiles = : how many files to keep after rotation. This parameter can be set independently for each section. It defaults to 5. * maxtime = : automatically rotate log files when they are older than this number of seconds. Every section can have a different maxtime value. maxsize and maxtime can be combined together, so that rotation occurs when any of these conditions occurs. Example : rotate log files when they are more than 1,000,000 bytes long, and every day, and keep only 3 history files : maxsize = 1000000 maxtime = 86400 maxfiles = 3 * showrepeats = 0|1 : disable printing the "last message repeated N times" summary messages. When showrepeats is set to 1, messages will be printed directly to the log, without summarizing them based on number received (which can cause problems for some log monitoring programs, as well as delaying repeated messages if another unique message isn't received promptyly). Defaults to 0 (i.e. repeated messages will be summarized) * logdir = : record messages in the specified directory. If the directory doesn't exist, it will be automatically created when the first matching message will be logged (the parent directory has to exist, though) . * perms = : permissions for the log directory. Defaults to 0700 Example : Let those in the group with the GID of the process read the log. Don't forget to run metalog as the group. perms = 0770 * command = : run a program or a shell-script when all conditions are met. This directive is not incompatible with logdir : a message can be both logged and passed to an external command. When the command is launched, the first argument is filled with the date the message was received, the second argument is the program name, and the last one is the text of the message itself. Environment variables aren't cleared. Example : send a mail to root when authentication failures occur : Mail password failures : regex = "(password|login|authentication)\s+(fail|invalid)" regex = "(failed|invalid)\s+(password|login|authentication)" regex = "ILLEGAL ROOT LOGIN" command = "/usr/local/sbin/pwdfail.sh" "pwdfail.sh" can be a simple shell script like this one : #! /bin/sh echo "$3" | mail -s "Password failure (program : $2)" root Don't forget to properly quote arguments to avoid security problems. * postrotate_cmd = : run a script after rotating If specified, the postrotate_cmd is run after log-files are rotated. The following parameters are provided, environments are not cleared: date : date of the last message received prog : program name of the last message received file : file name with path of the rotated log-file Example : compress rotated log files postrotate_cmd = /usr/local/bin/compress.sh "/usr/local/bin/compress.sh" provides bzip2 compression: #!/bin/sh #* #* Copyright (c) 2006 by Lukas Ruf (lukas.ruf@lpr.ch), #* Computer Engineering and Networks Laboratory (TIK), #* Swiss Federal Institute of Technology (ETH) Zurich #* DATE="$1" PRG="$2" FILE="$3" # tests to run bzip2 # parameter file is provided # file exists # file has a size greater than zero if [ -n "${FILE}" -a -f "${FILE}" -a -s "${FILE}" ]; then bzip2 -9 ${FILE} fi ------------------------ LOG FILES ------------------------ With the "logdir" directive, messages from a specific section are recorded in a directory. In that directory, the following files are created : - "current" : this file contains the latest recorded messages. It can be incomplete (ie. data is being written to the file when you are reading it) . - ".timestamp" : the creation date of "current". Needed for the "maxtime" directive. - "log----::" : old logs, chronologically sorted. If you ever delete these files by hand for some obscure reasons, it's not a bad idea to restart the daemon (and even better : to stop it before you mess the directories) . ------------------------ RUNNING METALOG ------------------------ Kill "klogd" and "syslogd" first. Don't run Metalog while they are running. Multiple programs listening for the same source of data is a silly idea. Once these programs are killed (and you are *sure* they are killed), check that /etc/metalog.conf is installed, and simply run : /usr/local/sbin/metalog & On GNU/Linux systems, two processes are spawned : one is the "MASTER", doing much of the work. The other one (named "KERNEL") is needed for Linux and replaces the "klogd" daemon. It is responsible for logging kernel messages. By default, messages are immediately recorded into log files, and the I/O cache is flushed to be sure that you don't loose any message if a fatal crash or a power outage occurs. But if you prefer speed, Metalog can work asynchronously, using the --async switch. In this mode, to avoid disk I/O, and unlike traditional syslog daemons, Metalog works in memory buffers, then flushes the data to disk by blocks. It improves a lot overall performances. If you temporarely want to switch to the asynchronous mode, send an USR2 signal to the process. Something like : kill -USR2 $(cat /var/run/metalog.pid) should do the trick. Later, if you want to watch activity in real-time (like a good old "tail -f" on a log file), you can disable buffering. Just send an USR1 signal to the "MASTER" process. You can always re-enable buffering afterwards. ------------------------ RUN-TIME OPTIONS ------------------------ Metalog accepts some run-time options : - '-a' or '--async' : improve performance by using buffers (but log files won't get updated in real time) . - '-B' or '--daemonize' : have the server start in background (daemonization) . - '-c ' or '--consolelevel=' : set the console log level on Linux. Valid values are from 0 to 7. The default is 7. - '-C ' or '--configfile=' : use an alternative configuration file. - '-g ' or '--group=' : change the GID of the metalog process. Created files will be owned by this group. - '-h' or '--help' : show help and version number. - '-p ' or '--pidfile=' : set the name of the file that will hold the PID number. It defaults to /var/run/metalog.pid - '-s' or '--sync' : start in synchronous mode, with no bufferization. -Frank DENIS "Jedi/Sector One" .