search

mbok / logsniffer

logsniffer is a sophisticated open source web tool for parsing, viewing, monitoring and analyzing log data - smarter, collaborative and easier. [No longer maintaned]
GNU Lesser General Public License v3.0
104 stars 46 forks source link

reader patterns #89

Open hyprvisr opened 7 years ago

hyprvisr commented 7 years ago

Hey can you provide some reader pattern examples?

archenroot commented 7 years ago

@hyprvisr Well, you could extend easily logsniffer by implementing your own Reader for specific Log to parse.

Otherwise as there are 2 main classes implemented, you have out-of-box log4j where you need to follow ConverionPattern (look at doc), or you can go with regular expressions.

Here is example for parsing PostgreSQL log file for messages that show tables being locked:

.*user=(?P<lock_user_name>\w+),db=(?P<lock_database>\w+) 
LOG:\s+process (?P<lock_process_id>\d+) 
acquired (?P<lock_type>\w+) on (?<lock_on>\w+) 
(?P<lock_tuple>[\(\)0-9,]+) of (?P<lock_object_type>\w+) 
(?P<lock_object_oid>\d+) of database (?P<lock_db_oid>\d+) 
after (?P<lock_wait_time>[0-9.]+).*

You can also define scanner under Events in GUI, where you can select log LEVEL you are interested in. Here you have some builtin mechanism for log4j, but you are again free to write custom regular expression.

As real world scenario, you can do your data source only for ERROR level (by regular expression) and the scanner to send emails/http requests to destination per specific content:

I plan to extend notifications to work with AMQP protocol additionally as I use message broker services mostly (very limited of HTTP - only for external usage).

kuzma725 commented 7 years ago

Hi, I can't get the log4j parsing to work. I just pasted in the pattern from my log4j.properties file:

%d{dd MMM HH:mm:ss} %m%n

and set %d to Date and %m to Message in the conversion mapping, but it gives me a parsing error on the date part. Also tried %dd %MMM %HH:mm:ss with corresponding mappings, still no luck.

mbok commented 7 years ago

@kuzma725: Strange behavior. I've tested %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c] %m%n and it works fine. Could you provide few log lines to see the syntax?

kuzma725 commented 7 years ago

@mbok Sure, copied this from Log entry details popup. You can see that it recognized the timestamp, but assumes year 1970 because the year is not specified (that's fine). But then it still considers the date as part of the message(?) which causes multiple entries to be concatted into one:

If_timestamp: Sep 6, 1970 11:17:34 PM lf_unformatted: true Message: 06 Sep 23:17:34 File watch poll still alive... 06 Sep 23:17:48 Close client socket with status 1002: 1277288408 06 Sep 23:18:24 File watch poll still alive... 06 Sep 23:19:14 File watch poll still alive...

kuzma725 commented 7 years ago

I should have mentioned this earlier: some of my log entries have multiple lines of text and extra newlines; for example, this is all 1 entry (modifed content to remove sensitve info):

06 Sep 16:28:25 Line 1 of Entry 1 Line 2 of Entry 1 Line 3 of Entry 1

I was able to get rid of the parsing errors by using [ ] around the message, but it still doesn't parse the log correctly because, as I showed in my previous post, it lumps multiple entries into one, including the timestamps as part of the message. I guess my question is more about the conversion pattern spec: for example, it doesn't explicitly say anywhere that [ ] means the pattern is optional, but I happen to know this from using regular expressions. If someone could give me a working pattern for the sample entry above, I would appreciate it.