Regexp for an slf4j log format

[thanosIrodotou]

Opening this as an issue to get your opinion. I was fairly confident the PR would be rejected since the format depends on the slf4 log formatter people use. But I added a regexp for a fairly used slf4j log format:

log line:

11:22:19.068 [Something Something #1] INFO  a.b.c.d.e.f.gh.KLMNO - Some words that make sense and numbers 10.2.32.84

regex:

Message pattern: ^([^\[]+)\s*(\[.*])\s*([A-Z]+)\s*\s*(\S*)\s*-(.+)$
Message start pattern: ^\d
Time format: HH:mm:ss.SSS
Time capture group: 1
Severity capture group: 3
Category capture group: 4

feel free to test it at https://regex101.com

[knah]

It's understandable that people use different log formats. This means that included formats serve more as examples, rather than something deemed universally useful, although including some common log formats is definitely nice.

Regarding your format: there is \s*\s* - probably that can be just one \s*. Additionally, it might be better to use [^\]]* instead of .* for the text in square brackets. Besides, it looks like there are two groups that have category-like functionality - the one in square brackets, and the one with what appears to be class name. Probably we need a more flexible approach for categories, given that there are multiple log format that contain more than one.

As another side note, we probably should include sample log lines with provided formats - trying to decipher the regex to get a feel for what a log format looks like is a bit too bothersome.

[thanosIrodotou]

Thanks for your feedback.

As another side note, we probably should include sample log lines with provided formats - trying to decipher the regex to get a feel for what a log format looks like is a bit too bothersome.

+1 for that, you could also have a static image showing the group highlighting, that could be even more helpful in understanding how the groups are split and used.

Probably we need a more flexible approach for categories, given that there are multiple log format that contain more than one.

Very much agree with this too. It seems like it can be further broken down to:

time capture group
severity capture group
category capture group - although I would suggest "source capture group"?
message capture group?

I understand that this would depend on the preferred or customised format of the logger but still a fairly popular pattern would emerge.

p.s. thanks for taking the time looking into this, really liking your plugin 👍

[steveloughran]

FWIW, [Something Something #1] is actually thread name...general best practise is for workers to call setThreadName() whenever they are doing some work, so that on logs & stack dumps you know who to blame. It's the %t option in the normal log4j pattern, and there because most people don't tune it from the defaults

It's usually the pattern

log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{2} - %m%n

or maybe they add in filename & line, which is very expensive (thread is created for the stack, very bad, handy for dev & debug though)

log4j.appender.stdout.layout.ConversionPattern=%d{ISO8601} [%t] %-5p %c{2} (%F:%M(%L)) - %m%n

If there's a specific bit of tuning for the generated log formats which helps with their parsing & display, that's something which could be included with any example

[sergey124]

@thanosIrodotou , maybe you could also give ConversionPattern for this slf4j log format, or appender you use?

I couldn't find how to configure log4j appender to use this format.

[rala72]

I've found another solution for my Spring logs. It works for example for this one:

2019-05-06 15:06:19.038 DEBUG 1484 --- [main] a.a.t.s.g.backend.BackendApplication     : Running with Spring Boot v2.1.2.RELEASE, Spring v5.1.4.RELEASE

my config:

message pattern: ^([^ ]* [^ ]*)\s*([^ ]*)\s*(\d*)\s*-*\s*(\[[^]]*\])\s*([^ ]*)\s*:\s*(.*)$
message start pattern: ^\d
timeformat: yyyy-MM-dd HH:mm:ss.SSS
time capture group: 1
severity capture group: 3
category capture group: 4

I also made the default patterns for severity case insensitve by adding (?i) at the beginning of all of them and added a debug pattern: (?i)^\s*d(ebug)?\s*$ with a dark blue color. Finally I made the ing of warning optional with this pattern: (?i)^\s*w(arn(ing)?)?\s*$. It works now for messages like the following:

2019-05-06 19:48:16.804  WARN 16156 --- [http-nio-8080-exec-1] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.web.bind.MethodArgumentNotValidException: [...]

[tylerlazenby]

I just tried that here, and I am using a VERY simplistic monolog, with no changes to the format.

Example: [2019-04-09 12:51:11] mail_log.ERROR: Could not send email to "Tyler Lazenby" <> [] []

I used the settings from rala72 and I am not seeing it recognize the Log format.

Log format not recognized

[avp90]

I use ^([\s\d-,:]+) \[(.*)] (\w.*) - (.*)$ for Play Framework logs (SLF4J with Logback). Log example:

2019-07-03 10:11:40,656 [INFO] from application in play-dev-mode-akka.actor.default-dispatcher-2 - Creating Pool for datasource 'default'
2019-07-03 10:11:40,815 [INFO] from play.api.db.DefaultDBApi in play-dev-mode-akka.actor.default-dispatcher-2 - Database [default] connected at jdbc:postgresql://127.0.0.1/demo?currentSchema=test
2019-07-03 10:11:43,079 [INFO] from play.api.Play in play-dev-mode-akka.actor.default-dispatcher-2 - Application started (Dev)