One day, my friend asked a question about how to use fluentBit (It's popular in k8s [^1]) to collect Java application logs. I had no idea how to do this at first, but finally the result seems good, so I want to give this tale to introduce the way I walk pasted.
The config visualizes
Usually, the log file pattern seems unified, it looks like
You can easily use regex to collect each line into a structure. But when the app throws exceptions, you need to use multiline parser [^2][^3]. Briefly, it's caught raw text read line by line, so each line needs to match rule, like this:
# rules | state name | regex pattern | next state
# --------|----------------|---------------------------------------------
rule "start_state" "/(Dec \d+ \d+\:\d+\:\d+)(.*)/" "cont"
rule "cont" "/^\s+at.*/" "cont"
Each line pipeline
2022-04-17 03:10:43.578 ERROR 28420 --- [scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task.
java.lang.RuntimeException: cs
at guru.sfg.brewery.config.TaskConfig.throwError(TaskConfig.java:25) ~[classes/:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.1.6.RELEASE.jar:5.1.6.RELEASE]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.1.6.RELEASE.jar:5.1.6.RELEASE]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[na:na]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
2022-04-17 03:10:43.591 INFO 28420 --- [restartedMain] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2022-04-17 03:10:43.593 INFO 28420 --- [restartedMain] g.sfg.brewery.SfgBreweryUiApplication : Started SfgBreweryUiApplication in 2.717 seconds (JVM running for 3.998)
2022-04-17 03:10:43.596 INFO 28420 --- [scheduling-1] o.h.h.i.QueryTranslatorFactoryInitiator : HHH000397: Using ASTQueryTranslatorFactory
But how to? The official Doc is good, but in this case, in the default spring logging pattern, log message with break-line, and, in Windows, it's \r\n, Me using windows, so be an attention
So, collusion below (remember when the up satisfied that one can be execute):
The start line is simple (\d{4}-\d{2}-\d{2}\s\d+\:\d+\:\d+.\d+)(.*) , see
One day, my friend asked a question about how to use fluentBit (It's popular in k8s [^1]) to collect Java application logs. I had no idea how to do this at first, but finally the result seems good, so I want to give this tale to introduce the way I walk pasted.
The config visualizes
Usually, the log file pattern seems unified, it looks like
2022-04-17 03:10:42.381 INFO 28420 --- [restartedMain] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...You can easily use regex to collect each line into a structure. But when the app throws exceptions, you need to use multiline parser [^2][^3]. Briefly, it's caught raw text read line by line, so each line needs to match rule, like this:
Each line pipeline
But how to? The official Doc is good, but in this case, in the default spring logging pattern, log message with break-line, and, in Windows, it's
\r\n, Me using windows, so be an attentionSo, collusion below (remember when the up satisfied that one can be execute):
(\d{4}-\d{2}-\d{2}\s\d+\:\d+\:\d+.\d+)(.*), see\r\n\s+name: default3, see\s+...]xxx.xxx.xxException\s+at XXXXIn the end, the exception snippet will produce a single line, you can use filter to exclude this.
gist: https://gist.github.com/bxb100/de46e5f708d03d509430d4767806fb14
Other thing
Using docker needs packing by yourself ^4, don't forget it should with ES under same network ^5
docker build -t fluentbit-cs:sim .docker run -id --name fluent -v /e/docker/fluentBit/log:/var/log --network 8x_default fluentbit-cs:sim/fluent-bit/etcdocker run -id --name fluent -v /e/docker/fluentBit/log:/var/log -v /e/docker/fluentBit/config:/fluent-bit/etc --network 8x_default fluent/fluent-bit:1.9.0If you are missing
cont2rule, the ES log will trigger exception like single document, it's not combined with log messageObviously It's not my wanted type
[^1]: https://gist.github.com/StevenACoffman/4e267f0f60c8e7fcb3f77b9e504f3bd7 fluent-filebeat-comparison
[^3]: https://docs.fluentbit.io/manual/pipeline/inputs/tail Tail input