Feat: make java patterns match in an ecs compatible way

logstash-plugins / logstash-patterns-core

Apache License 2.0

2.17k stars 979 forks source link

Feat: make java patterns match in an ecs compatible way #268

Closed kares closed 3 years ago

kares commented 4 years ago

should be straightforward except 2 hiccups:

Java class-name capture naming (no ECS convention really)
message sub-part wasn't being matched (appended) to message but rather to logmessage which I believe is good but we did not yet decide whether that's a convention to follow (in which case it should be followed for all ecs-ized patterns -> namespacing sub-message part)

resolves https://github.com/logstash-plugins/logstash-patterns-core/issues/123 closing https://github.com/logstash-plugins/logstash-patterns-core/pull/244

kares commented 4 years ago

OK, this has progressed but unfortunately trying to add a specs for CATALINALOG turned out weird since the patterns is pretty much broken (even for old Tomcat 6/7):

     "timestamp"=>"Jul 30, 2020 3:00:21 PM",
     "class"=>"org.apache.catalina.core.StandardEngine",
     "logmessage"=>"start\nINFO: Starting Servlet Engine: Apache Tomcat/6.0.20",

will need to decide whether to ECS-ize it broken as is or attempt to fix before hand...

kares commented 3 years ago

Since previous review this was still missing specs for the CATALINALOG/TOMCATLOG. Adding those lead me to actually trying to support the changes in format since Tomcat 4.x (all the way to 9.0).

The catalina.out (default) logging format and the context (a.k.a. tomcat-log) logging format have been chaging at times independently of each other. Let me know if this is too much and we should instead revert to base-line (without tests).

TOMCATLOG's problem really is that I am not sure what kind of (default) log format that matches - seems like it only supported (custom) user configured logging.

kares commented 3 years ago

instead of trying to support old legacy Tomcat <= 5.0 logging (no-one will really use) I rather decided to not do so and have TOMCATX_LOG aligned with CATALINAX_LOG: https://github.com/logstash-plugins/logstash-patterns-core/pull/268/commits/6e6da5efa7400aaaffdd2047a56fbad3ff01dc41

looking at old Tomcat logging formats really came out of the need of trying to identify smt TOMCATLOG would match all the way down to Tomcat 4.x (NONE of the logging defaults match the legacy patters)