JMX agent interfering with logging format of application

[mapsacosta]

We have started monitoring our application with the JMX exporter. Everything works well in terms of metrics, but we are experiencing a strange side effect. Our application log format gets overridden and we get a different logging format that messes with some of our other monitoring.

Our application has a specific logging format

    System.setProperty("java.util.logging.SimpleFormatter.format", "[%4$-8s %1$tF %1$tT %1$tL] %5$s %6$s%n");

which translates to:

[INFO     2024-06-15 15:45:09 627] Logging level set to CONFIG
[INFO     2024-06-15 15:45:09 658] Caching nodes
[INFO     2024-06-15 15:45:09 694] DB connections to XYZ servers will be direct

However, when we start the application with JMX exporter java agent, our logs become unreadable. The same startup log lines look like:

Jun 17, 2024 2:20:30 PM <redacted classpath>
INFO: Logging level set to CONFIG
Jun 17, 2024 2:20:30 PM <redacted classpath>
INFO: Caching nodes
Jun 17, 2024 2:20:30 PM <redacted classpath>
INFO: DB connections to XYZ servers will be direct

Has anyone else encountered this? Is there a way to override jmx exporter logging format to fit our original log structure? Why is the exporter overriding application-specific settings?

Thanks

[dhoard]

@mapsacosta The exporter doesn't set any system properties. The exporter Logger is a wrapper around java.util.logging.Logger

https://github.com/dhoard/jmx_exporter/blob/main/collector/src/main/java/io/prometheus/jmx/logger/Logger.java

I suspect what is happening is that the exporter Logger code is initializing the java.util.logging.Logger system with the default format.

Because...

System.setProperty("java.util.logging.SimpleFormatter.format", "[%4$-8s %1$tF %1$tT %1$tL] %5$s %6$s%n");

... is executed in application code, after the initial calls to java.util.logging.Logger.getLogger() the format is most likely not changing the format.

Can you test passing passing the format as JVM argument? i.e. -Djava.util.logging.SimpleFormatter.format

[zalmane]

Just encountered the same issue. My solution was to add this CLI flag to JAVA_OPTS:

-Dava.util.logging.manager=org.jboss.logmanager.LogManager

[mapsacosta]

We set the logger format through JAVA_OPTS as you all suggested and it fixed the problem for us. Thank you!