JBoss does not boot with jboss-logmanager-ecs-formatter configured

[guillermomolina]

I have JBoss 7.4.15.

I've added the module as the documentation suggested:

modules/co/elastic/logging/main/module.xml

<?xml version="1.0" encoding="UTF-8"?>
    <module xmlns="urn:jboss:module:1.9" name="co.elastic.logging">
    <resources>
        <resource-root path="ecs-logging-core-1.6.0.jar"/>
        <resource-root path="jboss-logmanager-ecs-formatter-1.6.0.jar"/>
    </resources>
    <dependencies>
        <module name="org.jboss.logmanager" />
    </dependencies>
</module>

standalone/configuration/standalone.xml

            <formatter name="ECS">
                <custom-formatter module="co.elastic.logging" class="co.elastic.logging.jboss.logmanager.EcsFormatter" />
            </formatter>

JBoss won't boot:

05:58:44,002 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("subsystem" => "logging")]): java.lang.IllegalArgumentException: Failed to instantiate class "co.elastic.logging.jboss.logmanager.EcsFormatter" for formatter "ECS"
        at org.jboss.logmanager.config.AbstractPropertyConfiguration$ConstructAction.lambda$validate$2(AbstractPropertyConfiguration.java:122)
        at org.jboss.logmanager.config.WrappedAction.execute(WrappedAction.java:47)
        at org.jboss.logmanager.config.AbstractPropertyConfiguration$ConstructAction.validate(AbstractPropertyConfiguration.java:118)
        at org.jboss.logmanager.config.LogContextConfigurationImpl.doPrepare(LogContextConfigurationImpl.java:336)
        at org.jboss.logmanager.config.LogContextConfigurationImpl.prepare(LogContextConfigurationImpl.java:289)
        at org.jboss.as.logging.logmanager.ConfigurationPersistence.prepare(ConfigurationPersistence.java:299)
        at org.jboss.as.logging.LoggingOperations$CommitOperationStepHandler.lambda$execute$0(LoggingOperations.java:131)
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:1063)
        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:784)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:470)
        at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1430)
        at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:559)
        at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:546)
        at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:508)
        at org.jboss.as.server.ServerService.boot(ServerService.java:462)
        at org.jboss.as.server.ServerService.boot(ServerService.java:415)
        at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:447)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.jboss.logmanager.config.AbstractPropertyConfiguration$ConstructAction.lambda$validate$2(AbstractPropertyConfiguration.java:120)
        ... 17 more
Caused by: java.lang.NoClassDefFoundError: java/util/logging/LogManager
        at co.elastic.logging.jboss.logmanager.EcsFormatter.getProperty(EcsFormatter.java:127)
        at co.elastic.logging.jboss.logmanager.EcsFormatter.<init>(EcsFormatter.java:48)
        ... 22 more
Caused by: java.lang.ClassNotFoundException: java.util.logging.LogManager from [Module "co.elastic.logging" from local module loader @56cbfb61 (finder: local module finder @1134affc (roots: /aplicacions/jboss/modules,/aplicacions/jboss/modules/system/layers/base))]
        at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:255)
        at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:410)
        at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)
        at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:116)
        ... 24 more

Please help me, is there anything more to add to the module definition? Thanks

[SylvainJuge]

Hi,

I am really not familiar with Jboss modules, but there is probably a way to allow a module to use the java.util.logging classes that seems to be disabled by default. For example there are a few references to adding java.logging as a module dependency.

What happens here is that Jboss modules are loaded in an isolated classloader that filter what is allowed to access in the JDK, which is why you get a NoClassDefFoundError on a standard JDK class.

As an alternative, maybe using the library directly into your application instead of JBoss classloader could be a simpler option.

[guillermomolina]

Hi,

Thank you very much for the tip:

This:

<?xml version="1.0" encoding="UTF-8"?>
    <module xmlns="urn:jboss:module:1.9" name="co.elastic.logging">
    <resources>
        <resource-root path="ecs-logging-core-1.6.0.jar"/>
        <resource-root path="jboss-logmanager-ecs-formatter-1.6.0.jar"/>
    </resources>
    <dependencies>
        <module name="org.jboss.logging"/>
        <module name="java.logging"/>
    </dependencies>
</module>

Does not throw the exception any more. But I've added a handler, to create a file log with this formatter in my standalone.xml:

            <periodic-rotating-file-handler name="ELASTIC" autoflush="true">
                <filter-spec value="not(match(&quot;JBAS015960&quot;))"/>
                <formatter>
                    <named-formatter name="ECS"/>
                </formatter>
                <file relative-to="jboss.server.log.dir" path="server.json"/>
                <suffix value=".yyyy-MM-dd"/>
                <append value="true"/>
            </periodic-rotating-file-handler>

The file is created but nothing is written into it:

[jboss@jb7-1 log]$ ls -la server.json 
-rw-r--r-- 1 jboss aplicacions 0 Jun 29 06:42 server.json

I've modified standalone/configuration/standalone.xml as documented:

            <formatter name="ECS">
                <custom-formatter module="co.elastic.logging"  class="co.elastic.logging.jboss.logmanager.EcsFormatter" >
                    <properties>
                        <property name="serviceName" value="my-app"/>
                        <property name="serviceVersion" value="my-app-version"/>
                        <property name="serviceEnvironment" value="my-app-environment"/>
                        <property name="serviceNodeName" value="my-app-cluster-node"/>
                    </properties>
                </custom-formatter>
            </formatter>

But log file is still 0 bytes, any thoughts?

By the way, the documentation is also missing that you have to configure some JAVA options in standalone.conf:

JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=org.jboss.byteman,org.jboss.logmanager"
JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
JAVA_OPTS="$JAVA_OPTS -Xbootclasspath/p:$(ls ${JBOSS_HOME}/modules/system/layers/base/org/wildfly/common/main/wildfly-common-*.jar)"
JAVA_OPTS="$JAVA_OPTS -Xbootclasspath/p:$(ls ${JBOSS_HOME}/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-*.jar)"
JAVA_OPTS="$JAVA_OPTS -Xbootclasspath/p:${JBOSS_HOME}/jboss-modules.jar"

[SylvainJuge]

Again, I have almost no expertise in Jboss logging, but does the filtering is mandatory with <filter-spec value="not(match(&quot;JBAS015960&quot;))"/> ? The URL-encoded characters seem off to me, but trying without it and simplify to the minimum could help here.

For the extra JVM options, those are very jboss/wildfly specific, so I am not sure how relevant they are in general to all jboss deployments. The most common use-case is to use those ECS loggers directly into the applications, not to deploy them globally on the server.

When using the Elastic APM Java agent, the jboss.modules.system.pkgs and java.util.logging.manager should be managed by the agent, which makes the setup simpler.