payara / Payara

Payara Server is an open source middleware platform that supports reliable and secure deployments of Java EE (Jakarta EE) and MicroProfile applications in any environment: on premise, in the cloud or hybrid.
http://www.payara.fish
Other
883 stars 307 forks source link

thousands of ClassNotFound warnings while deploying an EAR with many WAR'S #374

Closed fmeili closed 8 years ago

fmeili commented 9 years ago

While deploying an EAR with about 100 WEB-Modules and about 200 EJB modules, the server.log shows more than 5000 WARNING entries with ClassNotFound, but the EAR file deploys sucessfully and it works fine. All WEB-Modules are skinny war's without having jar files in their WEB-INF/lib/ direcotries, instead all jar files are in the common ear/lib/ directory.

Some of the WAR's are using JSF and some others using REST.

The following line is one of many thousands with this strange warning (there is no stack trace). Some of the lines are repeated some thousand times, with the same class name. In our EAR, we see about 5200 of these warning log lines for about 25 different classes. One of this class is logged 4000 times with this warning.

[2015-07-14T11:59:10.314+0200] [Payara 4.1] [WARNING] [AS-WEB-UTIL-00035] [javax.enterprise.web.util] [tid: _ThreadID=55 _ThreadName=admin-listener(1)] [timeMillis: 1436867950314] [levelValue: 900] [[ Unable to load class aaa.bbb.Ccc, reason: java.lang.ClassNotFoundException: aaa.bbb.Ccc]]

Most of these classes are using JSF and/or REST annotations.

After a little bit reasearch, we found the following 3 GlassFish issues which are all still open. We think they depends maybe on the same issue.

01/Jul/11 GLASSFISH-16937 Having REST services in separate WARs in a single EAR prints classloading warnings

26/Jul/12 GLASSFISH-18946 EAR with two CDI Jersey web archives will not deploy

22/Jul/13 GLASSFISH-20720 EAR deployment with multiple embedded WARs broken in 3.1.2.2 and 4.0

Additionally we found the following:

GLASSFISH-18946 links to JERSEY-1232, but it's closed because they think it's a WELD issue, but with the WELD-1175, the WELD team think, it should be investigated by GlassFish.

We can produce this problem with GlassFish 3.1.2.2 with Java 7u55, GlassFish 4.1 with Java 7u55 and Payara 4.1.152 with Java 8u45 on both Linux and Windows.

davewinters commented 9 years ago

Hi,

Can you send me a log snippet of the actual ClassNotFoundExceptions with the stack traces associated with these exceptions you are viewing? I would like to view the framework(s) and packages these errors are being thrown from. I have seen these exceptions before when an ear file contained war files which used Jersey, weld and primefaces packages. In that specific case, it appeared to be an issue with how Glassfish uses HK2 to load and resolve classes in the class ServletContainerInitializerUtil.

Thanks.

davewinters commented 9 years ago

Prevoiusly when I experienced and viewed the ClassNotFoundExceptions logged as warnings which I believe you maybe referencing here is from the ServletContainerInitializer.checkAgainstInterestList(..) method.

private static Map<Class<? extends ServletContainerInitializer>, Set<Class<?>>> checkAgainstInterestList( Types classInfo, Map<Class<?>, List<Class<? extends ServletContainerInitializer>>> interestList, Map<Class<? extends ServletContainerInitializer>, Set<Class<?>>> initializerList, ClassLoader cl) {

    if (classInfo==null) {
        return initializerList;
    }
    for (Map.Entry<Class<?>, List<Class<? extends ServletContainerInitializer>>> e:
            interestList.entrySet()) {

        Class<?> c = e.getKey();
        Type type = classInfo.getBy(c.getName());
        if (type==null)
            continue;

        Set<Class<?>> resultSet = new HashSet<Class<?>>();
        if (type instanceof AnnotationType) {
            for (AnnotatedElement ae : ((AnnotationType) type).allAnnotatedTypes()) {
                if (ae instanceof Member) {
                    ae = ((Member) ae).getDeclaringType();
                } else if (ae instanceof Parameter) {
                    ae = ((Parameter) ae).getMethod().getDeclaringType();
                }
                if (ae instanceof Type) {
                    try {
                        resultSet.add(cl.loadClass(ae.getName()));
                    } catch (Throwable t) {
                        if (log.isLoggable(Level.WARNING)) {
                            log.log(Level.WARNING,
                                CLASS_LOADING_ERROR,
                                new Object[] {ae.getName(), t.toString()});
                        }
                    }     
                }
            }
        } else {
            Collection<ClassModel> classes;
            if (type instanceof InterfaceModel) {
                classes = ((InterfaceModel) type).allImplementations();
            } else {
                classes = ((ClassModel) type).allSubTypes();
            }
            for (ClassModel classModel : classes) {
                try {
                    resultSet.add(cl.loadClass(classModel.getName()));
                } catch (Throwable t) {
                    if (log.isLoggable(Level.WARNING)) {
                        log.log(Level.WARNING,
                            CLASS_LOADING_ERROR,
                            new Object[] {classModel.getName(), t.toString()});
                    }
                }
            }
        }
        if(initializerList == null) {
            initializerList = new HashMap<Class<? extends ServletContainerInitializer>, Set<Class<?>>>();
        }
        List<Class<? extends ServletContainerInitializer>> containerInitializers = e.getValue();
        for(Class<? extends ServletContainerInitializer> initializer : containerInitializers) {
            Set<Class<?>> classSet = initializerList.get(initializer);
            if(classSet == null) {
                classSet = new HashSet<Class<?>>();
            }
            classSet.addAll(resultSet);
            initializerList.put(initializer, classSet);
        }

    }

    return initializerList;
}
fmeili commented 9 years ago

Hi,

thank you for your fast response. There is no real stack trace. But after activating FINEST log level for the "javax.enterprise.web.util" logger, there are some more lines in the logfile (but still no stack trace).

Some of our WAR's using Jersey or primefaces packages or both of them. CDI is used in nearly all WAR's.

For each (ot the thousands) ClassNotFound WARNING, there are the following log entries:

[2015-07-17T10:28:10.353+0200] [Payara 4.1] [FINER] [] [javax.enterprise.web.util] [tid: _ThreadID=58 _ThreadName=admin-listener(4)] [timeMillis: 1437121690353] [levelValue: 400] [CLASSNAME: org.glassfish.web.loader.WebappClassLoader] [METHODNAME: loadClass] [[Loading class from delegate]]

[2015-07-17T10:28:10.353+0200] [Payara 4.1] [FINER] [] [javax.enterprise.web.util] [tid: _ThreadID=58 _ThreadName=admin-listener(4)] [timeMillis: 1437121690353] [levelValue: 400] [CLASSNAME: org.glassfish.web.loader.WebappClassLoader] [METHODNAME: loadClass] [[loadClass(aaa.bbb.ccc.communication.rest.ServiceImpl)]]

[2015-07-17T10:28:10.353+0200] [Payara 4.1] [FINER] [] [javax.enterprise.web.util] [tid: _ThreadID=58 _ThreadName=admin-listener(4)] [timeMillis: 1437121690353] [levelValue: 400] [CLASSNAME: org.glassfish.web.loader.WebappClassLoader] [METHODNAME: loadClass] [[Delegating to classloader1 EarClassLoader : urlSet = [URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/datasource-inject_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/datasource-inject_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/ldap-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/ldap-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/config-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/config-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/config-jca-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/config-jca-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/timer-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/timer-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/admin-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/admin-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/api-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/api-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/erp-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/erp-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/help-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/help-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/core-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/core-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/imsapi-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/imsapi-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/ml-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/ml-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/pm-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/pm-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/qm-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/qm-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/smt-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/smt-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/tr-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/tr-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/opm-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/opm-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/opma-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/opma-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/dbchecker-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/dbchecker-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/ams-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/ams-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/report-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/report-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/extsys-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/extsys-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/script-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/ domains/user1/generated/ejb/ourapp/script-ejb_jar] doneCalled = false Parent -> org.glassfish.internal.api.DelegatingClassLoader@6d427ec0 ]]

[2015-07-17T10:28:10.354+0200] [Payara 4.1] [FINER] [] [javax.enterprise.web.util] [tid: _ThreadID=58 _ThreadName=admin-listener(4)] [timeMillis: 1437121690354] [levelValue: 400] [CLASSNAME: org.glassfish.web.loader.WebappClassLoader] [METHODNAME: loadClass] [[Searching local repositories]]

[2015-07-17T10:28:10.354+0200] [Payara 4.1] [FINER] [] [javax.enterprise.web.util] [tid: _ThreadID=58 _ThreadName=admin-listener(4)] [timeMillis: 1437121690354] [levelValue: 400] [CLASSNAME: org.glassfish.web.loader.WebappClassLoader] [METHODNAME: findClass] [[findClass(aaa.bbb.ccc.communication.rest.ServiceImpl)]]

[2015-07-17T10:28:10.354+0200] [Payara 4.1] [FINER] [] [javax.enterprise.web.util] [tid: _ThreadID=58 _ThreadName=admin-listener(4)] [timeMillis: 1437121690354] [levelValue: 400] [CLASSNAME: org.glassfish.web.loader.WebappClassLoader] [METHODNAME: findClass] [[findClassInternal(aaa.bbb.ccc.communication.rest.ServiceImpl)]]

[2015-07-17T10:28:10.354+0200] [Payara 4.1] [FINER] [] [javax.enterprise.web.util] [tid: _ThreadID=58 _ThreadName=admin-listener(4)] [timeMillis: 1437121690354] [levelValue: 400] [CLASSNAME: org.glassfish.web.loader.WebappClassLoader] [METHODNAME: findClass] [[--> Passing on ClassNotFoundException]]

[2015-07-17T10:28:10.354+0200] [Payara 4.1] [WARNING] [AS-WEB-UTIL-00035] [javax.enterprise.web.util] [tid: _ThreadID=58 _ThreadName=admin-listener(4)] [timeMillis: 1437121690354] [levelValue: 900] [[Unable to load class aaa.bbb.ccc.communication.rest.ServiceImpl, reason: java.lang.ClassNotFoundException: aaa.bbb.ccc.communication.rest.ServiceImpl]]

[2015-07-17T10:28:10.354+0200] [Payara 4.1] [FINER] [] [javax.enterprise.web.util] [tid: _ThreadID=58 _ThreadName=admin-listener(4)] [timeMillis: 1437121690354] [levelValue: 400] [CLASSNAME: org.glassfish.web.loader.WebappClassLoader] [METHODNAME: loadClass] [[loadClass(aaa.bbb.ccc.communication.rest.ServiceImpl)]]

[2015-07-17T10:28:10.354+0200] [Payara 4.1] [FINER] [] [javax.enterprise.web.util] [tid: _ThreadID=58 _ThreadName=admin-listener(4)] [timeMillis: 1437121690354] [levelValue: 400] [CLASSNAME: org.glassfish.web.loader.WebappClassLoader] [METHODNAME: loadClass] [[Delegating to classloader1 EarClassLoader : urlSet = [URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/datasource-inject_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/datasource-inject_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/ldap-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/ldap-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/config-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/config-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/config-jca-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/config-jca-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/timer-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/timer-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/admin-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/admin-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/api-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/api-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/erp-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/erp-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/help-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/help-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/core-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/core-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/imsapi-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/imsapi-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/ml-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/ml-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/pm-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/pm-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/qm-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/qm-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/smt-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/smt-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/tr-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/tr-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/opm-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/opm-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/opma-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/opma-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/dbchecker-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/dbchecker-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/ams-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/ams-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/report-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/report-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/extsys-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/generated/ejb/ourapp/extsys-ejb_jar, URLEntry : file:/Data/OurApp/appserver/glassfish/domains/user1/applications/ourapp/script-ejb_jar/, URLEntry : file:/Data/OurApp/appserver/glassfish/ domains/user1/generated/ejb/ourapp/script-ejb_jar] doneCalled = false Parent -> org.glassfish.internal.api.DelegatingClassLoader@6d427ec0 ]]

smillidge commented 9 years ago

In this specific case this is a spurious warning. We will look to change the code to not spit out a warning in this specific case.

lprimak commented 8 years ago

This will also fix this: https://java.net/jira/browse/JAX_WS-802

lprimak commented 8 years ago

Pull request https://github.com/payara/Payara/pull/822

mikecroft commented 8 years ago

This fix has now been merged