Suspected corruption of domain.xml or domain dir or default-web.xml (was domain stops working)

glassfishrobot commented 16 years ago

For some strange reason the domain is corrupt or something. All applications including the admin tool won't work. This is the second time in a week this has happened. The only solution i can find is recreating the domain.

The admin app gives the following exception: type Exception report

message

descriptionThe server encountered an internal error () that prevented it from fulfilling this request.

exception

javax.servlet.ServletException: PWC1391: Servlet.init() for servlet FacesServlet threw exception

root cause

java.lang.IllegalStateException: Application was not properly initialized at startup, could not find Factory: javax.faces.context.FacesContextFactory

Log:

Oct 1, 2008 11:39:39 AM com.sun.enterprise.admin.servermgmt.launch.ASLauncher buildCommand INFO: /usr/lib/jvm/java-6-sun/bin/java -Dcom.sun.aas.instanceRoot=/home/mark/glassfish-v2ur2/domains/domain1 -Dcom.sun.aas.ClassPathPrefix= -Dcom.sun.aas.ClassPathSuffix= -Dcom.sun.aas.ServerClassPath= -Dcom.sun.aas.classloader.appserverChainJars.ee= -Dcom.sun.aas.classloader.appserverChainJars=admin-cli.jar,admin-cli-ee.jar,j2ee-svc.jar -Dcom.sun.aas.classloader.excludesList=admin-cli.jar,appserv-upgrade.jar,sun-appserv-ant.jar -Dcom.sun.aas.classloader.optionalOverrideableChain.ee= -Dcom.sun.aas.classloader.optionalOverrideableChain=webservices-rt.jar,webservices-tools.jar -Dcom.sun.aas.classloader.serverClassPath.ee=/lib/hadbjdbc4.jar,/home/mark/glassfish-v2ur2/lib/SUNWjdmk/5.1/lib/jdmkrt.jar,/lib/dbstate.jar,/lib/hadbm.jar,/lib/hadbmgt.jar,/opt/sun/mfwk/share/lib/mfwk_instrum_tk.jar -Dcom.sun.aas.classloader.serverClassPath=/home/mark/glassfish-v2ur2/lib/install/applications/jmsra/imqjmsra.jar,/home/mark/glassfish-v2ur2/imq/lib/jaxm-api.jar,/home/mark/glassfish-v2ur2/imq/lib/fscontext.jar,/home/mark/glassfish-v2ur2/imq/lib/imqbroker.jar,/home/mark/glassfish-v2ur2/imq/lib/imqjmx.jar,/home/mark/glassfish-v2ur2/lib/ant/lib/ant.jar,/home/mark/glassfish-v2ur2/lib/SUNWjdmk/5.1/lib/jdmkrt.jar -Dcom.sun.aas.classloader.sharedChainJars.ee=appserv-se.jar,appserv-ee.jar,jesmf-plugin.jar,/lib/dbstate.jar,/lib/hadbjdbc4.jar,jgroups-all.jar,/opt/sun/mfwk/share/lib/mfwk_instrum_tk.jar -Dcom.sun.aas.classloader.sharedChainJars=javaee.jar,/usr/lib/jvm/java-6-sun/lib/tools.jar,install/applications/jmsra/imqjmsra.jar,com-sun-commons-launcher.jar,com-sun-commons-logging.jar,/home/mark/glassfish-v2ur2/imq/lib/jaxm-api.jar,/home/mark/glassfish-v2ur2/imq/lib/fscontext.jar,/home/mark/glassfish-v2ur2/imq/lib/imqbroker.jar,/home/mark/glassfish-v2ur2/imq/lib/imqjmx.jar,/home/mark/glassfish-v2ur2/imq/lib/imqxm.jar,webservices-rt.jar,webservices-tools.jar,mail.jar,appserv-jstl.jar,jmxremote_optional.jar,/home/mark/glassfish-v2ur2/lib/SUNWjdmk/5.1/lib/jdmkrt.jar,activation.jar,appserv-rt.jar,appserv-admin.jar,appserv-cmp.jar,/home/mark/glassfish-v2ur2/updatecenter/lib/updatecenter.jar,/home/mark/glassfish-v2ur2/jbi/lib/jbi.jar,/home/mark/glassfish-v2ur2/imq/lib/imqjmx.jar,/home/mark/glassfish-v2ur2/lib/ant/lib/ant.jar,dbschema.jar -Dcom.sun.aas.configName=server-config -Dcom.sun.aas.configRoot=/home/mark/glassfish-v2ur2/config -Dcom.sun.aas.defaultLogFile=/home/mark/glassfish-v2ur2/domains/domain1/logs/server.log -Dcom.sun.aas.domainName=domain1 -Dcom.sun.aas.installRoot=/home/mark/glassfish-v2ur2 -Dcom.sun.aas.instanceName=server -Dcom.sun.aas.processLauncher=SE -Dcom.sun.aas.promptForIdentity=true -Dcom.sun.enterprise.config.config_environment_factory_class=com.sun.enterprise.config.serverbeans.AppserverConfigEnvironmentFactory -Dcom.sun.enterprise.overrideablejavaxpackages=javax.help,javax.portlet -Dcom.sun.enterprise.taglibs=appserv-jstl.jar,jsf-impl.jar -Dcom.sun.enterprise.taglisteners=jsf-impl.jar -Dcom.sun.updatecenter.home=/home/mark/glassfish-v2ur2/updatecenter -Ddomain.name=domain1 -Djava.endorsed.dirs=/home/mark/glassfish-v2ur2/lib/endorsed -Djava.ext.dirs=/usr/lib/jvm/java-6-sun/lib/ext:/usr/lib/jvm/java-6-sun/jre/lib/ext:/home/mark/glassfish-v2ur2/domains/domain1/lib/ext:/home/mark/glassfish-v2ur2/javadb/lib -Djava.library.path=/home/mark/glassfish-v2ur2/lib:/home/mark/glassfish-v2ur2/lib:/home/mark/glassfish-v2ur2/lib -Djava.security.auth.login.config=/home/mark/glassfish-v2ur2/domains/domain1/config/login.conf -Djava.security.policy=/home/mark/glassfish-v2ur2/domains/domain1/config/server.policy -Djava.util.logging.manager=com.sun.enterprise.server.logging.ServerLogManager -Djavax.management.builder.initial=com.sun.enterprise.admin.server.core.jmx.AppServerMBeanServerBuilder -Djavax.net.ssl.keyStore=/home/mark/glassfish-v2ur2/domains/domain1/config/keystore.jks -Djavax.net.ssl.trustStore=/home/mark/glassfish-v2ur2/domains/domain1/config/cacerts.jks -Djdbc.drivers=org.apache.derby.jdbc.ClientDriver -Djmx.invoke.getters=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -client -XX:+UnlockDiagnosticVMOptions -XX:MaxPermSize=192m -Xmx512m -XX:NewRatio=2 -XX:+LogVMOutput -XX:LogFile=/home/mark/glassfish-v2ur2/domains/domain1/logs/jvm.log -cp /home/mark/glassfish-v2ur2/lib/jhall.jar:/home/mark/glassfish-v2ur2/lib/appserv-launch.jar com.sun.enterprise.server.PELaunch start [#|2008-10-01T11:39:40.597+0200|INFO|sun-appserver9.1|javax.enterprise.system.core|_ThreadID=10;_ThreadName=main;|Starting Sun Java System Application Server 9.1_02 (build b04-fcs) ...|#]

[#|2008-10-01T11:39:44.944+0200|INFO|sun-appserver9.1|javax.enterprise.system.tools.admin|_ThreadID=12;_ThreadName=Thread-16;service:jmx:rmi:///jndi/rmi://mark-laptop:8686/jmxrmi;|ADM1504: Here is the JMXServiceURL for the Standard JMXConnectorServer: [service:jmx:rmi:///jndi/rmi://mark-laptop:8686/jmxrmi]. This is where the remote administrative clients should connect using the standard JMX connectors|#]

[#|2008-10-01T11:39:45.502+0200|SEVERE|sun-appserver9.1|javax.enterprise.system.tools.deployment|_ThreadID=13;_ThreadName=pool-1-thread-3;Deployment descriptor file WEB-INF/web.xml in archive [null].;1;1;Premature end of file.;_RequestID=580efeb0-7a55-4ddd-8e24-91020a45cc29;|"DPL8015: Invalid Deployment Descriptors in Deployment descriptor file WEB-INF/web.xml in archive [null]. Line 1 Column 1 – Premature end of file."|#]

[#|2008-10-01T11:39:45.505+0200|WARNING|sun-appserver9.1|javax.enterprise.system.tools.deployment|_ThreadID=13;_ThreadName=pool-1-thread-3;Deployment descriptor file WEB-INF/web.xml in archive [null]. Premature end of file.;_RequestID=580efeb0-7a55-4ddd-8e24-91020a45cc29;|DPL5407:Exception in populating default WebBundleDescriptor from default-web.xml Deployment descriptor file WEB-INF/web.xml in archive [null]. Premature end of file.|#]

[#|2008-10-01T11:40:14.765+0200|SEVERE|sun-appserver9.1|javax.enterprise.system.container.web|_ThreadID=15;_ThreadName=httpWorkerThread-4848-0;_RequestID=a756923a-fd1a-40ac-8af6-a81d273032a8;|WebModule[]StandardWrapper.Throwable java.lang.IllegalStateException: Application was not properly initialized at startup, could not find Factory: javax.faces.context.FacesContextFactory at javax.faces.FactoryFinder.getFactory(FactoryFinder.java:256) at javax.faces.webapp.FacesServlet.init(FacesServlet.java:142) at com.sun.enterprise.tools.admingui.servlet.DelayedInitFacesServlet.init(DelayedInitFacesServlet.java:81) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1178) at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:832) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214) at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265) at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:116)

| #] |

[#|2008-10-01T11:40:14.767+0200|SEVERE|sun-appserver9.1|javax.enterprise.system.container.web|_ThreadID=15;_ThreadName=httpWorkerThread-4848-0;_RequestID=a756923a-fd1a-40ac-8af6-a81d273032a8;|StandardWrapperValve[FacesServlet]: PWC1382: Allocate exception for servlet FacesServlet java.lang.IllegalStateException: Application was not properly initialized at startup, could not find Factory: javax.faces.context.FacesContextFactory at javax.faces.FactoryFinder.getFactory(FactoryFinder.java:256) at javax.faces.webapp.FacesServlet.init(FacesServlet.java:142) at com.sun.enterprise.tools.admingui.servlet.DelayedInitFacesServlet.init(DelayedInitFacesServlet.java:81) at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1178) at org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:832) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) at org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:637) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:568) at com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:813) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:263) at com.sun.enterprise.web.connector.grizzly.DefaultReadTask.doTask(DefaultReadTask.java:214) at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265) at com.sun.enterprise.web.connector.grizzly.WorkerThreadImpl.run(WorkerThreadImpl.java:116)

| #] |

Environment

Operating System: Linux Platform: PC

Affected Versions

[9.1peur2]

glassfishrobot commented 6 years ago

Issue Imported From: https://github.com/javaee/glassfish/issues/6374
Original Issue Raised By:@glassfishrobot
Original Issue Assigned To: @tjquinno
Original Issue Closed By:@glassfishrobot

glassfishrobot commented 16 years ago

@glassfishrobot Commented km said: Looks like this is a V2 issue. Also, there is some problem in admin GUI/JSF initialization. Let's start with admin_gui.

glassfishrobot commented 16 years ago

@glassfishrobot Commented ijuma said: Adding myself to cc list.

glassfishrobot commented 16 years ago

@glassfishrobot Commented anilam said: -> ken

glassfishrobot commented 16 years ago

@glassfishrobot Commented harpreet said: Please scrub issue and see if it is critical to v2.1.

glassfishrobot commented 16 years ago

@glassfishrobot Commented kenpaulsen said: This isn't due to the admin console application afaict. The bug report mentions all applications (including the admin console) are inaccessible. The following error looks suspicious to me:

[#|2008-10-01T11:39:45.502+0200|SEVERE|sun-appserver9.1|javax.enterprise.system.tools.deployment|_ThreadID=13;_ThreadName=pool-1-thread-3;Deployment descriptor file WEB-INF/web.xml in archive [null].;1;1;Premature end of file.;_RequestID=580efeb0-7a55-4ddd-8e24-91020a45cc29;|"DPL8015: Invalid Deployment Descriptors in Deployment descriptor file WEB-INF/web.xml in archive [null]. Line 1 Column 1 – Premature end of file."|#]

Archive (null)... is that valid? Perhaps this is the corruption in the domain.xml which is leading to this issue? How did this happen? What steps reproduce this error? Does this show up after a failed deployment? Successful deployment? Server sitting around? In a cluster environment? Developer environment? Do we have a domain.xml that is broken?

There simply isn't enough information in this bug report to investigate much further (unless someone familiar with the deployment code knows what the error above means and could be caused by).

I am going to reassign to the deployment team. If they cannot diagnose the problem, I would suggest downgrading this unless more details can be provided by the submitter so that we can evaluate this issue.

Ken

glassfishrobot commented 16 years ago

@glassfishrobot Commented @tjquinno said: As Ken pointed out earlier, there is not much information to go on so far. We can only conjecture as to what is happening without added details on how to recreate this.

This could be a case of a corrupted domain.xml. It's also possible that the domain's applications directory is becoming corrupted/deleted/renamed, so that the domain.xml points correctly to where the apps should be but they are not there any more for whatever reason. That would also explain why deployment is finding an EOF before it expects from the deployment descriptor(s).

Another possibility is that the application-root attribute value in the top-level domain element in domain.xml has been changed so it points to somewhere other than where the apps actually reside.

marksm1977, do any of these possibilities match up with what you are seeing?

glassfishrobot commented 16 years ago

@glassfishrobot Commented @tjquinno said: I just noticed something else. The innermost error refers to an EOF reading default-web.xml.

I can reproduce the errors by replacing my default-web.xml with an empty file.

Could anything like that be happening in your environment?

Or, could the $

{domain-directory}

/config directory have been renamed or moved or otherwise changed? That's the directory where GlassFish looks for the default-web.xml file.

Tim

glassfishrobot commented 16 years ago

@glassfishrobot Commented @tjquinno said: Updating with some e-mail traffic:

From Mark:

Sorry guys but in the process getting my project up again i deleted the corrupt domain and recreated it. It is difficult to reproduce. To me it is not clear why the domain gets corrupt. I was using netbeans to deploy my project and after about 50 deploys that day it suddenly got corrupt. This was the second time this happened. Maybe netbeans glassfish plugin modifies domain files and fail for some reason ending up with corrupt config/domain.

Mark

glassfishrobot commented 16 years ago

@glassfishrobot Commented @tjquinno said: Updating with more e-mail contents:

From Ken:

It would help a lot to add a comment to this bug about 50+ redeploys in NB (and maybe specifically how you do the redeploys – i.e. stop / start server manually, or after several times of entering / leaving NB, or staying in NB and doing 50+ simultaneous redeploys). Then we can ask testing to consider adding this test case to try to reproduce this issue.

If you run into this situation again, it would help us a lot to see your domain.xml file so that we can at least find out what is broken after it has been corrupted (assuming the domain.xml is at fault).

You may also want to backup your working domain.xml before it gets corrupted. This will give you a chance to replace that one file to see if that fixes the problem without a re-install. This would also be helpful information for understanding this issue better.

Thanks for taking the time to submit this issue and follow up with additional comments!

glassfishrobot commented 16 years ago

@glassfishrobot Commented @tjquinno said: Mark,

Is there a possibility that at any point NB might have been deploying again before the previous one had completed? Of course this should not fail but if that's what happened it helps focus our efforts to try to understand what's happened.

Also, if you do see this happen again, please capture the domain.xml and also the output of

find $

{domain-dir}

so we can see if the applications directory and its descendants look as they should.

Also please check the existence and contents of ${domain-dir}

/default-web.xml since as I mentioned earlier I could reproduce your errors by replacing that with an empty file.

Thanks.

glassfishrobot commented 16 years ago

@glassfishrobot Commented @tjquinno said: Marking as p3, pending more details on how to reproduce this.

glassfishrobot commented 16 years ago

@glassfishrobot Commented harpreet said: Since this is not reproducible consistently and occurs only when deploying app about 50 times through Netbeans. I am marking this issue off the v2.1 release. Issue should be investigated and fixed in a future release.

glassfishrobot commented 15 years ago

@glassfishrobot Commented sanandal said: "Reclassifying as P4 because this issue is not deemed "must fix" for this v2.1 release whose primary release driver is SailFin. This issue will be scrubbed after this release and will be given the right priority for the next release."

glassfishrobot commented 14 years ago

@glassfishrobot Commented hegalor@java.net said: I've a similar problem: My ear deployment fails after restarting domain: "java.lang.IllegalStateException: Application was not properly initialized at startup, could not find Factory: javax.faces.context.FacesContextFactory"

Restarting domain again does noit help. Only redeploying the application solve the problem

glassfishrobot commented 12 years ago

@glassfishrobot Commented tmueller said: Bulk update to change fix version to "not determined" for all issues still open but with a fix version for a released version.

glassfishrobot commented 12 years ago

@glassfishrobot Commented @tjquinno said: I'm closing this. It has been years since any new data has been added to the bug.

glassfishrobot commented 7 years ago

@glassfishrobot Commented This issue was imported from java.net JIRA GLASSFISH-6374

glassfishrobot commented 16 years ago

@glassfishrobot Commented Reported by marksm1977

glassfishrobot commented 12 years ago

@glassfishrobot Commented Marked as cannot reproduce on Monday, June 11th 2012, 7:35:32 am

eclipse-ee4j / glassfish

Suspected corruption of domain.xml or domain dir or default-web.xml (was domain stops working) #6374

Environment

Affected Versions