Closed glassfishrobot closed 11 years ago
glassfishrobot
commented
6 years ago glassfishrobot
commented
11 years ago @glassfishrobot Commented @tjquinno said: There are at least two possibilities, and maybe more we haven't thought of yet.
The first, the one you have mentioned, is that the files are indeed being deleted. Unless the application containing the app client is being undeployed, GlassFish should not be deleting the files itself. That would happen only during undeployment. You could try manually changing the file permissions on the files that disappear to prevent their deletion, then see if any exceptions appear in the GlassFish log or other logs.
Here is a second possibility. Since the problem seems more likely (always?) to happen after a reboot, the IP address of the system could change if the addresses are assigned using DHCP which could confuse things. For example, the cached main JNLP document might contain what used to be the correct IP address for the server, but the reboots might have assigned that IP address to another server (just by luck) that does not have that app installed. Could you review the IP addresses assigned to the servers before and after the reboots and see if that might be contributing to the problem?
glassfishrobot
commented
11 years ago @glassfishrobot Commented skrall said: The files are certainly being deleted, I'm not sure how or why. But the files are gone. If we goto the Admin Console, and look at the applications. Glassfish still sees the Web Start applications, so they haven't been undeployed - at least totally. We will change the permissions on the file, see if that generates any logging.
The IP Addresses are static, but each machine does have 2 IpAddresses. One for an internal network, and one for an external facing network. There is a 192.168.0.x network which is very small, and a 10.30.x network wich is larger. All the traffic for the Web Start applications will happen over the 192.168.0.x network.
Here is the output of tree on a machine that has had the issue (Files deleted).
. ├── __admingui ├── Common ├── Dashboard ├── eforms-ear-8.0.0.0-SNAPSHOT │ └── eForms-war-8.0.0.0-SNAPSHOT_war ├── ejb-timer-service-app ├── genericra ├── Makeline ├── Orderentry ├── Profit-ear-8.0.0.0-SNAPSHOT │ ├── ProfitApi-war-8.0.0.0-SNAPSHOT_war │ ├── Profit-ejb-8.0.0.0-SNAPSHOT_jar │ └── ProfitRestfulServices-8.0.0.0-SNAPSHOT_war ├── ProfitMiddleware-ear-8.0.0.0-SNAPSHOT │ └── Profit-messaging-8.0.0.0-SNAPSHOT_jar ├── System ├── Timeclock └── wstx-services
Here is the tree from the generated/xml directory on a machine that is working correctly. (We've only launched POS-terminal, the other two web start applications are freshly deployed.)
. ├── __admingui ├── Common ├── Dashboard ├── DriverDispatch-ear-8.0.0.0-SNAPSHOT │ ├── DriverDispatch-ear-8.0.0.0-SNAPSHOTClient.jar │ ├── DriverDispatch_jar │ │ └── DriverDispatchClient.jar │ └── signed │ └── DriverDispatch_jar ├── eforms-ear-8.0.0.0-SNAPSHOT │ └── eForms-war-8.0.0.0-SNAPSHOT_war ├── ejb-timer-service-app ├── genericra ├── Makeline ├── Orderentry ├── POS-terminal-ear-8.0.0.0-SNAPSHOT │ ├── POS-terminal-ear-8.0.0.0-SNAPSHOTClient.jar │ ├── POS-terminal_jar │ │ └── POS-terminalClient.jar │ └── signed │ ├── POS-terminal-ear-8.0.0.0-SNAPSHOTClient │ │ ├── AnimatedTransitions-1.0.jar │ │ ├── aopalliance-1.0.jar │ │ ├── commons-codec-1.5.jar │ │ ├── commons-collections-3.2.1.jar │ │ ├── commons-exec-1.1.jar │ │ ├── commons-logging-1.1.jar │ │ ├── dpuareu-2.1.1.jar │ │ ├── jms-1.1.jar │ │ ├── jms-6.3.jar │ │ ├── lizard-1.1.3.jar │ │ ├── logback-classic-1.0.7.jar │ │ ├── logback-core-1.0.7.jar │ │ ├── posrealm-1.0.4.jar │ │ ├── POS-terminal.jar │ │ ├── ProfitApiWebServiceClients-8.0.0.0-SNAPSHOT.jar │ │ ├── ProfitServices-8.0.0.0-SNAPSHOT.jar │ │ ├── slf4j-api-1.7.2.jar │ │ ├── spring-aop-3.0.6.RELEASE.jar │ │ ├── spring-asm-3.0.6.RELEASE.jar │ │ ├── spring-beans-3.0.6.RELEASE.jar │ │ ├── spring-context-3.0.6.RELEASE.jar │ │ ├── spring-core-3.0.6.RELEASE.jar │ │ ├── spring-expression-3.0.6.RELEASE.jar │ │ ├── spring-jms-3.0.6.RELEASE.jar │ │ ├── spring-tx-3.0.6.RELEASE.jar │ │ └── timingframework-classic-1.1.jar │ ├── POS-terminal-ear-8.0.0.0-SNAPSHOTClient.jar │ └── POS-terminal_jar │ └── POS-terminalClient.jar ├── ProfitDesktopApp-ear-8.0.0.0-SNAPSHOT │ ├── ProfitDesktopApp-ear-8.0.0.0-SNAPSHOTClient.jar │ ├── ProfitDesktopApp_jar │ │ └── ProfitDesktopAppClient.jar │ └── signed │ └── ProfitDesktopApp_jar ├── Profit-ear-8.0.0.0-SNAPSHOT │ ├── ProfitApi-war-8.0.0.0-SNAPSHOT_war │ ├── Profit-ejb-8.0.0.0-SNAPSHOT_jar │ └── ProfitRestfulServices-8.0.0.0-SNAPSHOT_war ├── ProfitMiddleware-ear-8.0.0.0-SNAPSHOT │ └── Profit-messaging-8.0.0.0-SNAPSHOT_jar ├── System ├── Timeclock └── wstx-services
glassfishrobot
commented
11 years ago @glassfishrobot Commented skrall said: I just noticed a stack trace right above the one I posted previously. Here is the stack trace that was just before. I see these in the logs from time to time, but it may be a large problem when they happen to web start apps? I'm not even sure if the stack trace is related, but it seems suspicious that it happened right before the other one. And we do include a jms-1.1.jar in our web start project.
[#|2013-01-07T09:08:25.266-0500|SEVERE|glassfish3.1.2|javax.enterprise.system.core.com.sun.enterprise.v3.server|_ThreadID=48;_ThreadName=Thread-2;|Exception while visiting javax/jms/TopicSession.class of size 852 java.lang.NullPointerException at org.glassfish.hk2.classmodel.reflect.impl.TypesImpl.getType(TypesImpl.java:78) at org.glassfish.hk2.classmodel.reflect.impl.ModelClassVisitor.visit(ModelClassVisitor.java:119) at org.objectweb.asm.ClassReader.accept(Unknown Source) at org.objectweb.asm.ClassReader.accept(Unknown Source) at org.glassfish.hk2.classmodel.reflect.Parser$5.on(Parser.java:363) at com.sun.enterprise.v3.server.ReadableArchiveScannerAdapter.handleEntry(ReadableArchiveScannerAdapter.java:171) at com.sun.enterprise.v3.server.ReadableArchiveScannerAdapter.onSelectedEntries(ReadableArchiveScannerAdapter.java:133) at org.glassfish.hk2.classmodel.reflect.Parser.doJob(Parser.java:348) at org.glassfish.hk2.classmodel.reflect.Parser.access$300(Parser.java:70) at org.glassfish.hk2.classmodel.reflect.Parser$3.call(Parser.java:307) at org.glassfish.hk2.classmodel.reflect.Parser$3.call(Parser.java:296) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722)
| #] |
[#|2013-01-07T09:08:25.824-0500|SEVERE|glassfish3.1.2|javax.enterprise.system.tools.deployment.org.glassfish.deployment.common|_ThreadID=1;_ThreadName=Thread-2;|Exception while invoking class org.glassfish.appclient.server.core.AppClientServerApplication start method
java.lang.RuntimeException: java.io.FileNotFoundException: /opt/glassfish3/glassfish/domains/domain1/generated/xml/POS-terminal-ear-8.0.0.0-SNAPSHOT/POS-terminal_jar/POS-terminalClient.jar
at org.glassfish.appclient.server.core.jws.JavaWebStartInfo$1.run(JavaWebStartInfo.java:259)
at org.glassfish.appclient.server.core.jws.JavaWebStartState.transition(JavaWebStartState.java:85)
at org.glassfish.appclient.server.core.jws.JavaWebStartInfo.start(JavaWebStartInfo.java:253)
at org.glassfish.appclient.server.core.AppClientServerApplication.start(AppClientServerApplication.java:150)
at org.glassfish.appclient.server.core.AppClientServerApplication.start(AppClientServerApplication.java:139)
at org.glassfish.internal.data.EngineRef.start(EngineRef.java:130)
at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:269)
at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:301)
at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:461)
at com.sun.enterprise.v3.server.ApplicationLoaderService.processApplication(ApplicationLoaderService.java:375)
at com.sun.enterprise.v3.server.ApplicationLoaderService.postConstruct(ApplicationLoaderService.java:219)
at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:131)
at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:91)
at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:82)
at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67)
at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:139)
at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:78)
at com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:253)
at com.sun.enterprise.v3.server.AppServerStartup.doStart(AppServerStartup.java:145)
at com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:136)
at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:79)
at com.sun.enterprise.glassfish.bootstrap.GlassFishDecorator.start(GlassFishDecorator.java:63)
at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishImpl.start(OSGiGlassFishImpl.java:69)
at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:117)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97)
at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55)
Caused by: java.io.FileNotFoundException: /opt/glassfish3/glassfish/domains/domain1/generated/xml/POS-terminal-ear-8.0.0.0-SNAPSHOT/POS-terminal_jar/POS-terminalClient.jar
at org.glassfish.appclient.server.core.jws.servedcontent.AutoSignedContent.
| #] |
glassfishrobot
commented
11 years ago @glassfishrobot Commented @tjquinno said: Thanks for the update. The new stack trace doesn't tell (not to me at least) a lot of information; it does not track back to an identifiable part of the code.
That said, the Java Web Start-related stack trace is run only during deployment or start-up, and in fact in many ways GlassFish does much the same thing during restart and during deployment. One possible explanation could be that the system detects the NullPointerException (in the new exception) during start-up while processing part of the app, and in its attempt to recover the system or part of the system is incorrectly cleaning up the directories you mentioned. During deployment, cleaning up after such an error make sense, but not during restart.
Am I correct that these errors are reported during restart?
My suspicion is that there are at least two errors you've revealed: One is that the NPE should not occur, and the other is that if an error does occur the files should not disappear.
I'll check with others on the team to see if they have any ideas about what might be triggering the original NPE and the file deletions.
glassfishrobot
commented
11 years ago @glassfishrobot Commented skrall said: Yes, the errors are reporting after a restart. The applications would have been deployed fine for a period of time (usually days or weeks). But after a restart, sometimes, the files disappear.
I noticed a couple of other bugs related to the NPE that were open. We see the message on some other class files too, but don't seem to notice any ill effects. The other bugs are #18702 during an attempt to authenticate."), #18609 and #18513
glassfishrobot
commented
11 years ago @glassfishrobot Commented @honghzzhang said: Tim and I had some discussions on this. One theory Tim has was if the application failed to load during server restart for some reason (for example due to this NPE or other cause), the deployment failure roll back logic would try to clean up the generated directories and resulted in the JWS file being cleaned up.
So I have updated this part of the deployment logic to only clean up for initial deployment failure and not for subsequent loading (like during server restart) so the JWS files will not be removed if the application failed to load during server start.
We however are not too sure why the application would fail to load, that particular NPE as you found out most times seems harmless, maybe it's due to some other cause. But in any case the NPE issue is tracked through other issues already, so we will not create another new issue for that. The fix I have checked in will at least address the problem that the generated files got deleted when the application failed to load during restart when they should not be.
glassfishrobot
commented
11 years ago @glassfishrobot Commented skrall said: Is there a way we can get a patched 3.1.2.2 jar file? This is fairly painful for us, we keep having server's intermittently have this issue, which causes our application to not start.
glassfishrobot
commented
11 years ago @glassfishrobot Commented @honghzzhang said: Let me see if I can make a 3.1.2 patch jar and send to you...
glassfishrobot
commented
11 years ago @glassfishrobot Commented skrall said: Thanks, 3.1.2.2, you just have 3.1.2 in your comment, just want to avoid confusion.
And we will be able to test and let you know if it totally fixes the issue or not.
glassfishrobot
commented
11 years ago @glassfishrobot Commented @honghzzhang said: Thanks for the clarification. I have built a patch jar locally for 3.1.2.2 now and I will send it to you via email (as the attachments are disabled in the JIRA issue now).
glassfishrobot
commented
7 years ago @glassfishrobot Commented This issue was imported from java.net JIRA GLASSFISH-19498
glassfishrobot
commented
11 years ago @glassfishrobot Commented Reported by skrall
glassfishrobot
commented
11 years ago @glassfishrobot Commented Marked as fixed on Tuesday, January 8th 2013, 6:19:31 am
Somewhat rarely, our WebStart application's/glassfish/domains/domain1/generated/xml directory is getting deleted. We can not seem to determine the cause, or reproduce consistently. But it does happen fairly often. We have about 10 QA machines, and this happens to 1 or 2 a week it seems.
It seems to happen most of the time after a machine reboot. We have 3 WebStart applications, and all 3 of their generated/xml directories, and all the contents completely disappear. But our web applications generated/xml directories remain (Although they are empty, but they were always empty).
There was another report of a similar issue, but it was closed as can not reproduce. #17341
Any thoughts on how we can help debug this? I'm sure it will be tricky to reproduce?
Here is the last message before server shutdown (related to the Web Start app) [#|2013-01-07T08:55:06.276-0500|INFO|glassfish3.1.2|javax.enterprise.system.container.appclient.org.glassfish.appclient.server.core|_Threa dID=9658;_ThreadName=Thread-2;|ACDEPL104: Java Web Start services stopped for the app client POS-terminal-ear-8.0.0.0-SNAPSHOT/POS-termina l.jar|#]
And here is an error after the reboot:
[#|2013-01-07T09:08:25.824-0500|SEVERE|glassfish3.1.2|javax.enterprise.system.tools.deployment.org.glassfish.deployment.common|_ThreadID=1 ;_ThreadName=Thread-2;|Exception while invoking class org.glassfish.appclient.server.core.AppClientServerApplication start method java.lang.RuntimeException: java.io.FileNotFoundException: /opt/glassfish3/glassfish/domains/domain1/generated/xml/POS-terminal-ear-8.0.0.0-SNAPSHOT/POS-terminal_jar/POS-terminalClient.jar at org.glassfish.appclient.server.core.jws.JavaWebStartInfo$1.run(JavaWebStartInfo.java:259) at org.glassfish.appclient.server.core.jws.JavaWebStartState.transition(JavaWebStartState.java:85) at org.glassfish.appclient.server.core.jws.JavaWebStartInfo.start(JavaWebStartInfo.java:253) at org.glassfish.appclient.server.core.AppClientServerApplication.start(AppClientServerApplication.java:150) at org.glassfish.appclient.server.core.AppClientServerApplication.start(AppClientServerApplication.java:139) at org.glassfish.internal.data.EngineRef.start(EngineRef.java:130) at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:269) at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:301) at com.sun.enterprise.v3.server.ApplicationLifecycle.deploy(ApplicationLifecycle.java:461) at com.sun.enterprise.v3.server.ApplicationLoaderService.processApplication(ApplicationLoaderService.java:375) at com.sun.enterprise.v3.server.ApplicationLoaderService.postConstruct(ApplicationLoaderService.java:219) at com.sun.hk2.component.AbstractCreatorImpl.inject(AbstractCreatorImpl.java:131) at com.sun.hk2.component.ConstructorCreator.initialize(ConstructorCreator.java:91) at com.sun.hk2.component.AbstractCreatorImpl.get(AbstractCreatorImpl.java:82) at com.sun.hk2.component.SingletonInhabitant.get(SingletonInhabitant.java:67) at com.sun.hk2.component.EventPublishingInhabitant.get(EventPublishingInhabitant.java:139) at com.sun.hk2.component.AbstractInhabitantImpl.get(AbstractInhabitantImpl.java:78) at com.sun.enterprise.v3.server.AppServerStartup.run(AppServerStartup.java:253) at com.sun.enterprise.v3.server.AppServerStartup.doStart(AppServerStartup.java:145) at com.sun.enterprise.v3.server.AppServerStartup.start(AppServerStartup.java:136) at com.sun.enterprise.glassfish.bootstrap.GlassFishImpl.start(GlassFishImpl.java:79) at com.sun.enterprise.glassfish.bootstrap.GlassFishDecorator.start(GlassFishDecorator.java:63) at com.sun.enterprise.glassfish.bootstrap.osgi.OSGiGlassFishImpl.start(OSGiGlassFishImpl.java:69) at com.sun.enterprise.glassfish.bootstrap.GlassFishMain$Launcher.launch(GlassFishMain.java:117) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.sun.enterprise.glassfish.bootstrap.GlassFishMain.main(GlassFishMain.java:97) at com.sun.enterprise.glassfish.bootstrap.ASMain.main(ASMain.java:55) Caused by: java.io.FileNotFoundException: /opt/glassfish3/glassfish/domains/domain1/generated/xml/POS-terminal-ear-8.0.0.0-SNAPSHOT/POS-terminal_jar/POS-terminalClient.jar at org.glassfish.appclient.server.core.jws.servedcontent.AutoSignedContent.(AutoSignedContent.java:66)
at org.glassfish.appclient.server.core.jws.JavaWebStartInfo.createSignedStaticContent(JavaWebStartInfo.java:605)
at org.glassfish.appclient.server.core.jws.JavaWebStartInfo.createAndAddSignedStaticContent(JavaWebStartInfo.java:596)
at org.glassfish.appclient.server.core.jws.JavaWebStartInfo.createAndAddSignedStaticContentFromGeneratedFile(JavaWebStartInfo.java:576)
at org.glassfish.appclient.server.core.jws.JavaWebStartInfo.initClientStaticContent(JavaWebStartInfo.java:486)
at org.glassfish.appclient.server.core.jws.JavaWebStartInfo.addClientContentToHTTPAdapters(JavaWebStartInfo.java:456)
at org.glassfish.appclient.server.core.jws.JavaWebStartInfo.startJWSServices(JavaWebStartInfo.java:326)
at org.glassfish.appclient.server.core.jws.JavaWebStartInfo.access$000(JavaWebStartInfo.java:98)
at org.glassfish.appclient.server.core.jws.JavaWebStartInfo$1.run(JavaWebStartInfo.java:257)
... 29 more
| #] |
Environment
Java7u9 64Bit, Linux Fedora 17
Affected Versions
[3.1.2.2]