Getting OutOfMemory error deploying a war file

[glassfishrobot]

When I try and deploy my project's war file, sometimes i get an OutOfMemory error:

The server is set with PermGen space at 512Mb, -Xmx1024Mb. I've disabled any other webapps on the server, and done a fresh reboot before deploying. Before I deploy, looking at the windows task manager, glassfish is taking 218Mb or so of memory; after a failed deploy it is up to 778Mb, and stays there until I shut down glassfish.

[#|2008-03-18T13:04:03.405-0700|SEVERE|sun- appserver9.1|javax.enterprise.system.tools.deployment|_ThreadID=22;_ThreadName=Thread- 51;_RequestID=93312159-bd80-4b7c-8a62-ee9ed7631a0d;|Exception occured in J2EEC Phasejava.lang.OutOfMemoryError com.sun.enterprise.deployment.backend.IASDeploymentException: Error loading deployment descriptors for module [Pythia] – null at com.sun.enterprise.deployment.backend.Deployer.loadDescriptors(Deployer.java:390) at com.sun.enterprise.deployment.backend.ModuleDeployer.loadDescriptors(ModuleDeployer.java:423) at com.sun.enterprise.deployment.backend.WebModuleDeployer.deploy(WebModuleDeployer.java:157) at com.sun.enterprise.deployment.backend.ModuleDeployer.doRequestFinish(ModuleDeployer.java:179) at com.sun.enterprise.deployment.phasing.J2EECPhase.runPhase(J2EECPhase.java:191) at com.sun.enterprise.deployment.phasing.DeploymentPhase.executePhase(DeploymentPhase.java:108) at com.sun.enterprise.deployment.phasing.PEDeploymentService.executePhases(PEDeploymentService.java :919) at com.sun.enterprise.deployment.phasing.PEDeploymentService.deploy(PEDeploymentService.java:279) at com.sun.enterprise.deployment.phasing.PEDeploymentService.deploy(PEDeploymentService.java:788) at com.sun.enterprise.management.deploy.DeployThread.deploy(DeployThread.java:187) at com.sun.enterprise.management.deploy.DeployThread.run(DeployThread.java:223) Caused by: java.lang.OutOfMemoryError at java.util.zip.Inflater.init(Native Method) at java.util.zip.Inflater.(Inflater.java:81) at java.util.zip.ZipFile.getInflater(ZipFile.java:267) at java.util.zip.ZipFile.getInputStream(ZipFile.java:212) at java.util.zip.ZipFile.getInputStream(ZipFile.java:180) at java.util.jar.JarFile.getInputStream(JarFile.java:383) at com.sun.enterprise.deployment.annotation.impl.ModuleScanner.addScanJar(ModuleScanner.java:96) at com.sun.enterprise.deployment.annotation.impl.WarScanner.(WarScanner.java:101) at com.sun.enterprise.deployment.annotation.factory.ScannerFactory.createScanner(ScannerFactory.java:7 8) at com.sun.enterprise.deployment.archivist.Archivist.processAnnotations(Archivist.java:403) at com.sun.enterprise.deployment.archivist.Archivist.readAnnotations(Archivist.java:346) at com.sun.enterprise.deployment.archivist.Archivist.readDeploymentDescriptors(Archivist.java:318) at com.sun.enterprise.deployment.archivist.Archivist.open(Archivist.java:213) at com.sun.enterprise.deployment.archivist.ApplicationArchivist.openArchive(ApplicationArchivist.java:763 ) at com.sun.enterprise.deployment.archivist.ApplicationArchivist.openArchive(ApplicationArchivist.java:744 ) at com.sun.enterprise.deployment.backend.Deployer.loadDescriptors(Deployer.java:349) ... 10 more

| #] |

Environment

Operating System: other Platform: PC

Affected Versions

[9.1peur1]

[glassfishrobot]

• Issue Imported From: https://github.com/javaee/glassfish/issues/4452
• Original Issue Raised By:@glassfishrobot
• Original Issue Assigned To: @honghzzhang

[glassfishrobot]

@glassfishrobot Commented hexghost said: This is on Windows 2003 server by the way.

[glassfishrobot]

@glassfishrobot Commented hexghost said: Created an attachment (id=1387) normal, pre-deploy memory usage for GF.

[glassfishrobot]

@glassfishrobot Commented hexghost said: Created an attachment (id=1388) glassfish process AFTER deploy failed, memory usage stays at that level until i kill server.

[glassfishrobot]

@glassfishrobot Commented @honghzzhang said: Which build of the glassfish were you using? And which jdk were you using? Is it possible to share your war file for us to reproduce this problem? We cannot reproduce it by deploying a simple war obviously.

[glassfishrobot]

@glassfishrobot Commented hexghost said: I'm also using the admin GUI to deploy my war file. If I upload my war file to the server directly, run jar -xf Mywar.war to make it an uncompressed directory, and then try (through the same admin GUI) to just deploy that directory, it works fine. So, definitely looks like something happens during unzip to cause this.

[glassfishrobot]

@glassfishrobot Commented hexghost said: I'm on Glassfish 2 UR1, and using jdk1.6.0_05. Unfortunately I can't share the war file. Is there any other info I can provide instead?

[glassfishrobot]

@glassfishrobot Commented @honghzzhang said: It's probably not specific to the unzipping archive to directory as the previous stack trace you posted is actually part of the deployment (process the annotations). It's probably more about the state of the server when you did those operations. In any case, to start with, can you use jmap to collect the heap information?

jmap -heap

[glassfishrobot]

@glassfishrobot Commented hexghost said: I had two hints that lead me to believe it might indeed be something with the unzipping:

1. After a failure to deploy, I can see my application's directory has been created in glassfish/domains/domain1/applications/j2ee-modules/MyApp. However, not all the files are there, just a few that made the unzip.

2. I can successfully deploy my application if I un-archive the war, then ask Glassfish to deploy the directory of my application.

I don't see jmap as coming in my jdk - is this something i need to download? Or am I just not looking in the right place for it?

[glassfishrobot]

@glassfishrobot Commented jluehe said: Look in the bin directory of your jdk installation (and not in jre/bin!).

[glassfishrobot]

@glassfishrobot Commented hexghost said: Trying to get jmap to work on Windows, but I get: C:\Program Files\Java\jdk1.6.0_04\bin>jmap -dump:format=b,file=heap.bin 3928 3928: Not enough storage is available to process this command

[glassfishrobot]

@glassfishrobot Commented @honghzzhang said: How much memory and disk space in your system? What happens if you just run "jmap -heap "?

[glassfishrobot]

@glassfishrobot Commented hexghost said: Well I just tried to deploy a new version of our app, using the expanded directory instead of a war file, and received the same error :\ So maybe it has nothing to do with unzipping, although thats what the exception states (see below). As far as memory, the server has 2GB of RAM, and 121GB free space on the drive which the server resides on.

[#|2008-03-25T12:23:12.132-0700|SEVERE|sun- appserver9.1|javax.enterprise.system.tools.deployment|_ThreadID=22;_ThreadName=Thread- 51;_RequestID=109869fc-0375-4c9b-bd9b-450429fee1e1;|Exception occured in J2EEC Phasejava.lang.OutOfMemoryError com.sun.enterprise.deployment.backend.IASDeploymentException: Error loading deployment descriptors for module [Pythia] – null at com.sun.enterprise.deployment.backend.Deployer.loadDescriptors(Deployer.java:390) at com.sun.enterprise.deployment.backend.ModuleDeployer.loadDescriptors(ModuleDeployer.java:423) at com.sun.enterprise.deployment.backend.WebModuleDeployer.deploy(WebModuleDeployer.java:157) at com.sun.enterprise.deployment.backend.ModuleDeployer.doRequestFinish(ModuleDeployer.java:179) at com.sun.enterprise.deployment.phasing.J2EECPhase.runPhase(J2EECPhase.java:191) at com.sun.enterprise.deployment.phasing.DeploymentPhase.executePhase(DeploymentPhase.java:108) at com.sun.enterprise.deployment.phasing.PEDeploymentService.executePhases(PEDeploymentService.java :919) at com.sun.enterprise.deployment.phasing.PEDeploymentService.deploy(PEDeploymentService.java:279) at com.sun.enterprise.deployment.phasing.PEDeploymentService.deploy(PEDeploymentService.java:788) at com.sun.enterprise.management.deploy.DeployThread.deploy(DeployThread.java:187) at com.sun.enterprise.management.deploy.DeployThread.run(DeployThread.java:223) Caused by: java.lang.OutOfMemoryError at java.util.zip.Inflater.init(Native Method) at java.util.zip.Inflater.(Inflater.java:81) at java.util.zip.ZipFile.getInflater(ZipFile.java:267) at java.util.zip.ZipFile.getInputStream(ZipFile.java:212) at java.util.zip.ZipFile.getInputStream(ZipFile.java:180) at java.util.jar.JarFile.getInputStream(JarFile.java:383) at com.sun.enterprise.deployment.annotation.impl.ModuleScanner.addScanJar(ModuleScanner.java:96) at com.sun.enterprise.deployment.annotation.impl.WarScanner.(WarScanner.java:101) at com.sun.enterprise.deployment.annotation.factory.ScannerFactory.createScanner(ScannerFactory.java:7 8) at com.sun.enterprise.deployment.archivist.Archivist.processAnnotations(Archivist.java:403) at com.sun.enterprise.deployment.archivist.Archivist.readAnnotations(Archivist.java:346) at com.sun.enterprise.deployment.archivist.Archivist.readDeploymentDescriptors(Archivist.java:318) at com.sun.enterprise.deployment.archivist.Archivist.open(Archivist.java:213) at com.sun.enterprise.deployment.archivist.ApplicationArchivist.openArchive(ApplicationArchivist.java:763 ) at com.sun.enterprise.deployment.archivist.ApplicationArchivist.openArchive(ApplicationArchivist.java:744 ) at com.sun.enterprise.deployment.backend.Deployer.loadDescriptors(Deployer.java:349) ... 10 more

| #] |

[glassfishrobot]

@glassfishrobot Commented hexghost said: --What happens if you just run --"jmap -heap "?

jmap won't let me run that, it just prints the command line usage if I try.

[glassfishrobot]

@glassfishrobot Commented hexghost said: hzhang_jn: If i shutdown glassfish the service and run it via the command line, then I'm able to generate a heap file dump. It's rather large (102mb on server first starting up), what's the next step?

[glassfishrobot]

@glassfishrobot Commented @honghzzhang said: Hi, If you shut down the glassfish service and then collect a heap data, what

did you pass to the jmap command? One other thing that might help is to see which instances are eating up so much of the heap space. You can get this info by using jmap -histo or jmap -histo:live where the latter command prints only those instances that are not candidates for garbage collection. How big is your war file? Can you try to set your permgen to a higher number (e.g. 1024M) to see if it helps work around the problem?

[glassfishrobot]

@glassfishrobot Commented hexghost said: Sorry I should have been more specific. Normally Glassfish runs as a service on the windows server, which means it runs under the SYSTEM account user (or LOCAL SYSTEM, something like that). I wasn't able to run the jmap tools, logged in as administrator, against processes running as SYSTEM user. So I shut down the service and instead ran it via the command line (again, as the administrator user) which meant I could then run jmap. Sooo, it was still running, just as the same user I was logged in as instead of the system user.

My war file is about 19Mb; I'll try bumping the permgen up like you suggested and get back to you with the results, as well as the results of the jmap -histo commands.

Thanks!

[glassfishrobot]

@glassfishrobot Commented hexghost said: I finally found the jvm crash logs (they were in the domain\config directory). Every time I get a crash related to this deploying problem, the following section of the crash log looks the same:

Heap PSYoungGen total 263360K, used 171081K [0x4e400000, 0x609c0000, 0x63950000) eden space 241664K, 61% used [0x4e400000,0x575e4b78,0x5d000000) from space 21696K, 99% used [0x5d9b0000,0x5eedda10,0x5eee0000) to space 27520K, 0% used [0x5eee0000,0x5eee0000,0x609c0000) PSOldGen total 174784K, used 31176K [0x23950000, 0x2e400000, 0x4e400000) object space 174784K, 17% used [0x23950000,0x257c23d8,0x2e400000) PSPermGen total 68608K, used 68543K [0x03950000, 0x07c50000, 0x23950000) object space 68608K, 99% used [0x03950000,0x07c3fdf8,0x07c50000)

I see a few 99% - is there something I need to set in the jvm options for glassfish to fix this? I already have:

-XX:MaxPermSize=512m -Xmx1024m -Xms256m -XX:PermSize=512m

[glassfishrobot]

@glassfishrobot Commented @honghzzhang said: Sorry, we don't know how to interpret this crash.log.

Have you tried to set your perm size to 1024M to see if it makes a difference?

Also did you try to use the jmap -histo to see which instances are eating up the heap space?

[glassfishrobot]

@glassfishrobot Commented harpreet said: Not enough information to investigate the issue. Marking it as a P3 and removing it from 9.1.1 list and marking target release to 9.2. We have the server running for considerably long periods of time (north of 24x3) for SailFin and have not seen OOM exceptions in the recent past - hence the downgrade.

[glassfishrobot]

@glassfishrobot Commented sanandal said: "Reclassifying as P4 because this issue is not deemed "must fix" for this v2.1 release whose primary release driver is SailFin. This issue will be scrubbed after this release and will be given the right priority for the next release."

[glassfishrobot]

@glassfishrobot Commented kawazu@java.net said: Subscribing as I frequently tend to see this issue with one of my .war files as well (which is the largest one around, more or less 29 meg in size). Usually (re-)deploying this once or twice after restarting the app server works, until eventually the "OutOfMemoryError appears. Looking at jconsole, I so far failed to figure out which memory region is filling up here. Could provide the .war file for testing purposes if there's a chance of having it treated confidentially. Application contains spring, JAX-WS web services and some apache stuff (tika metadata extractor framework). Platform is Ubuntu GNU/Linux, JDK 1.6.0_17 (vanilla sun.com download), app server is Sun GlassFish Enterprise Server v2.1 (9.1.1) (build b60e-fcs).

[glassfishrobot]

@glassfishrobot Commented seanespn said: We're seeing this exception as well.

What version of Glassish open source has it supposedly been fixed in?

[glassfishrobot]

@glassfishrobot Commented @honghzzhang said: what version of the glassfish are you using? Do you see the problem in 3.1? We are not doing any more active development on 2.x.

[glassfishrobot]

@glassfishrobot Commented seanespn said: We're using version Glassish open source v2.1.

I saw the note that it's been fixed in 9.2 so I'm wondering what version we'd need to update to get the fix.

Is 9.2 v2.2 or v3.something?

[glassfishrobot]

@glassfishrobot Commented @honghzzhang said: v2 and v3 are of different code base, and 9.x corresponds to v2.x. I think 9.2 is v2.1.1?

[glassfishrobot]

@glassfishrobot Commented seanespn said: Is there a link somewhere to all the bugs fixed in version 2.1.1?

Would like to see that while weighing whether to upgrade our production DAS to (hopefully) eliminate the OOM error.

FYI, I downloaded 2.1.1, upgraded my dev instance and now I don't see the OOM error anymore at deploy time.

So perhaps the fix to this issue is in 2.1.1. But I'm not 100% sure since the OOM seems somewhat random: the same exact ear (in my case) will work on one 2.1 domain but cause the error on another 2.1 domain.

[glassfishrobot]

@glassfishrobot Commented seanespn said: For the next poor sucker who runs into this:

we were hitting this consistently with our production DAS. Deploy an ear, boom dead.

We REDUCED our Java heap (-Xmx) from 1024 to 512 for the DAS and suddenly the deploy worked.

That would seem to indicate that it's a PROCESS space out of memory error - so reducing the Java heap freed up more memory the windows process space to handle unzipping the ear file.

Regardless, the workaround seems to be to REDUCE the server's max heap size.

[glassfishrobot]

@glassfishrobot Commented File: dead.tiff Attached By: hexghost

[glassfishrobot]

@glassfishrobot Commented File: normal.tiff Attached By: hexghost

[glassfishrobot]

@glassfishrobot Commented This issue was imported from java.net JIRA GLASSFISH-4452

[github-actions[bot]]

This issue has been marked as inactive and old and will be closed in 7 days if there is no further activity. If you want the issue to remain open please add a comment