search

Esri / geoportal-server

Geoportal Server is a standards-based, open source product that enables discovery and use of geospatial resources including data and services.
https://gptogc.esri.com/geoportal
Apache License 2.0
244 stars 149 forks source link

java.lang.OutOfMemoryError: PermGen space and GC #185

Closed rmbradley closed 8 years ago

rmbradley commented 9 years ago

Hi,

We have a production instance of Geoportal 1.2.5 that we've noticed getting the following errors in the gpt logs:

Sep 14, 2015 1:59:05 PM org.apache.catalina.core.ApplicationDispatcher invoke SEVERE: Servlet.service() for servlet jsp threw exception java.lang.OutOfMemoryError: PermGen space

and

Sep 14, 2015 12:42:24 PM com.esri.gpt.control.georss.RestQueryServlet execute SEVERE: Error parsing request. java.lang.OutOfMemoryError: GC overhead limit exceeded

Our instance runs on Tomcat 7 (7.0.54) on a Windows 2008 R2 virtual machine. During the time it was heavily CPU-bound these errors occurred. However the RAM was about 50% unsued. The Tomcat process was using about 454MB of memory for its working set (usually about 270MB).

Do we need to increase the JVM memory parameters when the instance starts? If so, how do we do that (I'm not a Tomcat server pro at that point)?

Thanks,

Rick

gcampanile commented 9 years ago

Hi Rick,

we use these parameters and in most cases they solve the PermGen space: OPen Tomcat monitor properties and go to Java settings tab: Initial Memory pool : 1024 Maximum memory pool: 1024 Thread stack size: 256

In the Java Options text box add: -XX:MaxPermSize=1024m See image attached:

image

Hope this solves the problem; these parameters instruct Tomcat to make a more extensive use of the memory, so monitor memory usage to see if they are appropriate for your system.

Gianni

rmbradley commented 8 years ago

Hi Gianni,

Our problem with out of memory errors seems to have subsided, but we are keeping an eye on the logs. It could have been caused by something external to the application. If it happens again we will likely increase the parameters you mentioned above. Thanks!

mhogeweg commented 8 years ago

while not the only cause, understanding what other machines run on the VM host could be helpful. We have seen VM hosts being over allocated in terms of RAM assigned to VM's compared to the total physical RAM (not specific to Geoportal Server, but also in ArcGIS Server deployments). The VM may think it only uses half of its allocated RAM, while the host has exhausted its physical RAM.

Another situation where I've seen this is when I update gpt.properties frequently during development. My tomcat reloads the app automatically and specifically with this file, it appears to get into trouble after repeated updates and reloads. Restarting the tomcat server resolves the issue in those cases.

rmbradley commented 8 years ago

Thanks Marten for your input on this and thanks to Gianni as well. We left this episode alone for a while just to observe. The VM has 2CPU (Intel Xeon X5670 @2.93GHz) 6GB RAM. While we thought about adjusting the JVM options above, we think this event might be a self-inflicted wound. By modifying certain config files in ..\WEB-INF\classes\gpt... on the fly seems to greatly aggravate the problem. As a novice Java/Geoportal administrator, I learned this the hard way. Since modifying config files when Tomcat is stopped, the problem doesn't appear upon restart. Not sure what is going on under the hood, but I can imagine it's not pretty. Symptoms before the "on-the-fly" modifications are increased working set for the JVM "container" and thus resulting in the JVM limits.

Thanks,

Rick

On Sat, Oct 31, 2015 at 10:41 AM, Marten notifications@github.com wrote:

while not the only cause, understanding what other machines run on the VM host could be helpful. We have seen VM hosts being over allocated in terms of RAM assigned to VM's compared to the total physical RAM (not specific to Geoportal Server, but also in ArcGIS Server deployments). The VM may think it only uses half of its allocated RAM, while the host has exhausted its physical RAM.

Another situation where I've seen this is when I update gpt.properties frequently during development. My tomcat reloads the app automatically and specifically with this file, it appears to get into trouble after repeated updates and reloads. Restarting the tomcat server resolves the issue in those cases.

— Reply to this email directly or view it on GitHub https://github.com/Esri/geoportal-server/issues/185#issuecomment-152747726 .

Richard M. Bradley Contractor - Northrop Grumman Technical Services Geospatial Engineer ArcGIS Server Administrator U.S. Dept. of the Interior/Bureau of Land Management