search

stumash / CoursePlanner

http://conucourseplanner.online
MIT License
5 stars 3 forks source link

Improve backend memory usage #116

Closed Davidster closed 6 years ago

Davidster commented 6 years ago

Summary

The problem

I've been noticing lately that our tomcat server has been shutting down unexpectedly due to an OutOfMemoryError. So I set up memory profiling with a program called yourkit and I noticed that every time a request is made to a servlet which does any db operations (a.k.a DBServlet), the memory usage of the application would increase by like 300kb and this memory would not get freed up over time/by garbage collection. I haven't been able to directly reproduce the OutOfMemoryError but I assume it is related to this issue.

The solution

I checked out the type of java objects that were making up the majority of this extra memory usage and it was mostly HashMaps used under the hood by MongoDB (see screenshot below). This led me to re-read the mongodb docs where I found that the normal usage is to create only one MongoClient object instance to be reused by all servlets in your application. This is different from what we're currently doing which is creating a unique MongoClient object for each servlet. I followed a guide on how to share a single reference to a MongoClient object accross all servlets and I made the necessary changes.

The version of java we have running on our server is 1.7 so I configured our maven build to use the same version. Also, to be able to use annotations in our backend code I needed to bump up the version of the javax.servlet library from 2.5 to 3.1.0

After this, I found that the problem of memory being used then not freed up on every db request went away. If we ever end up getting an OutOfMemoryError again, I've configured the JVM to do a memory dump when that occurs, in which case we can take a look at what's taking up the most memory in the application.

image

Java memory profiling

If you want to try out the memory profiling yourself, you can profile our server remotely by installing yourkit and clicking Connect to remote application then for the hostname put conucourseplanner.online.

Davidster commented 6 years ago

tbh I was playing around a lot with the java version on the server when trying to get memory profiling to work. We can probably upgrade to java 8 easily so we should probably do it at some point.

PeterGhimself commented 6 years ago

@Davidster according to this thread (from March 2017), 4.7% of all servers now use Java 7