gjdeval
commented
3 years ago A couple of preliminary comments:
-
The issue mentions using "open liberty 20.0.x" - depending on the value of 'x' that could be pretty old. It's always a good idea to use the latest version of Liberty, as enhancements and bug fixes are added continuously. As you are probably aware, Open Liberty docker images are available at https://hub.docker.com/_/open-liberty
-
The description includes a long list of JVM arguments that have been used. Those JVM args are in the form used by Oracle java, and may or may not be supported by IBM Java or OpenJ9. Documentation for OpenJ9 (generally also applicable to IBM Java), including the supported JVM args, is available at https://www.eclipse.org/openj9/docs/
Now to the particular issue raised - "high spikes in CPU server utilization". A few basic questions, to get some context
- Have you run this application in some other environment, e.g. in Open Liberty on a non-cloud server, with the same workload, and seen better performance, without CPU spikes?
- Have you examined the application logs to see if the server is functioning as expected, apart from CPU load, or perhaps has some errors or exceptions that may be increasing CPU usage?
- What is the baseline (non-spike) server CPU utilization, and how high are the spikes and how long do they last?
- Are the CPU spikes correlated with applied workload? If so, is the workload a controlled test workload, and how is it generated?
There are many possible causes of high cpu. For example, if the java heap becomes over-full, then garbage collection may become very active, and since GC is generally multi-threaded, this may cause high cpu. Among the JVM args listed is one that will cause a GC log to be generated. Have you examined that log to check the heap condition? If the GC log shows that the heap is very full, increasing the heap size may reduce the cpu spikes.
If the GC log does not clarify the cause of high cpu, then a more fundamental analysis of the system may be necessary. The WebSphere Performance Cookbook provides comprehensive guidance for investigating and resolving performance problems commonly encountered in application server deployments. https://publib.boulder.ibm.com/httpserv/cookbook/cookbook.html
If you are a member of an organization that has a support agreement with IBM for Open Liberty, you can request help by opening a trouble ticket. In that case, you will probably be asked to collect a set of data from your system while the problem is occurring. https://www.ibm.com/support/pages/node/1073532
I hope this information is helpful.
Regards, Gary
We are deploying docker image for open liberty 20.0.x on Azure cloud and on AKS cluster and seeing high usage of Application Server CPU during performance run.
We have tried with all below base images:
Also did added in server.xml some performance parameters by referring below link: https://www.ibm.com/docs/en/was-liberty/zos?topic=tuning-liberty
<connectionManager minPoolSize="20" maxPoolSize="300" maxIdleTime="90s" purgePolicy="FailingConnectionOnly" maxConnectionsPerThread="20" numConnectionsPerThreadLocal="10"/>
Tried also setting JAVA_OPTS:
JAVA_OPTS: “ -Xloggc:verbosegc.txt -XX:AdaptiveSizePolicyWeight=90 -XX:CICompilerCount=2 -XX:+ExitOnOutOfMemoryError -XX:GCTimeRatio=4 -XX:MaxHeapFreeRatio=20 -XX:MinHeapFreeRatio=10 -XX:ParallelGCThreads=8 -XX:+UseParallelOldGC -XX:CompressedClassSpaceSize=528482304 -XX:InitialHeapSize=2147483648 -XX:MaxHeapSize=5368709120 -XX:MaxMetaspaceSize=536870912 -XX:+PrintGC -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops ”
But still seeing high spikes in CPU server utilization. Can someone provide some help on this? What other approach we should try?