Description:
High CPU when a single request hits the Gateway.
Followed troubleshooting documentation and findings are like this:
usage result:
%CPU CPU NI S TIME PID TID
...
78.1 - 0 R 00:17:27 167 356
77.9 - 0 R 00:17:25 167 357
...
dump result according to thread ids:
"HTTPS-Listener I/O dispatcher-1" #190 prio=5 os_prio=0 cpu=1043696.51ms elapsed=1336.89s tid=0x00007fdc54094000 nid=0x164 runnable [0x00007fdc1b606000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPoll.wait(java.base@11.0.8/Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.8/EPollSelectorImpl.java:120)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.8/SelectorImpl.java:124)
- locked <0x00000000eafef8d8> (a sun.nio.ch.Util$2)
- locked <0x00000000eafef778> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(java.base@11.0.8/SelectorImpl.java:136)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:586)
at java.lang.Thread.run(java.base@11.0.8/Thread.java:834)
"HTTPS-Listener I/O dispatcher-2" #191 prio=5 os_prio=0 cpu=1041420.55ms elapsed=1336.89s tid=0x00007fdc5409d000 nid=0x165 runnable [0x00007fdc1b505000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPoll.wait(java.base@11.0.8/Native Method)
at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.8/EPollSelectorImpl.java:120)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.8/SelectorImpl.java:124)
- locked <0x00000000eaff0950> (a sun.nio.ch.Util$2)
- locked <0x00000000eaff08f8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(java.base@11.0.8/SelectorImpl.java:136)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:256)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:586)
at java.lang.Thread.run(java.base@11.0.8/Thread.java:834)
This two threads consuming a lot of resources and I couldn't find a specific reason.
Steps to reproduce:
Deploy helm chart on AWS m5a.large worker nodes.
Affected Product Version:
API-M version 3.2.0
Environment details (with versions):
OS: AWS Linux
Env (Docker/K8s): K8s 1.18
Optional Fields
Related Issues:
Suggested Labels:
Suggested Assignees:
Description: High CPU when a single request hits the Gateway.
Followed troubleshooting documentation and findings are like this:
usage result: %CPU CPU NI S TIME PID TID ... 78.1 - 0 R 00:17:27 167 356 77.9 - 0 R 00:17:25 167 357 ...
dump result according to thread ids:
This two threads consuming a lot of resources and I couldn't find a specific reason.
Steps to reproduce: Deploy helm chart on AWS m5a.large worker nodes.
Affected Product Version: API-M version 3.2.0
Environment details (with versions): OS: AWS Linux Env (Docker/K8s): K8s 1.18 Optional Fields Related Issues: Suggested Labels: Suggested Assignees: