xnio high cpu usage jhipster 7.9.3

[aiasmartakis]

Overview of the issue

Generated jhipster gateway intermittently consumes 100% cpu until restarted. Suspected infinite loop in undertow + xnio. Unclear as to why it goes into the loop.

Motivation for or Use Case

Reproduce the error

The gateway service is run from a docker container in a k8s cluster and configured to authenticate against a keycloak server. The jhipster gateway will consume such high cpu it will potentially starve other microservices of resources. Steps:

1. generate jhipster gateway
2. setup keycloak (might not be needed)
3. perform high volumes of logins

Related issues

In a thread-dump we found XNIO threads (part of undertow) in an infinite loop.

"XNIO-1 I/O-1" #36 prio=5 os_prio=0 cpu=91062321.39ms elapsed=584143.95s tid=0x00007f4c5afa0000 nid=0x27 runnable  [0x00007f4c09df8000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.19/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.19/Unknown Source)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.19/Unknown Source)
    - locked <0x00000000fd8f5230> (a sun.nio.ch.Util$2)
    - locked <0x00000000fd8f51d8> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.19/Unknown Source)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:532)

"XNIO-1 I/O-2" #37 prio=5 os_prio=0 cpu=91111472.81ms elapsed=584143.95s tid=0x00007f4c5afa1000 nid=0x28 runnable  [0x00007f4c09cf7000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.19/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.19/Unknown Source)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.19/Unknown Source)
    - locked <0x00000000fd8f54b0> (a sun.nio.ch.Util$2)
    - locked <0x00000000fd8f5458> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.19/Unknown Source)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:532)

"XNIO-1 Accept" #38 prio=5 os_prio=0 cpu=51.92ms elapsed=584143.94s tid=0x00007f4c5afae000 nid=0x29 runnable  [0x00007f4c09bf6000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPoll.wait(java.base@11.0.19/Native Method)
    at sun.nio.ch.EPollSelectorImpl.doSelect(java.base@11.0.19/Unknown Source)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@11.0.19/Unknown Source)
    - locked <0x00000000fd8f5730> (a sun.nio.ch.Util$2)
    - locked <0x00000000fd8f56d8> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(java.base@11.0.19/Unknown Source)
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:532)

I've seen reports of other users having issues with xnio that might be related: https://stackoverflow.com/questions/44665552/undertow-xnio-i-o-thread-consistently-eat-cpu

Suggest a Fix

JHipster Version(s)

7.9.3

JHipster configuration

INFO! Using JHipster version installed locally in current project's node_modules INFO! Using JHipster version installed globally INFO! Executing jhipster:info INFO! Options: from-cli: true Welcome to the JHipster Information Sub-Generator

JHipster Version(s)

oauth2-gateway@0.0.1-SNAPSHOT /Users/user/Projects/dev/cloud/oauth2-gateway
└── (empty)

JHipster configuration, a .yo-rc.json file generated in the root folder

.yo-rc.json file

{
  "generator-jhipster": {
    "applicationType": "gateway",
    "authenticationType": "oauth2",
    "baseName": "oauth2gateway",
    "blueprints": [],
    "buildTool": "maven",
    "cacheProvider": "no",
    "clientFramework": "angularX",
    "clientPackageManager": "npm",
    "clientTheme": "none",
    "clientThemeVariant": "",
    "creationTimestamp": 1668183868569,
    "databaseType": "sql",
    "devDatabaseType": "mysql",
    "devServerPort": 4200,
    "dtoSuffix": "DTO",
    "enableGradleEnterprise": false,
    "enableHibernateCache": false,
    "enableSwaggerCodegen": false,
    "enableTranslation": true,
    "entitySuffix": "",
    "jhiPrefix": "jhi",
    "jhipsterVersion": "7.9.3",
    "languages": ["en"],
    "messageBroker": false,
    "microfrontend": false,
    "microfrontends": [],
    "nativeLanguage": "en",
    "otherModules": [],
    "packageName": "com.company.gateway",
    "pages": [],
    "prodDatabaseType": "mysql",
    "reactive": true,
    "searchEngine": false,
    "serverPort": "8081",
    "serverSideOptions": [],
    "serviceDiscoveryType": "eureka",
    "skipCheckLengthOfIdentifier": false,
    "skipClient": false,
    "skipFakeData": false,
    "skipUserManagement": true,
    "testFrameworks": [],
    "websocket": false,
    "withAdminUi": true
  }
}

JDL for the Entity configuration(s) entityName.json files generated in the .jhipster directory

JDL entity definitions

Environment and Tools

openjdk version "11.0.16.1" 2022-08-12 LTS OpenJDK Runtime Environment Microsoft-40648 (build 11.0.16.1+1-LTS) OpenJDK 64-Bit Server VM Microsoft-40648 (build 11.0.16.1+1-LTS, mixed mode)

git version 2.46.0

node: v8.17.0

npm: 6.13.4

yarn: 1.22.19

Docker version 27.2.0, build 3ab4256

INFO! Congratulations, JHipster execution is complete!

Browsers and Operating System

• [x ] Checking this box is mandatory (this is just to show you read everything)

[aiasmartakis]

we switched from undertow to tomcat to see if that removes the seemingly randomly occurring 100% cpu usage in undertow/xnio.

[mraible]

Can you reproduce with the latest version (8.7.0) or the main branch?

[mraible]

Closing due to lack of feedback. If you can reproduce this issue with v8.7.1, please reopen this issue.

[aiasmartakis]

FWIW: We've not had the issue since moving the gateway to Tomcat.