search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.25k stars 715 forks source link

Hanging java process in Debug Mode with newest jdk-17.0.6+10_openj9-0.36.0 #16841

Open c-koell opened 1 year ago

c-koell commented 1 year ago

Java -version output

openjdk version "17.0.6" 2023-01-17 IBM Semeru Runtime Open Edition 17.0.6.0 (build 17.0.6+10) Eclipse OpenJ9 VM 17.0.6.0 (build openj9-0.36.0, JRE 17 Windows 10 amd64-64-Bit Compressed References 20230117_367 (JIT enabled, AOT enabled) OpenJ9 - e68fb241f OMR - f491bbf6f JCL - 927b34f84c8 based on jdk-17.0.6+10)

Summary of problem

We normally starting in Eclipse a Maven Process like "liberty:dev" in Debug Mode. Since jdk-17.0.6+10_openj9-0.36.0 we are seeing very often that the maven process hangs. The funny thing is that it always hangs when the maven goal tries to install the Liberty Features. The last Output is always Installing features: [mpconfig-3.0, concurrent-2.0, webprofile-9.1, mail-2.0]

Diagnostic files

I have created a Dump File. What info do you need from jdmpview since the dump file is large. I can also upload the Dump file to a Cloud Storage if needed.

c-koell commented 1 year ago

You can find the dump under following cloud storage https://tbox.tirol.gv.at/index.php/s/sX5rKC83zjjLjfz with "D8k3LwKPzq" as password

pshipton commented 1 year ago

The core is not in a clean state, I don't believe it was taken while the JVM was paused. Can you use bin/jcmd <pid> Dump.system to get a core? Pls get two cores, 1 minutes apart so we can see what's changing. Also use bin/jcmd <pid> Dump.java request=exclusive+preempt to get a javacore file.

pshipton commented 1 year ago

I'll tentatively assign this to the JIT. @0xdaryl did anything change wrt debug mode in the 0.36 release, are there any options worth trying to see if the problem stops occurring?

c-koell commented 1 year ago

Hi @pshipton! I have tried withbin/jcmd <pid> Dump.system but jcmd is unable to connect to the vm (TimeOut). I have created the attached Dump with Process Explorer from the SysInternal Suite. (I'm on windows) Is there a other way to get a Dump ?

pshipton commented 1 year ago

I assume you are replacing <pid> with the correct pid of the java process? jcmd uses Attach API, which is running. jcmd needs to be run by the same user as the running process. I suppose the problem that is causing the hang could also be stopping Attach API from working.

An interesting thread I saw in the core is below. If you take multiple core files one minute apart, do you see this or similar Thread stuck in the same place? After running jdmpview -core <core> you can run info thread * to get a list of all the threads. Or perhaps an easier way is to run !threads to get a compact list, pick out the similar thread, get the thread id, which is 1748 in the following example, and then run info thread 1748. !stack 0x00ef0800 !j9vmthread 0x00ef0800 !j9thread 0x19fea7cd0e0 tid 0x6d4 (1748) // (Thread-7)

   properties:
    Affinity=0x000000000000003F                CreateTime=133226734167498308
    CreateTime_Formatted=2023/03/07 09:36:56   DumpError=0x00000000
    DumpFlags=0x00000000                       ExitStatus=0x00000103
    ExitTime=0                                 KernelTime=5156250
    StartAddress=0x00007FFE3BDE1B70            UserTime=137812500
    priority=0                                 priorityClass=32
   associated Java thread: 
    name:          Thread-7
    Thread object: java/lang/Thread @ 0x7037165c8
    Native info:   !j9vmthread 0xef0800  !stack 0xef0800
    Daemon:        false
    ID:            36 (0x24)
    Priority:      5
    Thread.State:  WAITING 
    JVMTI state:   ALIVE WAITING WAITING_INDEFINITELY IN_OBJECT_WAIT 
      waiting to be notified on: <monitor information not available>
    Java stack frames: 
     bp: 0x0000000000000000  method: reflect.Method java/lang/reflect/Method.copy()  source: Method.java:161
      objects: <no objects in this frame>
     bp: 0x0000000000000000  method: reflect.Method java/lang/reflect/ReflectAccess.copyMethod(reflect.Method)  source: ReflectAccess.java:102
      objects: <no objects in this frame>
     bp: 0x0000000000000000  method: reflect.Method jdk/internal/reflect/ReflectionFactory.copyMethod(reflect.Method)  source: ReflectionFactory.java:309
      objects: <no objects in this frame>
     bp: 0x0000000000d2a0a0  method: reflect.Method[] java/lang/Class.copyMethods(reflect.Method[])  source: Class.java:4384
      objects: 0x703722438 0x70346c0d8 0x7fe41e880 0x7fe41e880 0x7fe41d950 0x7fe41d950 0x70346d758 0x703832ab8 0x705557148 0x703832ab8 0x7fe41d950 0x703608b30 0x703832aa0 0x7033f0540 0x703832a88
     bp: 0x0000000000d2a128  method: reflect.Method[] java/lang/Class.lookupCachedMethods(Class$CacheKey)  source: Class.java:4405
      objects: 0x703457218 0x703608b30 0x703722438 0x7036e13a0
     bp: 0x0000000000d2a1d0  method: reflect.Method[] java/lang/Class.getMethods()  source: Class.java:1763
      objects: 0x703608b30
     bp: 0x0000000000d2a420  method: com.ibm.ws.repository.transport.client.DataModelSerializer$ClassAndMethod com/ibm/ws/repository/transport/client/DataModelSerializer.internalGetClassForFieldName(String, Class, boolean)  source: DataModelSerializer.java:429
      objects: 0x703608b30 0x7086d7a50 0x7fe41d908 0x7fe41d800
     bp: 0x0000000000000000  method: com.ibm.ws.repository.transport.client.DataModelSerializer$ClassAndMethod com/ibm/ws/repository/transport/client/DataModelSerializer.getClassForFieldName(String, Class)  source: DataModelSerializer.java:398
      objects: <no objects in this frame>
     bp: 0x0000000000d2a760  method: Object com/ibm/ws/repository/transport/client/DataModelSerializer.processJsonObjectBackIntoDataModelInstance(javax.json.JsonObject, Class, com.ibm.ws.repository.transport.client.DataModelSerializer$Verification)  source: DataModelSerializer.java:578
      objects: 0x7036e1150 0x703608b30 0x7080a3e98 0x7fe41d7f0 0x7fe41d7f0 0x7fe41d7f0 0x7fe407a80 0x7034f44d8 0x703608b30 0x703608b30 0x7fe409518 0x7fe409518 0x70366bbd0 0x7fe405cb8 0x7fe407a80 0x7fe4104e8 0x7fe40d150 0x7fe40d128 0x7086d95b0 0x7086d7a68 0x7086d7a50 0x7fe41d7f0 0x7086d7a50 0x7fe405fd0 0x7fe405cb8 0x7eecba8c0 0x703608b30
     bp: 0x0000000000d2aaa0  method: Object com/ibm/ws/repository/transport/client/DataModelSerializer.processJsonObjectBackIntoDataModelInstance(javax.json.JsonObject, Class, com.ibm.ws.repository.transport.client.DataModelSerializer$Verification)  source: DataModelSerializer.java:548
      objects: 0x7036e1150 0x7034f4250 0x70808ed98 0x7fe400e80 0x7fe400e80 0x7fe400e80 0x7fe405ad8 0x7fe3fd7f8 0x7034f44d8 0x7fe3fe520 0x7fe3feb90 0x7fe3feb90 0x7078a02b0 0x7034f44d8 0x7fe3fe200 0x7078a02b0 0x7034f44d8 0x7fe3fdc00 0x7fe3ff500 0x7034f44d8 0x7034f4250 0x7034f4250 0x7fe400e68 0x7fe400e68 0x70366bbd0 0x7fe3eca00 0x7fe3ff500 0x7fe3fea08 0x7fe3fe200 0x7fe3fd980 0x7fe405ad8 0x7fe405ab0 0x7fe402828 0x7080a3e98 0x7080a3e80 0x7fe400e80 0x70808ed98 0x7fe3eca68 0x7fe3eca00 0x7eecb7ce0 0x7fe405ab0
     bp: 0x0000000000000000  method: Object com/ibm/ws/repository/transport/client/DataModelSerializer.deserializeObject(javax.json.JsonObject, Class, com.ibm.ws.repository.transport.client.DataModelSerializer$Verification)  source: DataModelSerializer.java:761
      objects: <no objects in this frame>
     bp: 0x0000000000d2ac10  method: java.util.Collection com/ibm/ws/repository/transport/client/SingleFileClient.getAllAssets()  source: SingleFileClient.java:134
      objects: 0x70785e370 0x70785e398 0x70808d9d0 0x7fe36e3b8 0x7fe354a40 0x70808d9b8 0x7eeba5ae8 0x7eeba5b08 0x7fe354a40 0x7036e1150 0x7034f4250 0x70808ed98 0x70808d9b8
     bp: 0x0000000000d2ad08  method: java.util.Collection com/ibm/ws/repository/transport/client/AbstractRepositoryClient.getFilteredAssets(java.util.Map)  source: AbstractRepositoryClient.java:105
      objects: 0x7eeba5b20 0x70785e370
     bp: 0x0000000000d2ad70  method: java.util.Collection com/ibm/ws/repository/transport/client/AbstractRepositoryClient.getFilteredAssets(java.util.Collection, java.util.Collection, com.ibm.ws.repository.common.enums.Visibility, java.util.Collection, boolean)  source: AbstractRepositoryClient.java:181
      objects: 0x70785e370 0x70785f3c0 0x70785f3d0 0x7eeba5b20
     bp: 0x0000000000d2ada8  method: java.util.Collection com/ibm/ws/repository/transport/client/AbstractRepositoryClient.getAssetsWithUnboundedMaxVersion(java.util.Collection, java.util.Collection, com.ibm.ws.repository.common.enums.Visibility)  source: AbstractRepositoryClient.java:72
      objects: 0x70785e370 0x70785f3c0 0x70785f3d0
     bp: 0x0000000000d2ae30  method: java.util.Map com/ibm/ws/repository/connections/internal/AbstractRepositoryConnection.getResources(java.util.Collection, java.util.Collection, com.ibm.ws.repository.common.enums.Visibility)  source: AbstractRepositoryConnection.java:166
      objects: 0x703722748 0x703722758 0x70785f3c0 0x70785f3d0 0x70785f3e0 0x70785f3f0 0x70785e370 0x7eeba5b48 0x7eeba5b48
     bp: 0x0000000000d2aeb0  method: java.util.Map com/ibm/ws/repository/connections/RepositoryConnectionList.getResources(java.util.Collection, java.util.Collection, com.ibm.ws.repository.common.enums.Visibility)  source: RepositoryConnectionList.java:541
      objects: 0x703722788 0x703722758 0x70785f3c0
     bp: 0x0000000000d2af00  method: void com/ibm/ws/repository/resolver/RepositoryResolver.fetchFromRepository(java.util.Collection)  source: RepositoryResolver.java:208
      objects: 0x70785f400 0x703722758 0x70785f3c0
     bp: 0x0000000000d2af40  method: void com/ibm/ws/repository/resolver/RepositoryResolver.<init>(java.util.Collection, java.util.Collection, java.util.Collection, com.ibm.ws.repository.connections.RepositoryConnectionList)  source: RepositoryResolver.java:168
      objects: 0x70785f400 0x703722758 0x7037227f0 0x70346d690 0x703722788
     bp: 0x0000000000d2b030  method: java.util.Collection com/ibm/ws/install/internal/InstallKernelMap.singleFileResolve()  source: InstallKernelMap.java:931
      objects: 0x703722800 0x703722758 0x70785f450 0x7037227a0 0x703722858 0x703722788 0x703722870 0x703722888 0x7037227f0 0x703722898 0x7037228b0 0x70785f450 0x70785f400
     bp: 0x0000000000d2b060  method: Object com/ibm/ws/install/internal/InstallKernelMap.get(Object)  source: InstallKernelMap.java:273
      objects: 0x703722800 0x7063e0bf0 0x70346d268
     bp: 0x0000000000d2b088  method: Object com/ibm/ws/install/map/InstallMap.get(Object)  source: InstallMap.java:275
      objects: 0x7037228d8 0x7063e0bf0
     bp: 0x0000000000d2b168  method: void io/openliberty/tools/common/plugins/util/InstallFeatureUtil.installFeatures(boolean, java.util.List)  source: InstallFeatureUtil.java:593
      objects: 0x7037228f8 0x703722938 0x703722950 0x703722898 0x703722870 0x703722978 0x7037228d8 0x703719fc8
     bp: 0x0000000000d2b1b8  method: void io/openliberty/tools/maven/server/InstallFeatureMojo.installFeatures()  source: InstallFeatureMojo.java:99
      objects: 0x7037205e8 0x7037229b8 0x7037229d0 0x7037229e8 0x7037229f8 0x7037228f8 0x703722a10
     bp: 0x0000000000d2b1d8  method: void io/openliberty/tools/maven/server/InstallFeatureMojo.doExecute()  source: InstallFeatureMojo.java:70
      objects: 0x7037205e8
     bp: 0x0000000000d2b210  method: void org/codehaus/mojo/pluginsupport/MojoSupport.execute()  source: MojoSupport.java:122
      objects: 0x7037205e8
     bp: 0x0000000000d2b298  method: void org/apache/maven/plugin/DefaultBuildPluginManager.executeMojo(org.apache.maven.execution.MavenSession, org.apache.maven.plugin.MojoExecution)  source: DefaultBuildPluginManager.java:137
      objects: 0x703bbc9d8 0x703bc2038 0x703722a20 0x703c371a8 0x703722a48 0x7037205e8 0x7048fdd50 0x7048fdd50 0x703bc2038 0x703722af0
     bp: 0x0000000000d2b300  method: void org/twdata/maven/mojoexecutor/MojoExecutor.executeMojo(org.apache.maven.model.Plugin, String, org.codehaus.plexus.util.xml.Xpp3Dom, org.twdata.maven.mojoexecutor.MojoExecutor$ExecutionEnvironment)  source: MojoExecutor.java:120
      objects: 0x703b419d8 0x706219918 0x703722b10 0x703722b30 0x703bc2038 0x703c371a8 0x7036e60f0 0x703722a48 0x703722a20
     bp: 0x0000000000d2b330  method: void io/openliberty/tools/maven/server/StartDebugMojoSupport.runLibertyMojo(String, org.codehaus.plexus.util.xml.Xpp3Dom)  source: StartDebugMojoSupport.java:363
      objects: 0x704b37910 0x706219918 0x703722b10
     bp: 0x0000000000d2b378  method: void io/openliberty/tools/maven/server/StartDebugMojoSupport.runLibertyMojoInstallFeature(org.twdata.maven.mojoexecutor.MojoExecutor$Element, java.io.File, String)  source: StartDebugMojoSupport.java:348
      objects: 0x704b37910 0x703722b10
     bp: 0x0000000000d2b3b0  method: void io/openliberty/tools/maven/server/DevMojo.runLibertyMojoInstallFeature(org.twdata.maven.mojoexecutor.MojoExecutor$Element, java.io.File, String)  source: DevMojo.java:1911
      objects: 0x704b37910
     bp: 0x0000000000d2b3d8  method: void io/openliberty/tools/maven/server/DevMojo$DevMojoUtil.libertyInstallFeature()  source: DevMojo.java:406
      objects: 0x7036de538
     bp: 0x0000000000d2b418  method: void io/openliberty/tools/common/plugins/util/DevUtil.restartServer(boolean)  source: DevUtil.java:1853
      objects: 0x7036de538
     bp: 0x0000000000d2b480  method: void io/openliberty/tools/common/plugins/util/DevUtil$HotkeyReader.readInput()  source: DevUtil.java:2606
      objects: 0x703722b48 0x703722b60 0x703722b70 0x703722b80 0x703722b90 0x703722ba0 0x703722bb0 0x703722bc0
     bp: 0x0000000000d2b4a8  method: void io/openliberty/tools/common/plugins/util/DevUtil$HotkeyReader.run()  source: DevUtil.java:2573
      objects: 0x703722b48
     bp: 0x0000000000d2b4c8  method: void java/lang/Thread.run()  source: Thread.java:857
      objects: 0x7037165c8
pshipton commented 1 year ago

Another possibility for debugging, does the problem also happen on non-Windows platforms? It should be easier to get a proper core file on non-Windows.

c-koell commented 1 year ago

Yes i have replaced the pid :-) and also the user is the same .. so i assume the attach api ist not working, as you sayed. I have only a windows client to reproduce the problem .. sorry

I have taken 3 Dumps with Process Explorer and i see always the same StackTrace as you described. I have attached the dumps to the cloud storage

cloud storage https://tbox.tirol.gv.at/index.php/s/sX5rKC83zjjLjfz with "D8k3LwKPzq" as password

c-koell commented 1 year ago

@pshipton do you have found some time to look at the dumps ?

pshipton commented 1 year ago

We're waiting for someone to have time to look at it. I'm told it won't likely be resolved in the 0.38 time frame but be deferred to 0.40.

0xdaryl commented 1 year ago

This should move to 0.40. It won't be resolved for 0.38.

hzongaro commented 8 months ago

Moving this to the backlog for now, rather than moving it from release to release.