search

eclipse-openj9 / openj9

Eclipse OpenJ9: A Java Virtual Machine for OpenJDK that's optimized for small footprint, fast start-up, and high throughput. Builds on Eclipse OMR (https://github.com/eclipse/omr) and combines with the Extensions for OpenJDK for OpenJ9 repo.
Other
3.28k stars 720 forks source link

AttacherNotSupportedException thrown unexpectedly #13571

Open mateusbandeiraa opened 3 years ago

mateusbandeiraa commented 3 years ago

This issue was initially discussed at raphw/byte-buddy#1118. There are more details in the original issue.

It seems that com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.attachTargetImpl throws com.sun.tools.attach.AttachNotSupportedException when there's more than one instance of OpenJ9 processes running on Windows.

...
Caused by: com.sun.tools.attach.AttachNotSupportedException: target 10612 not found
    at com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.attachTargetImpl(OpenJ9VirtualMachine.java:150)
    at com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.lambda$attachTarget$1(OpenJ9VirtualMachine.java:128)
    at com.ibm.tools.attach.attacher.OpenJ9VirtualMachine$$Lambda$208/000000005C6D9C20.run(Unknown Source)
    at java.security.AccessController.doPrivileged(AccessController.java:734)
    at com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.attachTarget(OpenJ9VirtualMachine.java:130)
    at com.ibm.tools.attach.attacher.OpenJ9AttachProvider.attachVirtualMachine(OpenJ9AttachProvider.java:65)
    at com.ibm.tools.attach.attacher.OpenJ9AttachProvider.attachVirtualMachine(OpenJ9AttachProvider.java:48)
    at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
    ... 76 more

I discovered that if I run Eclipse with OpenJ9 as Administrator, other applications also running on OpenJ9, but not as administrator fail to call the attachTargetImpl method. As soon as I close Eclipse, the exact same application works as expected.

If all OpenJ9 processes are running as admin, or all processes are running as user, though, the error also dosn't arise. This only happens if there's an admin process running, and I try running an application as a regular user. In this case, the former would throw this exception.

I made some experiments:

Command prompt as user: Eclipse JVM Opened Eclipse as admin? Oracle 11 IBM J9 1.8 Eclipse OpenJ9 11
Oracle 11 Yes ✅ ✅ ✅
Oracle 11 No ✅ ✅ ✅
IBM J9 1.8 Yes ✅ ❌ ❌
IBM J9 1.8 No ✅ ✅ ✅
Eclipse OpenJ9 11 Yes ✅ ❌ ❌
Eclipse OpenJ9 11 No ✅ ✅ ✅
Command prompt as admin: Eclipse JVM Opened Eclipse as admin? Eclipse OpenJ9 11
IBM J9 1.8 Yes ✅
Eclipse OpenJ9 11 Yes ✅

Originally posted by @mateusbandeiraa in https://github.com/raphw/byte-buddy/issues/1118#issuecomment-926983877

JasonFengJ9 commented 3 years ago

Not able to reproduce the AttachNotSupportedException as per https://github.com/raphw/byte-buddy/issues/1118#issuecomment-926602347 @mateusbandeiraa any hints?

pshipton commented 3 years ago

Did you catch this part?

I discovered that if I run Eclipse with OpenJ9 as Administrator [on Windows], other applications also running on OpenJ9, but not as administrator

pshipton commented 3 years ago

What version of IBM J9 Java 1.8 is used to run Eclipse?

mateusbandeiraa commented 3 years ago

IBM J9 version:

java version "1.8.0_261"
Java(TM) SE Runtime Environment (build 8.0.6.15 - pwa6480sr6fp15ifix-20200820_01(SR6 FP15+IJ26443))
IBM J9 VM (build 2.9, JRE 1.8.0 Windows 10 amd64-64-Bit Compressed References 20200729_452626 (JIT enabled, AOT enabled)
OpenJ9   - 43296bc
OMR      - 08b0594
IBM      - 70917a2)
JCL - 20200720_01 based on Oracle jdk8u261-b13

EclipseJ9 version:

openjdk version "11.0.9" 2020-10-20
OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.9+11)
Eclipse OpenJ9 VM AdoptOpenJDK (build openj9-0.23.0, JRE 11 Windows 10 amd64-64-Bit Compressed References 20201022_795 (JIT enabled, AOT enabled)
OpenJ9   - 0394ef754
OMR      - 582366ae5
JCL      - 3b09cfd7e9 based on jdk-11.0.9+11)
JasonFengJ9 commented 3 years ago

I discovered that if I run Eclipse with OpenJ9 as Administrator [on Windows], other applications also running on OpenJ9, but not as administrator

Yea

openjdk version "11.0.12" 2021-07-20
IBM Semeru Runtime Open Edition 11.0.12.0 (build 11.0.12+7)
Eclipse OpenJ9 VM 11.0.12.0 (build openj9-0.27.0, JRE 11 Windows 10 amd64-64-Bit Compressed References 20210730_175 (JIT enabled, AOT enabled)
OpenJ9   - 1851b0074
OMR      - 9db1c870d
JCL      - 21849e2ca0 based on jdk-11.0.12+7)

@mateusbandeiraa you were running a 0.23 20201022_795 build, can you try a recent version [1]?

[1] https://github.com/ibmruntimes/semeru11-binaries/tags

mateusbandeiraa commented 3 years ago

I was able to reproduce the issue in openj9-0.27.0.

openjdk version "11.0.12" 2021-07-20
IBM Semeru Runtime Open Edition 11.0.12.0 (build 11.0.12+7)
Eclipse OpenJ9 VM 11.0.12.0 (build openj9-0.27.0, JRE 11 Windows 10 amd64-64-Bit Compressed References 20210730_175 (JIT enabled, AOT enabled)
OpenJ9   - 1851b0074
OMR      - 9db1c870d
JCL      - 21849e2ca0 based on jdk-11.0.12+7)

Here's a video of the steps I took:

https://user-images.githubusercontent.com/15495980/135171748-23d04086-b7be-4c13-903c-9c60db4222ed.mp4

Here's the project I used to reproduce the issue: https://github.com/mateusbandeiraa/ByteBuddyTest

Note: ByteBuddy actually throws a different exception (java.lang.IllegalStateException: Could not self-attach to current VM using external process), but I was able to verify by looking at the source code that it is caused by the AttacherNotSupportedException thrown by attachTargetImpl.

JasonFengJ9 commented 3 years ago

@mateusbandeiraa can you try another run after deleting the .com_ibm_tools_attach folder within %LOCALAPPDATA%\Temp [1]?

[1] https://www.eclipse.org/openj9/docs/attachapi/

mateusbandeiraa commented 3 years ago

I deleted the .com_ibm_tools_attach folder and it did not fix the issue.

By the way: the documentation on https://www.eclipse.org/openj9/docs/attachapi/ is horribly outdated. The C:\Documents and Settings\ directory doesn't exist since Windows XP. It was replaced in Windows Vista.

I was able to find the folder at %LOCALAPPDATA%\Temp\.com_ibm_tools_attach

(I can submit a PR to update this... Just didn't find where the documentation files are located in the repository)

mateusbandeiraa commented 3 years ago

I recorded another video, this time showing the temp files created by J9 in .com_ibm_tools_attach. It looks like that the user process is not creating the temp folders with its PID if there's already an admin process running.

https://user-images.githubusercontent.com/15495980/135275484-1e85e17a-e61d-4f80-bf0b-b13a88c8c012.mp4

JasonFengJ9 commented 3 years ago

Still not able to reproduce the failure, wondering if it is a timing issue. At mean time, @mateusbandeiraa could you collect some runtime logs? Run set OPENJ9_JAVA_OPTIONS="-Dcom.ibm.tools.attach.logging=yes" at each console, collect the .com_ibm_tools_attach folder and all newly created log files under current directory C:\Users\MateusTeixeiraBandei\eclipse-workspace\bytebuddytest (by default the process id with suffix log)

mateusbandeiraa commented 3 years ago

Here's the logs you requested. The user logs confirms that the JVM is not able to create the temp files it needs.

1632950696848(Wed Sep 29 18:24:56 BRT 2021) 21444: 25 [Attach API initializer]: AttachHandler IOException while creating files: semaphore _notifier status= -150
1632950696848(Wed Sep 29 18:24:56 BRT 2021) 21444: 25 [Attach API initializer]: AttachHandler terminate: Attach API is being shut down, currentAttachThread = Thread[Attach API initializer,5,main]
1632950696848(Wed Sep 29 18:24:56 BRT 2021) 21444: 25 [Attach API initializer]: AttachHandler terminate removing directory : C:\Users\MATEUS~1\AppData\Local\Temp\.com_ibm_tools_attach\21444

admin-19720.log admin-22532.log admin-26452.log

user-8664.log user-10340.log user-21444.log

.com_ibm_tools_attach.zip

Funny thing: for the first time in the past 3 days I wasn't able to replicate the issue (running as admin, then running as user did work). I rebooted my PC and the issue came back. Could this indicate that there's some random aspect to this issue (something like order of process creation or something like that)?

JasonFengJ9 commented 3 years ago

1632950696848(Wed Sep 29 18:24:56 BRT 2021) 21444: 25 [Attach API initializer]: AttachHandler IOException while creating files: semaphore _notifier status= -150

-150 is mapped to J9PORT_ERROR_SHSEM_OPFAILED https://github.com/eclipse-openj9/openj9/blob/6889aa118d0ff8d2df5067bebdae9f8d9eaa142e/runtime/oti/j9porterror.h#L69-L70 which indicates j9shsem_open() failure, however this error code could be returned in multiple exit points so trace point will be required to find exact failing routine.

for the first time in the past 3 days I wasn't able to replicate the issue (running as admin, then running as user did work). I rebooted my PC and the issue came back. Could this indicate that there's some random aspect to this issue (something like order of process creation or something like that)?

Could you provide Windows version & SP level? Which user group is current login user? Are you able to create a new local user, and verify the issue is still reproducible?

mateusbandeiraa commented 3 years ago

My main computer runs Windows 10 build 19043.1237. I was also able to replicate the issue on another machine running Windows 10 build 18363.1801.

Which user group is current login user

I'm not sure what you mean by that.

JasonFengJ9 commented 3 years ago

Is current login user an Administrator which belongs to Administrators group, or an account in Guests group, etc.? lusrmgr lists system users & groups.

mateusbandeiraa commented 3 years ago

All my tests were done in a user which belongs to the Administrators group.

JasonFengJ9 commented 3 years ago

Update:

Reproduced the error reported. The failure cause is because the less privileged user can't access the file created by privileged administrator account.

A workaround is for the less privileged user to create its own directory/files via setting -Dcom.ibm.tools.attach.directory=/user/own/folder.

fyi @mateusbandeiraa