Problem typing uppercase letters with java.awt.Robot. (Temurin 19)

BlakeTNC commented 1 year ago

Please provide a brief summary of the bug

Summary java.awt.Robot is erroneously typing lowercase letters instead of capital letters on MacOS Ventura under certain conditions described herein. I wrote an example java program to isolate the problem. This issue occurs when this example program is run on macOS with apple silicon using the temurin-17 JDK, but not when run using temurin-8. The problem only occurs when the user is simultaneously moving their mouse pointer while the Robot class is typing text. If the mouse pointer is in motion, the example program will type 100% lowercase letters (for me). If the mouse pointer is holding perfectly still, the program will produce 100% uppercase letters. I know this description sounds somewhat strange. This took a while to troubleshoot and isolate.

Impact for us I'm a developer of an on-screen keyboard accessibility application. For us, this behavior breaks the usability of the on screen keyboard program and the robot class under Temurin 17. There is no known workaround for our use case except reverting to Java 8.

Please provide steps to reproduce where possible

Environment: Here is the environment I was using. I have not tried it on other platforms. A MacOS laptop computer, updated to MacOS Ventura, with an M1 chip (apple silicon). Running NetBeans 12.2, with two JDKs installed, Temurin 17 and Temurin 8.

Steps to reproduce:

Open TextEdit. Run the example program (in Netbeans or another IDE), using Temurin JDK 17. After you start the program, you have seven seconds to put the text cursor into TextEdit. Then the program will type characters for about 15 seconds, and then it will exit. You can repeat running the program as needed while testing. Move the mouse while the typing occurs. (Don't touch the keyboard or the mouse buttons, just move the mouse around. I was using the built in trackpad.). While the mouse is moving, the letters will be erroneously lowercase. While the mouse is holding still, uppercase.
Repeat these steps while running the program in Temurin JDK 8, and the problem will no longer exist.

Things that did not help. (For me.) None of these items helped to solve or improve the issue.

Setting robot.setAutoWaitForIdle() to true or false.
Adjusting or removing robot.setAutoDelay();
Using thread.sleep() or robot.delay() to insert arbitrarily high delays between the keystrokes of the key combination.
Placing the entire key combination into a synchronized (atomic) function.

=== Example Program ===

package somepackage;

import java.awt.Robot;
import java.awt.AWTException;
import java.awt.Toolkit;
import java.awt.datatransfer.Clipboard;
import java.awt.datatransfer.StringSelection;
import java.awt.event.KeyEvent;

public class Start {

    private static Robot robot;

    public static void main(String[] arguments)
            throws AWTException, InterruptedException {

        // Create a robot instance, with an autodelay between keystrokes.
        robot = new Robot();
        robot.setAutoWaitForIdle(true);
        robot.setAutoDelay(100);

        // Wait 7 second for the user to open a text editor, to catch the text.
        Thread.sleep(7000);

        // Paste the java version. (optional)
        String javaVersion = System.getProperty("java.vendor") + " "
                + System.getProperty("java.version") + " ";
        StringSelection stringSelection = new StringSelection(javaVersion);
        Clipboard clipboard = Toolkit.getDefaultToolkit().getSystemClipboard();
        clipboard.setContents(stringSelection, stringSelection);
        robot.keyPress(KeyEvent.VK_META);
        robot.keyPress(KeyEvent.VK_V);
        robot.keyRelease(KeyEvent.VK_V);
        robot.keyRelease(KeyEvent.VK_META);

        // Run the typing loop.
        for (int i = 0; i < 10; ++i) {

            // Press shift-a.
            robot.keyPress(KeyEvent.VK_SHIFT);
            robot.keyPress(KeyEvent.VK_A);
            robot.keyRelease(KeyEvent.VK_A);
            robot.keyRelease(KeyEvent.VK_SHIFT);

            // Wait 1 second between loops.
            Thread.sleep(1000);
        }

        // Press enter.
        robot.keyPress(KeyEvent.VK_ENTER);
        robot.keyRelease(KeyEvent.VK_ENTER);

        // Exit the program.
    }
}

BlakeTNC commented 1 year ago

I ran the test with several different JDK vendors and versions. The vendors were Amazon Corretto, Eclipse Temurin, and Azul Systems Zulu. The JDK versions were Java 8, 11, 17, and 19. The problem does not exist in all vendors in Java 8. The problem does exist for all vendors for all Java versions 11 through 19. Below are the test results.

Java 1.8 Amazon.com Inc. 1.8.0_362 AAAAAAA (GOOD) Temurin 1.8.0_362 AAAAAAA (GOOD) Azul Systems, Inc. 1.8.0_362 AAAAAAA (GOOD)

Java 11 Amazon.com Inc. 11.0.18 AaaaaAA (BUGGED) Eclipse Adoptium 11.0.18 AaaaAAA (BUGGED) Azul Systems, Inc. 11.0.18 AaaAAAA (BUGGED)

Java 17 Amazon.com Inc. 17.0.6 AaaaaAA (BUGGED) Eclipse Adoptium 17.0.6 AaaAaAA (BUGGED) Azul Systems, Inc. 17.0.6 AaAaAaA (BUGGED)

Java 19 Amazon.com Inc. 19.0.2 AaaaAAa (BUGGED) Eclipse Adoptium 19.0.2 AaaAaAA (BUGGED) Azul Systems, Inc. 19.0.2 aAAaAaA (BUGGED)

karianna commented 1 year ago

It likely works for Java 8 as there is no Apple Silicon port for Java and so you're running through Rosetta there. I suspect there may be an issue with the Apple Silicon port. Checking the OpenJDK bug DB now...

karianna commented 1 year ago

https://bugs.openjdk.org/browse/JDK-8302618

BlakeTNC commented 1 year ago

The Temurin Java 8 build was x86. The other java 8 builds were described as aarch64. (example: amazon-corretto-8-aarch64-macos-jdk.pkg) Additional info, I just tested this bug on a mac x86 computer, and the bug does exist on both cpu types.

BlakeTNC commented 1 year ago

By the way, how do I (or can I) create an account at https://bugs.openjdk.org to comment on the bug there? I don't see a signup form.

karianna commented 1 year ago

By the way, how do I (or can I) create an account at https://bugs.openjdk.org to comment on the bug there? I don't see a signup form.

You have to be an OpenJDK author (yes we know this is painful, but OpenJDK themselves simply got flooded with too many low quality bug reports). I was very happy to add your one on your behalf as you gave great, reproducible info!

BlakeTNC commented 1 year ago

Thank you for posting it there. If I have some particularly useful info later, I'll just comment here. I might suggest removing "for Apple M1" from your title since I also saw the bug on a mac x86 just now. So you were able to reproduce?

karianna commented 1 year ago

Hmm, actually I can't repo on my App M2 - I get capital letters

karianna commented 1 year ago

Oh hang on, let me try again

karianna commented 1 year ago

Yeah I can repro, whenever you move the mouse pointer you get lower case. Interesting!

BlakeTNC commented 1 year ago

Yeah I thought so too, interesting and odd both. Thanks!

BlakeTNC commented 1 year ago

Would you mind adding this comment over at openjdk?: The issue is not specific to the shift key, it also affects other key combos. For example the Mac paste command also fails when the mouse is moving. (Command-V or Meta-V)

github-actions[bot] commented 1 year ago

We are marking this issue as stale because it has not been updated for a while. This is just a way to keep the support issues queue manageable. It will be closed soon unless the stale label is removed by a committer, or a new comment is made.

karianna commented 1 year ago

Fix is currently set for Java 22

honkar-jdk commented 1 year ago

@BlakeTNC @karianna

Can you please explain the use case involving the on-screen keyboard and the robot in detail as it will help to determine whether it is a JDK or Apple related issue. Are you making the Robot to move to specific key (using robot.mouseMove) on the on-screen keyboard and then use robot.keyPress to generate the letters ?

BlakeTNC commented 1 year ago

@honkar-jdk A concise example program is provided above, with the original issue report. The example includes some comments on how to replicate the issue. The function robot.mouseMove is never called in the example program. (The mouse pointer should be moved by hand while the program is running to replicate the error.)

The mentioned on screen keyboard use case is located at the following link. It is called TouchBoard. But FYI, the TouchBoard software is large: https://github.com/TouchBoardSoftware/TouchBoard

honkar-jdk commented 1 year ago

@BlakeTNC Thank you for the details.

Please let me know if I understood the original TouchBoard use case correctly. TouchBoard is hover-based which requires the user to move and hover over the character to be typed, when the mouse pointer is over a key, robot is used to simulate a keypress/release, but with capital letters when there is manual mouse movement by user (to move to next key or to hover) , it causes the letters to be typed in smaller case just like in the example program.

BlakeTNC commented 1 year ago

@honkar-jdk That is mostly correct. I'll edit your paragraph for some extra clarity.

TouchBoard is is a hover-based user input application which requires the user to move the mouse and hover over the desired on screen "Key" to be activated. When the mouse pointer is over a Key, any number of user customizable actions can occur, ranging from simply typing one or more characters of text, up to a variety of more complex scripted behaviors. One possible Key action is that robot is used to simulate any number of consecutive key presses/releases, but with capital letters when there is simultaneous manual mouse movement by user (for any reason), it causes all letters to be typed in lower case just like in the example program. Specifically, it appears that the mouse movement disables the correct functioning of the key combination between the virtual key codes for the Shift key and the applicable letter key. Any mouse movement also causes -any- other type of key combination to fail. Other examples of known failing key combinations are copy and paste operations (Like Control-c Control-v, or Meta-c Meta-v on mac).

honkar-jdk commented 1 year ago

@BlakeTNC Thank you for adding in the details. Will let you know when we have an update on this issue.

BlakeTNC commented 5 months ago

I posted an update yesterday about this bug on the open JDK pull request where what some work was being done on this issue. https://github.com/openjdk/jdk/pull/14744

A possible fix and some unit tests were developed for the bug in the openJDK, in that pull request. However, the pull request was never merged with the openJDK.

I believe at least one developer may have thought that Apple fixed the bug in macOS, but according to my testing yesterday, this bug still exists on the latest version of macOS Sonoma 14.5. I also read that Apple never replied to any issue reports that were filed.

Some additional details are in my comment on the pull request thread, linked above. thank you.

adoptium / adoptium-support

Problem typing uppercase letters with java.awt.Robot. (Temurin 19) #710

Please provide a brief summary of the bug

Please provide steps to reproduce where possible