search

assertj / assertj-swing

Fluent assertions for Swing apps
Other
107 stars 52 forks source link

Failure to move the mouse to the correct spot - Windows 10 #235

Closed smccants closed 5 years ago

smccants commented 5 years ago

First off, I looked for a mailing list or news group to post this in, but didn't find anything. So apologies if this is posted in the wrong place.

We are using the Robot in AssertJ Swing in a Demo version of our Java Application to perform some automated configuration changes to our application. This is much better and easier than teaching our salesman to do these things correctly.

The code is working for us on several Linux machines, a Mac and one Windows 10 machine. However, it is not working on our salesman's Windows 10 machine. His machine has Java 1.8 and we have run successfully under that version of Java on the other machines.

I've spent some time debugging the problem and haven't been able to resolve it. Here is the code that uses the Robot:

    FrameFixture fullScreenFrame = robot.getFullScreenFrameFixture();
    // Go to the configuration screen
    robot.runCommand("MENU config");
    robot.waitForIdle();

    // Select Buttons tab
    JTabbedPaneFixture tabbedPane = fullScreenFrame.tabbedPane();
    LOG.log(Level.WARNING, "Found the Tab Pane "+tabbedPane);
    tabbedPane.selectTab("Buttons");
    LOG.log(Level.WARNING, "Found the Buttons Tab");
    robot.waitForIdle();
    // Select the "or" button group
    robot.clickOnTable(fullScreenFrame, "buttons", "buttongroups", "Name", "or");
    robot.waitForIdle();

Here is the method that gets the fixture:

public FrameFixture getFullScreenFrameFixture() {
      return new FrameFixture(ROBOT, DataSource.getInstance().getFullScreenFrame());
}

DataSource.getInstance().getFullScreenFrame() returns the top level JFrame. The ROBOT variable is:

public static final Robot ROBOT = BasicRobot.robotWithCurrentAwtHierarchy();

The method robot.runCommand(...) doesn't even use the robot and looks like this:

public void runCommand(final String command) {
   try {
      SwingUtilities.invokeAndWait(new Runnable() {
          @Override
          public void run() {
              DataSource.getInstance().getFullScreenFrame().handleCommand(command);
          }
      });
  } catch (InterruptedException ex) {
      Logger.getLogger("com.hcs.orc.demo.DemoRobot").log(Level.SEVERE, "Bad news - Robot failed to run the command "+command, ex);
  } catch (InvocationTargetException ex) {
      Logger.getLogger("com.hcs.orc.demo.DemoRobot").log(Level.SEVERE, "Bad news - Robot failed to run the command "+command, ex);
  }
  waitForIdle();
}

This code is working correctly (and ties into our larger application) and brings up the configuration screen for our application.

Here is the wait for idle method:

public void waitForIdle() {
   ROBOT.waitForIdle();
}

The clickOnTable() method fails on the Windows 10 machine, because the the tabbedPane.selectTab("Buttons") method didn't actually click on the tab, so the table we are looking for isn't present on the screen.

Oddly, the selectTab() method thinks it did the right thing and so doesn't throw any errors. However, when I watch it, the mouse goes to the top of the screen above the tab and so clicks (if it clicks at all - I'm not sure) on the application title bar. Very strange, especially since it works on all the other systems (including Windows 10) that we've tried.

Here are some things I've looked at as possible problems on the failing machine and come up empty:

So, I'm looking for what I might have missed that is interfering with AssertJ Swing. Any suggestions or ideas?

croesch commented 5 years ago

Hello @smccants

Thanks for opening this issue, it is surely the correct place - a mailing list is missing - but sometimes it takes some months that you receive an answer here...

I'm pretty clueless what the problem could be, you could try calling selectTab(int) it uses slightly different code than selectTab(String). You could try to receive the location of the point where the mouse should be clicked and use Robot's click methods directly to see if it makes any difference.

You mentioned that the code runs fine on one Windows 10 machine? Are the properties you wrote above the same for the correct machine?

smccants commented 5 years ago

Hello @croesch Thank you for your reply. We've tried in on a variety of systems (Linux, Mac, Windows 10) and found two Lenovo laptops with identical hardware and similar configurations that it fails on.

We tried the selecting the tab manually, but it just fails on the next step. It seems anything that involves a mouse positioning is wrong on these two laptops.

I've posted some more details here: https://stackoverflow.com/questions/56859893/assertj-swing-not-working-on-one-windows-10-machine

These aren't development laptops and so we haven't loaded our development environment and tried running through the debugger to see where the coordinates for the mouse are getting munged.

Otherwise we are out of ideas.

croesch commented 5 years ago

Are you able to try different JDK versions? Try an old and a new JDK 8 and a recent JDK 11/12..

I'd suggest that you add lots of log output to your application, to the test and to AssertJ Swing (which you could build for that purpose) and see where it starts getting wrong without having to deploy a debugger..

leeangh commented 5 years ago

@smccants Have you found a solution for this?

I see a similar issue on my environment, but the automation scripts runs correctly on another computer with windows 10. Only fails on my laptop, my test case fails to click button but clicked at a unexpected point.

--- Updates I just changed my laptop's resolution from 1920x1080 to 1366x768, restart the jsystem then it works as expected. It looks like this only occurs at a high-dpi computer, but I will try java 11.

smccants commented 5 years ago

@leeangh Interesting workaround you found. Unfortunately, I don't have access to the two (identical) failing laptops as they belong to colleagues who are frequently on the road. They are 1080p, but maybe only 13" screens. The other Windows 10 box we've tried is probably not considered high-dpi as it is running on a 65" TV at 1080p. We've had no problems on Linux, even 4K systems. Maybe a Windows JDK/JRE bug for high-dpi screens?

smccants commented 5 years ago

@leeangh I can confirm the same behavior on one of our laptops that is a 14" Lenovo X1 Carbon. Fails if the resolution is 1920x1080, but works if the resolution is 1366x768. This is under Windows 10. Looks like a hi-dpi problem only on Windows. The laptop is using the latest Java 8.

chrisly42 commented 5 years ago

Hi there. We recently upgraded assertj-swing from 2.3.0 (ahem, last Java 7 compatible version) to 3.9.2 and are now facing a similar problem where our GUI tests work with resolutions up to 2560x1400 without problems (Windows 10, Java 8), but no longer work on bigger resolutions (4K). Looks like it there is an error in calculating the positions for clicks and such.

(Unrelated: The upgrade also broke several keypress related tests due to undocumentated (!) changes in modifier handling -- the tests would press alt and ctrl keys (followed by some more keys) but not to create a modifier combination but as a serial single press and release sequence and the new code would create modifiers for the following keys).

smccants commented 5 years ago

I've started debugging the problem on one of our failing machines and it looks like it is a Java 8 bug. I'm running on Windows 10 with Java 1.8.0_u221. Here is the stack trace I have:

"Robot - Adding Transport Button" Thread at java.awt.Robot.mouseMove(Robot.java:200) at org.assertj.swing.core.RobotEventGenerator.moveMouse(RobotEventGenerator.java:112) at org.assertj.swing.core.RobotEventGenerator.pressMouse(RobotEventGenerator.java:84) at org.assertj.swing.core.RobotEventGenerator.pressMouse(RobotEventGenerator.java:79) at org.assertj.swing.core.BasicRobot.doClickWhileModifiersPressed(BasicRobot.java:479) at org.assertj.swing.core.BasicRobot.lambda$doClick$1(BasicRobot.java:461) at org.assertj.swing.core.BasicRobot$$Lambda$44.2042785389.run at org.assertj.swing.core.BasicRobot.pressModifiersWhileRunning(BasicRobot.java:504) at org.assertj.swing.core.BasicRobot.doClick(BasicRobot.java:461) at org.assertj.swing.core.BasicRobot.click(BasicRobot.java:453) at org.assertj.swing.driver.JTableDriver.click(JTableDriver.java:437) at org.assertj.swing.fixture.JTableFixture.click(JTableFixture.java:412) at org.assertj.swing.fixture.JTableFixture.click(JTableFixture.java:388) at com.hcs.orc.demo.DemoRobot.clickOnTable(DemoRobot.java:91) at com.hcs.orc.demo.AddTransportButton.work(AddTransportButton.java:61) at com.hcs.util.OAThread.run(OAThread.java:45)

The coordinates passed into java.awt.Robot.mouseMove(Robot.java:200) seem correct according to my pixel ruler, but the result is the mouse is in the wrong spot.

smccants commented 5 years ago

I can confirm that running under Java 11 seems to fix the problem. My application has other problems with Java 11.04 (Oracle JDK), but that is a different story. I suspect the Java bug is:

https://bugs.openjdk.java.net/browse/JDK-8196030?attachmentOrder=des

Which has been fixed for Java 11.0.1 according to the ticket, but a fix was never backported to Java 8 as Oracle mistakenly believed the problem didn't affect Java 8. I ran Oracle's test case SimpleAWTRobotTest and it failed under Java 1.8.0_u221, so they were mistaken that the problem didn't affect Java 8.

smccants commented 5 years ago

Just submitted a bug report to Oracle through https://bugreport.java.com, but since we don't pay for support my hopes of seeing this resolved in a timely manner are low.

chrisly42 commented 5 years ago

@smccants: Great find. However, due to the native parts that need to be patched, there's hardly a chance fixing it manually without JDK release (which will probably not happen with Java 8). I will tell our team that they need to set the UI scaling to 100% to make the tests work. Fortunately, only a few members have 4K screens so far.

Or maybe the workaround described here https://superuser.com/questions/988379/how-do-i-run-java-apps-upscaled-on-a-high-dpi-display might also work (forcing the scaling to 100% for JDK 8 java.exe)

smccants commented 5 years ago

@chrisly42 We found that setting the UI scaling to 100% did not fix the problem. For our specific purpose (the demo that our salesman uses) we are just going to move to Java 11 (AdoptOpenJDK version), although that may not be a solution for everyone.

chrisly42 commented 5 years ago

@smccants I can confirm that here the UI scaling workaround did not change anything either. Unfortunately, switching upgrading the JDK is no option for us. Fortunately, the automated testing in CI does not have that problem... Thanks for finding out the actual reason for the problem! And as a shameless plug: Try out my Cajon Plugin for AssertJ for IntellIJ :-D

smccants commented 5 years ago

Oracle has reproduced the bug on their end and accepted it as a defect in Java 8:

https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8230660

I'm closing this ticket, because it is not an AssertJ Swing defect.