Animation rendering glitch on JavaFX WebView in recent releases (not present in v9.2.0)

[KlemenDEV]

Check for duplicates

• [X] I have searched for similar issues before opening a new one.

Description

When dragging blocks, a strange glitch sometimes happens in the JavaFX WebView window that did not happen in Blockly v9.2.0 but happens in 10.0.x and in 10.1.x.

Unfortunately, the same thing does not happen in the minimal reproducible example I tried to make when executed in Google Chrome. It does not seem that anything special is required for this to start happening in the WebView, however, in Chrome it never happens.

https://github.com/google/blockly/assets/16374228/1b80e0b8-982b-4f77-baa7-2b96120061e3

I understand this could be a problem with JavaFX WebView, but I find it interesting this is used perfectly with previous releases.

---

Updated demo: demo-with-run.zip

To build: ./gradlew build

To run: ./gradlew run

Java 17 is needed

[rachel-fenichel]

This looks like the mouse pointer is jumping, rather than the animation. To debug more, can you add an event listener on the page that just logs the mouse location, and see if it's changing unexpectedly mid-drag?

[KlemenDEV]

The glitch happens the moment dragging is finished. So one grabs the block, moves it to attach and after/at the moment when releasing the mouse cursor, the glitch happens.

I tried this code

let lastMouseX = null, lastMouseY = null;

document.addEventListener('mousemove', (event) => {
    if (lastMouseX !== null) {
        const distance = Math.sqrt((event.clientX - lastMouseX) ** 2 + (event.clientY - lastMouseY) ** 2).toFixed(2);
        console.log(`X=${event.clientX}, Y=${event.clientY}, Distance=${distance} pixels`);
    }

    lastMouseX = event.clientX;
    lastMouseY = event.clientY;
});

but the mousemove event is not firing while dragging Blockly block.

However, I have visually observed the mouse visually, and it seems only the Blockly block moves, while the cursor remains at a fixed location, at least visually.

[BeksOmega]

@KlemenDEV would you be able to provide a minimal reproducable example of this using JavaFX? That would be very helpful for debugging!

[KlemenDEV]

Here is an example workspace, the minimal needed to get Blockly running on JavaFX webview: (https://github.com/google/blockly/files/12574277/demo.zip)

Here is video showing glitches in this minimal reproducible example (watch whole video as it does not happen with every block move):

https://github.com/google/blockly/assets/16374228/1b80e0b8-982b-4f77-baa7-2b96120061e3

The workspace uses Gradle to load JavaFX. Can be opened in IDEA or launched from command line using gradle.

JS files (jsdist folder) in demo are from https://github.com/google/blockly/releases/tag/blockly-v10.1.2

[KlemenDEV]

If you need anything else or help getting the demo up and running, let me know :)

Glad to help to make Blockly work with JavaFX webview properly too.

[KlemenDEV]

Additional info. This is not visible in the example video above, but in some (rare) cases, when grabbing block connected to another block and disconnecting it, an additional insertion marker is shown at aj incorrect position (see black shadow block on top of if block) as can be visible in this screenshot I managed to grab it while messing around to see any patterns on when this happens:

[BeksOmega]

If you need anything else or help getting the demo up and running, let me know :)

Glad to help to make Blockly work with JavaFX webview properly too.

Hello! Thank you for writing up the demo =) I cloned it and built it using ./gradlew build but I'm not sure how to get it to run. If I run java -jar ./build/libs/BlocklyJava-1.0-SNAPSHOT.jar I get no main manifest attribute, in ./build/libs/BlocklyJava-1.0-SNAPSHOT.jar.

[KlemenDEV]

Updated demo: demo-with-run.zip

To build: ./gradlew build

To run: ./gradlew run

Java 17 is needed

If glitches do not happen or are rare, adding more blocks to demo.xml usually make them more common, especially if more blocks are connected

[BeksOmega]

Ran a bisect and found the root cause to be render queueing: https://github.com/google/blockly/pull/6851

So sounds like it's probably an issue on the JavaFX end if they're not properly triggering callbacks before the frame is drawn. I'm going to send this issue back to triage for feedback.

[KlemenDEV]

Thanks for bisecting the issue. JavaFX (OpenJFX) uses WebKit "behind the scenes" if it helps the triage team.

In case there is not much that can be done in regard to #6851 as of course the benefit of that PR is much bigger than the need for JavaFX support, could some workaround be implemented for JavaFX at least, maybe togglable by some parameter?

[BeksOmega]

If we wanted to it would be technically possible to add a toggle to the render management system that always triggers an immediate render on queueRender instead of waiting for the requestAnimationFrame callback. But I'll leave it up to other folks whether we want to do that or not!

[KlemenDEV]

Alright, hope something like this could be done.

Also if someone knows how I could phrase bug report to the OpenJFX developers, it would be welcome as right now I am not sure what exactly happens "behind the scenes" for this to happen.

[rachel-fenichel]

Are we able to detect that we're in JavaFX through the user agent or some other form of inspection? If yes, I'm fine with adding behaviour to always trigger an immediate render. However, I want it to be entirely internal (that is, Blockly decides whether to do it), not something that a developer can set through a function.

[BeksOmega]

Are we able to detect that we're in JavaFX through the user agent or some other form of inspection? If yes, I'm fine with adding behaviour to always trigger an immediate render. However, I want it to be entirely internal (that is, Blockly decides whether to do it), not something that a developer can set through a function.

JavaFX is one of our user agent consts! https://github.com/google/blockly/blob/00d870ee03ffd502984aa2881d6c1d1d1c406596/core/utils/useragent.ts#L74

[KlemenDEV]

If yes, I'm fine with adding behaviour to always trigger an immediate render. However, I want it to be entirely internal (that is, Blockly decides whether to do it), not something that a developer can set through a function.

That would be really neat :)

[KlemenDEV]

Thank you!

[BeksOmega]

Thank you!

Change should go out with v10.2 on the 28th!

[KlemenDEV]

Would it be possible to rever this fix as the fix now makes Blockly unusable due to: https://github.com/google/blockly/issues/7556?

This fix fixes visual glitch, but introduces much more concerning usability issue reported at https://github.com/google/blockly/issues/7556