[BUG] - MouseEvents are captured by Pharo IDE and others Bloc Windows

[Nyan11]

Hello,

Description

I was trying to do a drag and drop in Bloc using the MouseEvents. And I found a strange bug. The Pharo IDE and other Bloc Windows seem to break the MouseEvents.

severity: medium

Example 1 (no bug)

In this GIF, I start to drag the blue square inside the Window (the right button of my mouse is down). Then I keep dragging the blue square but i move my mouse outside the window (the right button of my mouse is down). Then I stop the drag outside the window (the right button of my mouse is up). Then i go back inside the window without pressing any button (the right button of my mouse is up). The shape teleport one time to the mouse location and stop moving (the right button of my mouse is up).

Example 2 (bug)

In this GIF, I start to drag the blue square inside the Window (the right button of my mouse is down). Then I keep dragging the blue square but i move my mouse to the Pharo IDE Window (the right button of my mouse is down). Then I stop the drag inside Pharo IDE Window (the right button of my mouse is up). Then i go back inside the window without pressing any button (the right button of my mouse is up). The shape teleport to the mouse location and it keep folowing the mouse without any button pressed (the right button of my mouse is up)_.

Example 3 (bug)

In this GIF, I start to drag the blue square inside the Window (the right button of my mouse is down). Then I keep dragging the blue square but i move my mouse to a new Bloc Window (the right button of my mouse is down). Then I stop the drag inside the Bloc Window (the right button of my mouse is up). Then i go back inside the first window without pressing any button (the right button of my mouse is up). The shape teleport to the mouse location and it keep folowing the mouse without any button pressed (the right button of my mouse is up).

Reproduction

code to reproduce:

position1 := nil.
offset1 := nil.

element1 := BlElement new background: Color blue; relocate: 100 asPoint; yourself.
element1 when: BlDragStartEvent do: [ :evt |
    evt consumed: true.
    offset1 := element1 position.
    position1 := evt position ].
element1 when: BlDragEvent do: [ :evt |
    element1 relocate: (evt position + offset1 - position1) ].

position2 := nil.
offset2 := nil.

element2 := BlElement new background: Color red; relocate: 100 asPoint; yourself.

element1 openInNewSpace.
element2 openInNewSpace.

expected result:

The shape should stop following the mouse after i stop pressing the right button of my mouse.

Other

pharo image : Pharo-11.0.0+build.346.sha.b6ce4527b4f4881d6c6b4f968f1d1348b783b7a2 (64 Bit) bloc version : [4d71a7e] - 2023-01-16 12:43 OS: Microsoft Windows

[Nyan11]

My bloc settings:

[tinchodias]

It's really great news that you are working on drag and drop between windows. I never explored well this field. I wonder what's the behavior you want to have. Ideally I would expect to see a small red rectangle behind the cursor until I drop it and it appears in the other window. But it's not clear to me. I tested e.g. a LibreOffice or other software, how they works when I drag stuff from one window to another on Windows 10, on Macos and Gnome desktop on linux, and it is not homogeneous.

The SDL2 host of Bloc processes the events from the same loop as the Pharo IDE. This is implemented in OSWindow-SDL2 package. I don't know if this package was used with multiple windows. So this issue requires low-level debugging so I'll tell you some tricks.

First: If you look for any SDL2 tutorial, you will find a C main with a while loop that processes SDL_Events obtained from a call to SDL_PollEvent. The same is in Pharo. You can check it by opening Transcript and place a traceCr in OSSDL2Driver>>#processEvent:.

For example, adding a sdlEvent traceCr. will show MANY of these lines:

a SDL_Event(#[0 3 0 0 219 215 4 0 1 0 0 0 1 0 0 0 82 0 0 0 82 0 0 64 0 0 129 12 172 127 0 0 2 198 137 54 248 127 0 0 213 233 137 54 248 127 0 0 148 112 136 54 248 127 0 0])
a SDL_Event(#[1 3 0 0 73 216 4 0 1 0 0 0 0 0 0 0 82 0 0 0 82 0 0 64 0 0 129 12 172 127 0 0 2 198 137 54 248 127 0 0 213 233 137 54 248 127 0 0 148 112 136 54 248 127 0 0])
a SDL_Event(#[0 3 0 0 170 216 4 0 1 0 0 0 1 0 0 0 225 0 0 0 225 0 0 64 1 0 16 11 172 127 0 0 128 28 148 0 0 96 0 0 0 190 87 22 248 127 0 0 208 102 194 180 247 127 0 0])

These are the raw events that Pharo obtains from the SDL_poll function (called every 5 milliseconds).

Or, if you place a mappedEvent traceCr. in OSSDL2Driver>>#processEvent:, you will log in Transcript the events already translated to instances of the SDL2MappedEvent hierarchy.

The drag and drop of the blue rectangle looks like:

[...]
a SDL_MouseButtonDownEvent
a SDL_MouseMotionEvent
[ ... many more SDL_MouseMotionEvent ... ]
a SDL_MouseMotionEvent
a SDL_MouseButtonUpEvent
[...]

But the printOn: of these classes is not very informative, only the class name, so you can compile on them a better one. They have in common the windowID accessor, so this code can already help to debug:

printOn: aStream
    super printOn: aStream.
    aStream
        nextPutAll: ' windowId: ';
        print: self windowID

You must add this method on each of these classes (or maybe add a Trait with the method to be more sophisticated): SDL_MouseButtonUpEvent, SDL_MouseButtonDownEvent and SDL_MouseMotionEvent. But there are more event properties that can be printed on each case.

Last trick: Instead of logging a string from the events, you can save the original event objects. Insert these lines in OSSDL2Driver>>#processEvent::

        TRACED_EVENTS ifNil: [ TRACED_EVENTS := OrderedCollection new].
        TRACED_EVENTS add: sdlEvent.

The compiler will ask you what is the undeclared upper-cased name, and you choose "Declare new Global".

Then go to a Workspace and you can do things like:

• Inspect TRACED_EVENTS and click on refresh button from time to time, to see the new events.
• Inspect TRACED_EVENTS copy which will open a snapshot of the collection of events that doesn't gets new events.
• Do TRACED_EVENTS := nil to reset the object log. Or more.

Of course there are packages like Beacon or Announcements to log the events, too. But this simple approach also works.

Hope it helps to spot what is wrong. Hope it's something fixable at the level of SDL2 users. It could also happen that SDL2 is not well prepared to handle this feature, but don't think it's the case.

[Nyan11]

Thank you for the debugging tips. I will investigate futher.

As for the drag and drop behaviour, i will think about it.

[tinchodias]

Hi, did you find something on this issue? I can help... last week only commented how would I start debugging it, so you count with that information. But I can take a look. Regards.

[tinchodias]

After searching a bit more on the web, I think there is no SDL2 support nor any cross-platform C library to start the drag-and-drop. We may need to implement bindings to platform-specific APIs to do it. Stuff like:

• https://learn.microsoft.com/en-us/windows/win32/com/drag-and-drop (Windows)
• https://developer.apple.com/documentation/appkit/nsdragginginfo/1416092-draggingpasteboard (MacOS)

In SDL2 we only saw support for "Drop" but not to start the drag process, i.e. the user drags a file into the SDL2 window from the system file browser. Doc: https://wiki.libsdl.org/SDL2/SDL_DropEvent

[tinchodias]

https://wiki.libsdl.org/SDL2/SDL_CaptureMouse

Has good news:

"Please note that as of SDL 2.0.22, SDL will attempt to "auto capture" the mouse while the user is pressing a button; this is to try and make mouse behavior more consistent between platforms, and deal with the common case of a user dragging the mouse outside of the window. This means that if you are calling SDL_CaptureMouse() only to deal with this situation, you no longer have to (although it is safe to do so). If this causes problems for your app, you can disable auto capture by setting the SDL_HINT_MOUSE_AUTO_CAPTURE hint to zero."

(we have this version)

I will check

[tinchodias]

I did a test without Bloc, it was SDL2+Cairo only.

• It worked on MacOS Ventura: https://youtu.be/kbXim5OY9sQ
• Also on Windows 10: https://youtu.be/S9hbx0-SOTg
• Partially worked on the Gnome of my Ubuntu: the cursor was black/white, no colors like in other platforms.

[tinchodias]

Details of the test recorded in those videos:

• When you hold the mouse button pressed, SDL keeps the original cursor until you release the button. Very convenient for dragndrop.
• The cursor was created with SDL_CreateColorCursor, which receives a "RGBA image". That means we could render a custom cursor e.g. with a hand and a representation of what the user is dragging behind (didn't check what happened on my gnome, maybe color cursors are not supported in such platform).
• On mouse button release, I manually tell the "drop" window to grab the focus. Otherwise, it looked strange.
• This doesn't work on other native applications where the user might want to drop content. I didn't see any support from SDL for that. I think this should be implemented calling native Windows/MacOS/etc. APIs.

[tinchodias]

For the record:

Ubuntu screen recorder didn't capture the buggy cursor: https://youtu.be/t2Csc47EcpE

The cursor looks kind of converted from 32bits to 1-bit depth. I first tried on X11 then switched to Wayland and same.

However, the cursor looked correct with system's SDL2, which is v2.0.20. (I deleted the SDL2 library that comes with PharoVM to let Pharo finds it).

I think it is this SDL2 issue: https://github.com/libsdl-org/SDL/issues/4856

So:

• It may be driver-dependent, and e.g. work fine on another machine with same Ubuntu.
• It may work fine with a newer SDL2 version (now 2.26.x)

And drag and drop can be done with a cursor without color. This is too off-topic for this issue... sorry for that. But I considered it was a good place to present these observations... I stop here for the moment!