Press/drag fails for some macOS apps

[GuruGurra]

Some macOS apps seem to monitor mouse events in some way that SharpHook doesn't cover. Often, it is click-drag operations that don't work.

Bitwig is such an example. If I use ShatrpHook to position the mouse cursor over a Mute button in the app and press the left mouse button, it works as expected.

But if I move the mouse cursor to a volume fader, click and hold the left mouse button and move the mouse cursor, it does not work. It should "grab" the volume fader and move it, which it does when I do it manually but not when using SharpHook. It registers the left mouse button press (evident since the object is being selected), but nothing else happens when the mouse cursor moves; the volume fader handle isn't brought along with it. The app behaves as if I press the mouse button and then keep the mouse stationary.

What could cause such a behavior?

[TolikPylypchuk]

Hello! Thanks for posting this issue! I will look into it. macOS has some weird specifics about mouse events, this is probably happening because some event flag hasn't been set by libuiohook.

[TolikPylypchuk]

Could you please also provide a minimal reproducible example so that it's easier to debug?

[GuruGurra]

I realize that having example code for debugging would simplify things a lot, and I'd love to supply that if I had something like that.

My application is a Stream Deck plugin, and as such, it's not a self-sustained executable and cannot be run outside of the Stream Deck environment.

To give you a minimal reproducible example, I must start from scratch and invent something new, and the question is if it is quicker if I do it than if you do it.

I'm very interested in solving this problem and will help however I can. I'm just not sure that creating a new solution from scratch is the best way to help.

I can look at it, but I can't say when I can have a new solution to share.

---

Från: Tolik Pylypchuk @.> Skickat: den 6 augusti 2024 20:48 Till: TolikPylypchuk/SharpHook @.> Kopia: Gunnar Carlson @.>; Author @.> Ämne: Re: [TolikPylypchuk/SharpHook] Press/drag fails for some macOS apps (Issue #116)

Could you please also provide a minimal reproducible example so that it's easier to debug?

— Reply to this email directly, view it on GitHubhttps://github.com/TolikPylypchuk/SharpHook/issues/116#issuecomment-2271929133, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKMO447WZAD2IZXR67SFBFLZQEK6BAVCNFSM6AAAAABL6FSPUCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENZRHEZDSMJTGM. You are receiving this because you authored the thread.Message ID: @.***>

[TolikPylypchuk]

Yeah, no problem, I will try to reproduce it on my own then. If I can't, then we'll have to think of something.

[GuruGurra]

It was easier than I thought to extract the things from my code. With the following code, I can replicate the issue on my mac.

using SharpHook;
using SharpHook.Native;

namespace MouseMoveTest
{
    internal class Program
    {
        //TODO: Set startX and startY to a suitable screen location
        static short startX = 466;
        static short startY = 280;

        static short stepSize = -2;
        static int stepDelay = 50;

        static short currentX;
        static short currentY;
        static EventSimulator? simulator;

        static void Main(string[] args)
        {
            simulator = new EventSimulator();
            currentX = startX;
            currentY = startY;

            // Move to start position
            MoveMouse(0, 0);

            // Press left mouse button
            PressMouseButton();

            // Move mouse left for 1 second
            for (int i = 0; i < 20; i++)
            {
                MoveMouse(stepSize, 0);
                Thread.Sleep(stepDelay);
            }

            // release mouse button
            ReleaseMouseButton();
        }

        static void MoveMouse(short offsetX, short offsetY)
        {
            simulator?.SimulateMouseMovement((short)(currentX + offsetX), (short)(currentY + offsetY));
            currentX += offsetX;
            currentY += offsetY;
        }

        static void PressMouseButton()
        {
            simulator?.SimulateMousePress(currentX, currentY, SharpHook.Native.MouseButton.Button1);
        }

        static void ReleaseMouseButton()
        {
            simulator?.SimulateMouseRelease(currentX, currentY, SharpHook.Native.MouseButton.Button1);
        }
    }
}

[TolikPylypchuk]

I've tried running this code on macOS Sonoma, and everything works fine. I've tried this out with Visual Studio Code, Safari, and JetBrains Rider, and text was selected in these applications, meaning the mouse cursor really was dragged.

I've also checked the code of libuiohook - it does simulate kCGEvent{Left,Right,Other}MouseDragged if a button was perviously clicked.

Could it be a problem with the target application? Maybe, it doesn't handle simulated drag events correctly? Sorry, I don't have the ability to install Bigwig as it appears to be a huge application. Do you have other examples of applications that misbehave when mouse dragging is simulated? If not, then unfortunately, it appears that there's nothing I can really do here.

[GuruGurra]

I don't know how you can see Bitwig as a huge app. It uses 1.75 GB RAM on my Mac.

I know there are other apps with similar problems but I haven't the details. I have asked to get more info about that, but I know they are advanced graphical apps so they will probably be larger than Bitwig.

Could it be a problem with the target application? Maybe, it doesn't handle simulated drag events correctly?

Those are interesting questions. I didn't think an app could detect that it was a simulated event? If an app reacts to physical mouse movement but not to simulated mouse movement, does that mean that it uses some technology that is not possible to simulate?

[TolikPylypchuk]

Ok, my choice of words was poor - I meant that Bitwig looks complex.

Regarding the problem being in the target app - here's an example. SharpHook can simulate text entry, but on macOS applications are not required to handle such events, meaning that text entry simulation may simply not work in some apps and there's nothing SharpHook can do about it. Maybe, it's a similar situation here where Bitwig doesn't handle drag events, instead it handles mouse click + mouse move? This is just an assumption though.

Could you please try to reproduce this in other applications like browsers or text editors? If you see that the mouse really is dragged (e.g. text is selected) then I think SharpHook simulates this corectly.

[GuruGurra]

As I mentioned in my initial report, this problem only affects some apps. I thought it affected more apps, but it turns out that Bitwig may be the only one. For the majority of apps, it works as expected.

So, the verdict is that if it doesn't work, it doesn't work, and there's nothing we can do about it?

[TolikPylypchuk]

Yes, I believe that there's nothing we can currently do. Maybe, you can contact Bitwig and submit a bug report for it?

I'm going to close this issue, but we can continue the discussion, and I will reopen it if some new info comes up about this bug.

[GuruGurra]

I installed Karabiner-EventViewer on my Mac to see if I could get some more information on what was going on. It doesn't track mouse movements, but it does log mouse buttons.

When using the physical mouse, the mouse button presses (and releases) were logged as expected. When using SharpHook, nothing was logged.

I don't know what I want to say with this, other than there are clearly different ways to detect mouse events on a Mac.

[TolikPylypchuk]

Hm, ok, I will take a more detailed look at this. Thanks for checking this!

[TolikPylypchuk]

I've tried investigating this issue using Karabiner-EventViewer, and yes, it doesn't show any simulated mouse events. I don't know what can be changed about simulating mouse events on macOS, and I didn't find anything that can be useful. Of course, I pretty much have surface-level knowledge about the way macOS simulates events so I may have missed something.

Currently I don't think I can do much about this issue, but I will keep this open for a while - maybe something will come up.