xdotool and Multi-pointer_X

[Nugrud]

Hi, I'm trying to run xdotool with two mouse cursors, which I wanted to use in testing: move and click with one mouse in an app and copy those movements with an offset in another window. I know how to have two mouse cursors with Multi-pointer X extension of Xorg

xinput list ⎡ Virtual core pointer id=2 [master pointer (3)] ⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)] ⎜ ↳ A4Tech USB Optical Mouse id=13 [slave pointer (2)] (...) ⎡ Second pointer id=15 [master pointer (16)] ⎜ ↳ USB OPTICAL MOUSE id=9 [slave pointer (15)] ⎜ ↳ Second XTEST pointer id=17 [slave pointer (15)]

xdotool getmouselocation gets only the location of the A4Tech mouse; Now how to send mousemove to the the other mouse?

[jordansissel]

I've never tried multi-pointer on X11, so at this time I don't know exactly how to solve. Here's what I can share (some of what follows is function/code research notes)

xdotool getmouselocation invokes the XQueryPointer() X11 API. This X11 function does not appear to support multiple pointers.

In order to make this library support multiple pointer, we'd need to figure out how to ask X11 about each pointer/cursor.

I think this would require using the xinput library for X11. For example, It seems like XIQueryPointer() from the xinput library would be a good place to start. Something like, using XListInputDevices() to find pointers, then XIQueryPointer to find the pointer location of each pointer devices? Seems doable.

[Nugrud]

Thanks for the answer, so it would be more difficult than I thought to implement, and I'm not that skilled in programming. But from what I've gathered XIQueryDevices function returns an informative structure for every device with an "int use" member, which when is equal to 0 - it's a pointer. So after a loop running through all devices it should be run, in my case, twice for the devices with id-s number 13 (A4Tech) and 9 (USB Optical Mouse), so XIQueryPointer would also need two runs; This function also needs XIButtonState, XIModifierState and XIGroupState pointers as the last three argument, which are not needed be XQueryPointer, I have no idea why, yet.

[jordansissel]

@Nugrud you did good! Some of the more confusing parts of XIQueryPointer() are mostly due to the weird ways C programmers achieve things. I can explain XIQueryPointer():

       Bool XIQueryPointer( Display *display,
                            int deviceid,
                            Window win,
                            Window *root_return,
                            Window *child_return,
                            double *root_x_return,
                            double *root_y_return,
                            double *win_x_return,
                            double *win_y_return,
                            XIButtonState *buttons_return,
                            XIModifierState *modifiers_return,
                            XIGroupState *group_return);

In C, you can output exactly 1 value using return from a function. In a lot of cases, it is nice to be able to have a function "return" more than one value, such as a function "Tell me where the mouse cursor is" might have a few outputs: x and y coordinate, which screen the cursor is on, what window the cursor is over, and maybe the cursor/pointer state (any buttons held, etc).

Such is the "multiple outputs" of XIQueryPointer. This function is nicely documented in that all the outputs are labeled with names ending in _return. In this case, the function is expecting you (or me, whatever, the programmer) to provide a place to store things like the cursor location (x,y), and other items. XIQueryPointer is kind of a funny function, to me, because it returns a true/false value based on whether the cursor is over the win window. We wouldn't need that for your proposed feature, so I won't focus on that.

How would this work in practice? The documentation for XIQueryPointer says that the function behaves the same as XQueryPointer with the small exception that XIQueryPointer needs to know what deviceid you are asking about.

I wrote some sample code to kind of show this: https://gist.github.com/jordansissel/49d0d1afe87d8654f31a255b4b613d47

In writing this sample code, I learned that there's actually two versions of XInput, and only the newer one (v2) supports multiple pointers, it appears? I learned this as I added a second master pointer with xinput create-master ... and my sample code does not show the new pointer! Confusing!

With confusion, I saw that the xinput tool does see this new pointer. So, off to the xinput source code I went.

In xinput, when listing devices, it does a version check for xinput v1 or v2: https://github.com/freedesktop/xorg-xinput/blob/cef07c0c8280d7e7b82c3bcc62a1dfbe8cc43ff8/src/list.c#L403-L406

If v1, it uses XListInputDevices(). If v2, it uses XIQueryDevices.

I was kind of shocked to learn there are two semi-compatible versions of the XInput extension, and I didn't notice this fact anywhere in the documentation. Very weird.

Looking further, it seems like XInput v2 was added in 2009 and brought multiple-pointers to X11.

So, the short version is that this code keeps growing in complexity. Maybe we don't have to do a version check. I'll try some other sample code to see what I can learn.

[jordansissel]

I got some sample XInput v2 code working: https://gist.github.com/jordansissel/d1b434e15a79ed0990055c911d0e3e9a

The changes between xinput v1 and v2 pretty short, for the example code I wrote.

• XListInputDevices becomes XIQueryDevice() asking for XIAllMasterDevices
• IsXPointer becomes XIMasterPointer
• device ids are int in v2, but are unsigned long in v1
• XFreeDeviceList becomes XIFreeDeviceInfo

--- main.c  2021-05-26 23:43:02.008574733 -0700
+++ main-v2.c   2021-05-26 23:43:21.668650686 -0700
@@ -14,12 +14,14 @@

   // Fetch the list of XInput devices.
   // Store the "count" of these devices in `device_count`
-  XDeviceInfo *inputs = XListInputDevices(d, &device_count);
+  // XDeviceInfo *inputs = XListInputDevices(d, &device_count);
+  XIDeviceInfo *inputs = XIQueryDevice(d, XIAllMasterDevices, &device_count);
+

   for (int i = 0; i < device_count; i++) {

     // Only process pointer inputs
-    if (inputs[i].use == IsXPointer) {
+    if (inputs[i].use == XIMasterPointer) {
       double x, y;

       // Create throw-away storage for saving things we don't want
@@ -30,7 +32,7 @@
       XIGroupState _xig;

       XIQueryPointer(d,                    // display
-                     inputs[i].id,         // deviceid
+                     inputs[i].deviceid,         // deviceid
                      DefaultRootWindow(d), // "win"

                      // The rest of the arguments are passed as pointers intended for XIQueryPointer
@@ -46,9 +48,9 @@
                      &_xig  // group_return, don't care for this example.
       );

-      printf("Pointer id:%lu at location %0.0f,%0.0f\n", inputs[i].id, x, y);
+      printf("Pointer id:%d at location %0.0f,%0.0f\n", inputs[i].deviceid, x, y);
     }
   }

-  XFreeDeviceList(inputs);
+  XIFreeDeviceInfo(inputs);
 }

[jordansissel]

From a technical "can it be done with code" perspective, I think we can add this to xdotool.

How, I'm not quite sure yet. Should we add another command? getpointers or something? What do you think? How would you use the results?

[Nugrud]

Thank you for all the explanations! It's really useful to me. I think I could use only the sample code to parse pointer locations from the mouse not available from xdotool and send them for use in 'xdotool mousemove' command and it would solve my task, so I'm really cool now :)

I think that if anybody else needed that functionality in xdotool one would need the option to pass deviceid to xdotool, say 'xdotool -id 9 mousemove 100 100' and then you wouldn't even need XIQueryDevice, only directly pass that id to XIQueryPointer? People who want two pointers, which is nowhere default or out-of-the-box, have to run 'xinput list' && 'xinput create-master Second'' && 'xinput reattach id1 id2', and so they would know the deviceid's.

[Nugrud]

Heh, silly me, I need to copy mouse clicks too the other pointer too, not just coordinates.

[Nugrud]

Well, after trying many combinations with deviceid's in function I now know how tu use, I discovered that xdo_click_window function from xdo.h always grabs the pointer I used to click and so I can never reproduce the click on the second pointer... https://gist.github.com/Nugrud/79a560f66af3c5696fa2b88978622c60

probably because xdo_mouse_move grabs the pointer that was last clicked, and there is no XIGrabPointer equivalent for XInput2 like there is for QueryPointer....

[jordansissel]

Hmm. xdo_click_window uses XTEST (XTestFakeButtonEvent function) in most cases to send mouse clicks (Unless you specify a specific window to click, then it uses XSendEvent)

I wonder... this might get weird, but you could try moving the XTEST pointer to your other pointer master?

% xinput
⎡ Virtual core pointer                          id=2    [master pointer  (3)]
⎜   ↳ Virtual core XTEST pointer                id=4    [slave  pointer  (2)]
...

If the XTEST pointer is reattached to your second pointer, maybe xdotool click could be used to click with your second pointer? I haven't tried this, but something like xinput reattach might help you achieve this.

Thoughts?

[Nugrud]

both master pointers have slave XTEST pointers and they are hard linked with each other, one can't reattach them; and only one set is "core", and that one "sends core events". I am learning about it every day now, but still can't find any solution.

EDIT: I'll try with this one:

include <X11/extensions/XInput2.h>

Status XIGrabDevice( Display display, int deviceid, Window grab_window, Time time, Cursor cursor, int grab_mode, int paired_device_mode, Bool owner_events, XIEventMask mask);

[jordansissel]

both master pointers have slave XTEST pointers

Fascinating!

If none of these X11 experiments work out for you, there's still another option. Assuming the OS is Linux, that is. Linux has a system called uinput which allows you (or software) to act as a peripheral like a mouse or keyboard. https://www.kernel.org/doc/html/v4.12/input/uinput.html

This is most commonly (in my experience) used with libevdev. Python has bindings which make this a more accessible than C might be.

Here's an example which creates a tablet device and makes some artificial movements with it: https://gitlab.freedesktop.org/libevdev/python-libevdev/-/blob/master/examples/fake-tablet.py

[jccolson]

Hello

Actually, I use xdotool and xte with two mouses (two pointers), two screens and four workspaces. @Nugrud : Try xte -i id ..., (where id is a XTEST pointer or a XTEST keyboad), this let me "drive" the two mouses. http://hoopajoo.net/projects/xautomation.html

But I must stay on the same workspace because xte don't directly send events to windows (no windows option).

@jordansissel : Can you add a parameter like the xte -i option ? Ex: xdotool getmouselocation -i 15

[jordansissel]

I’m open to adding xinput2 support for cases like this. 👍🏻👍🏻

On Thu, Jun 10, 2021 at 2:27 PM Jean-Claude @.***> wrote:

Hello

Actually, I use xdotool and xte with two mouses (two pointers), two screens and four workspaces. @Nugrud https://github.com/Nugrud : Try xte -i id ..., (where id is a XTEST pointer or a XTEST keyboad), this let me "drive" the two mouses. http://hoopajoo.net/projects/xautomation.html

But I must stay on the same workspace because xte don't directly send events to windows (no windows option).

@jordansissel https://github.com/jordansissel : Can you add a parameter like the xte -i option ? Ex: xdotool getmouselocation -i 15

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/jordansissel/xdotool/issues/330#issuecomment-859085092, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABAF2U76D5TCS2MYZNZ2NDTSEU3PANCNFSM45SPH6RQ .

[Nugrud]

Hello

Actually, I use xdotool and xte with two mouses (two pointers), two screens and four workspaces. @Nugrud : Try xte -i id ..., (where id is a XTEST pointer or a XTEST keyboad), this let me "drive" the two mouses. http://hoopajoo.net/projects/xautomation.html

But I must stay on the same workspace because xte don't directly send events to windows (no windows option).

Thanks! It worked, although I had to add usleep(200000) between mousemove and mouseclick to have an effect.

[schrmh]

http://who-t.blogspot.com/2009/05/xi2-recipes-part-1.html has a five part series about XInput2 and MPX. I would like to request something that should be implemented when support for multiple pointers is implemented: Clicks that won't give focus to windows. That is actually possible with xse / XSendEvent: xse -win WindowID '<Btn1Down> 10 10' https://gist.github.com/schrmh/989fe3a5316ba7b50b864f7acf978ce4

On the other hand I haven't found out how to do the same by using xdotool / xte / xautomation / XTest: https://gist.github.com/schrmh/74c96ecc9f0d30bd37b68af4755be102

[jordansissel]

Clicks that won't give focus to windows.

Not sure exactly how this could work with XTEST or XInput2, but I think this works today:

your mention of XSendEvent reminded me: xdotool will use XSendEvent if you specify the --window flag for input commands like key, click, type, etc.

On Thu, Aug 12, 2021 at 9:37 PM schrmh @.***> wrote:

http://who-t.blogspot.com/2009/05/xi2-recipes-part-1.html has a five part series about XInput2 and MPX. I would like to request something that should be implemented when support for multiple cursors is implemented: Clicks that won't give focus to windows. That is actually possible with xse / XSendEvent: xse -win WindowID ' 10 10' https://gist.github.com/schrmh/989fe3a5316ba7b50b864f7acf978ce4

On the other hand I haven't found out how to do the same by using xdotool / xte / xautomation / XTest: https://gist.github.com/schrmh/74c96ecc9f0d30bd37b68af4755be102

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jordansissel/xdotool/issues/330#issuecomment-898185901, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABAF2TL5YUPI6DK4QTXVWTT4SORNANCNFSM45SPH6RQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[whot]

A few general comments here:

• XSendEvent is usually filtered by toolkits and may end up being a noop, so focusing on that is likely not useful
• all core requests use PickPointer or PickKeyboard in the server to decide which device to use. Which device is picked is decided by the ClientPointer. For XTest devices it works the same way, if you set the ClientPointer before issuing the XTest request, it'll pick the other device. But to do so, you'd need minimal xi2 support in xdotool. This will also work with grabs, it you set the CP before the grab it'll pick that for the grab.
• I recommend building support for device name resolution from day one, the deviceid is prone to change and thus sucks for scripting, better to support what xinput does where you can just specify the device name and it'll do a name match.

there is no XIGrabPointer equivalent for XInput2

see XIGrabDevice()

As for the rest: XI2 is finished in regards to the core API but other things are missing, e.g. there is no XI2 support for XTest and thus no support for touch events through XTest. No-one's ever shown enough interest for it (that goes for virtually all the missing features). There are odd corner-cases in other protocols where a device may be used but there may not be an XI2 call for it. XI2 is widely used but multiple master devices is virtually unsupported everywhere [1] and I only hear about a use case maybe once a year. Even if the infrastructure is in place most of the rest of the UI falls apart with multiple pointers anyway and no-one's interested in rewriting all this for zero users.

[1] Wayland had multiple pointer support from day 1 and still most compositors wouldn't handle it correctly - because the UI is too hard.

[schrmh]

Thanks a bunch for your clarifications and tips, Peter (also your response to my second e-mail was faster than me getting ready to write a response on this Issue, lol).

Even tho it does not add much to this discussion, I want to mention that the touch situation might get better? Cause distros for ARM devices (especially for phones/tablets) are slowly seeing progress. I tried postmarketOS with phosh and at least some applications handle multi-touch well and I guess at least the more well-known gnome apps will work well on mobile devices in the future.

[whot]

Yes, touch is getting standard and wildly supported. Multitouch has been supported in X since 2012 or so (XInput 2.2) and the major toolkits support it. There's some quirkiness in X because the X server has to emulate a pointer for the first touch but that goes away in Wayland which has touch from the get-go and expects all clients to handle this (effectively pushing the emulation to the toolkits for legacy apps).

but multi-pointer and multi-touch are two completely different beasts. Multi-pointer divides the input devices into logical seats (i.e. one pair per user), whereas multitouch adds multiple points per user. You can have multi-touch-multi-pointer but only with some specific hardware (e.g. google for diamondtouch) but then too you have all the same UX issues that multi-pointer faces.

[rustyx]

What's the status of this? It would be really nice to have this feature.

Perhaps the most user-friendly solution is to add an optional --pointer-id parameter to getmouselocation, mousemove, mousedown, mouseup, etc. commands that takes a master pointer ID, and automatically use XI APIs.