search

moses-palmer / pynput

Sends virtual input commands
GNU Lesser General Public License v3.0
1.77k stars 245 forks source link

Is Pynput compatible with ~~x11vnc~~ Xvfb ? #239

Open yannrichet opened 4 years ago

yannrichet commented 4 years ago

Seems strange : recording works well (both keyboard and mouse), but replaying fails (prints some spaces instead of letters). I used the standard examples. Also, using debug mode of x11vnc shows that nothing is given as input to x11vnc...

Regards.

yannrichet commented 4 years ago

python3 -c "import pynput; pynput.keyboard.Controller().press(pynput.keyboard.Key.space)" works well, while python3 -c "import pynput; pynput.keyboard.Controller().press(pynput.keyboard.Key('a'))" does not give anything...

gregseth commented 4 years ago

the second line is working if you use the KeyCode.from_char class instead:

python3 -c "import pynput; pynput.keyboard.Controller().press(pynput.keyboard.KeyCode.from_char('a'))"

but you can also use the plain character as argument to the press method:

python3 -c "import pynput; pynput.keyboard.Controller().press('a')"
yannrichet commented 4 years ago

Right, my mistake, sorry. But it remains inactive inside my x11vnc/docker...

moses-palmer commented 4 years ago

I am afraid that I do not completely understand your use case: are your trying to control x11vnc from your host using pynput, or are you running pynput on the remote machine?

If you provide a script to reproduce the issue, troubleshooting would be greatly simplified.

yannrichet commented 4 years ago

Yes, you are right :) I try to use pynput on the remote machine (in facts, it is not a 'true' remote one, just a docker one). Here is the needed stuff & process (remove all .txt extensions):

gregseth commented 4 years ago

while the expression was wrong, it seems that I was wrong too : it doesn't work with Xvfb. Here's a minimal example exhibiting the issue, with the following Dockerfile:

FROM ubuntu:18.04
RUN apt-get update

RUN DEBIAN_FRONTEND=noninteractive apt-get install -y xvfb python3-tk python3-dev python3-pip
RUN python3 -m pip install pynput

ENV DISPLAY=:0.0

reproduce with:

$ docker build -t pynput .
$ docker run -it --rm pynput bash
# Xvfb :0.0 -screen 0 1600x900x24 &
# python3 -c "import pynput; pynput.keyboard.Controller().type('XXX')"

nothing is printed to the console.

yannrichet commented 4 years ago

might be related to https://github.com/python-xlib/python-xlib/issues/156 ?

yannrichet commented 4 years ago

while the expression was wrong, it seems that I was wrong too : it doesn't work with Xvfb. Here's a minimal example exhibiting the issue, with the following Dockerfile:

FROM ubuntu:18.04
RUN apt-get update

RUN DEBIAN_FRONTEND=noninteractive apt-get install -y xvfb python3-tk python3-dev python3-pip
RUN python3 -m pip install pynput

reproduce with:

$ docker build -t pynput .
$ docker run -it --rm pynput bash
# Xvfb :0.0 -screen 0 1600x900x24 &
# python3 -c "import pynput; pynput.keyboard.Controller().type('XXX')"

nothing is printed to the console.

Well, I received 'Xlib.error.DisplayNameError: Bad display name ""' when running python3 -c "import pynput; pynput.keyboard.Controller().type('XXX')". But this is solved using just export DISPLAY=:0 :

root@085be1766e78:/# Xvfb :0.0 -screen 0 1600x900x24 &
[1] 11
root@085be1766e78:/# python3 -c "import pynput; pynput.keyboard.Controller().type('XXX')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/pynput/__init__.py", line 40, in <module>
    from . import keyboard
  File "/usr/local/lib/python3.6/dist-packages/pynput/keyboard/__init__.py", line 52, in <module>
    from ._xorg import KeyCode, Key, Controller, Listener
  File "/usr/local/lib/python3.6/dist-packages/pynput/keyboard/_xorg.py", line 39, in <module>
    from pynput._util.xorg import (
  File "/usr/local/lib/python3.6/dist-packages/pynput/_util/xorg.py", line 40, in <module>
    _check()
  File "/usr/local/lib/python3.6/dist-packages/pynput/_util/xorg.py", line 38, in _check
    display = Xlib.display.Display()
  File "/usr/local/lib/python3.6/dist-packages/Xlib/display.py", line 89, in __init__
    self.display = _BaseDisplay(display)
  File "/usr/local/lib/python3.6/dist-packages/Xlib/display.py", line 71, in __init__
    protocol_display.Display.__init__(self, *args, **keys)
  File "/usr/local/lib/python3.6/dist-packages/Xlib/protocol/display.py", line 84, in __init__
    name, protocol, host, displayno, screenno = connect.get_display(display)
  File "/usr/local/lib/python3.6/dist-packages/Xlib/support/connect.py", line 73, in get_display
    return mod.get_display(display)
  File "/usr/local/lib/python3.6/dist-packages/Xlib/support/unix_connect.py", line 76, in get_display
    raise error.DisplayNameError(display)
Xlib.error.DisplayNameError: Bad display name ""
root@085be1766e78:/# $DISPLAY
root@085be1766e78:/# export DISPLAY=:0
root@085be1766e78:/# python3 -c "import pynput; pynput.keyboard.Controller().type('XXX')"
root@085be1766e78:/#
yannrichet commented 4 years ago

Anyway, 'x11vnc' is never used by you, @gregseth . So it might not be related to...

gregseth commented 4 years ago

@yannrichet yeah, fixed the Dockerfile in my comment to set the $DISPLAY variable properly

yannrichet commented 4 years ago

One more step : the following code works well on docker:

import pynput
import Xlib
k=pynput.keyboard.Controller()
self=k
Xlib.ext.xtest.fake_input(self._display,Xlib.X.KeyPress,self._display.keysym_to_keycode(Xlib.XK.string_to_keysym('a')))
self._display.sync()

which let me think that it should work with fake_input anyway...

yannrichet commented 4 years ago

Well, I can solve the issue when I force calling fake_input instead of send_event in pynput.keyboard._xorg:

    def _handle(self, key, is_press):
...
        # If the key has a virtual key code, use that immediately with
        # fake_input; fake input,being an X server extension, has access to more
        # internal state that we
        if key.vk is not None:
            with display_manager(self._display) as dm:
                Xlib.ext.xtest.fake_input(
                    dm,
                    Xlib.X.KeyPress if is_press else Xlib.X.KeyRelease,
                    dm.keysym_to_keycode(key.vk))

        # Otherwise use XSendEvent; we need to use this in the general case to
        # work around problems with keyboard layouts
        else:
            with display_manager(self._display) as dm:
                Xlib.ext.xtest.fake_input(
                    dm,
                    Xlib.X.KeyPress if is_press else Xlib.X.KeyRelease,
                    dm.keysym_to_keycode(keysym))
#            try:
#                keycode, shift_state = self.keyboard_mapping[keysym]
#                self._send_key(event, keycode, shift_state)
#
#            except KeyError:
#                with self._borrow_lock:
#                    keycode, index, count = self._borrows[keysym]
#                    self._send_key(
#                        event,
#                        keycode,
#                        index_to_shift(self._display, index))
#                    count += 1 if is_press else -1
#                    self._borrows[keysym] = (keycode, index, count)

but I cannot decide how to properly implement this in _send_key() instead. @moses-palmer , what do you think is better ? Do we need to detect Xvfb (I imagine there is a way to), and so return to the baseline af using Xlib.ext.xtest.fake_input ?

moses-palmer commented 4 years ago

Thank you for your detailed instructions.

I ran your dockerfile and managed to confirm that it indeed did not work. Using xev inside the container I found that the only discernible difference between the events sent when using pynput and using the XTEST extension is the SYNTHETIC flag. I guess this is possible since XTEST is an extension that runs inside the server.

I have a partial solution in fixup-xorg-fake-events. It ensures that keys create from characters---such as pynput.keyboard.KeyCode.from_char('a') or anything passed to pynput.keyboard.Controller.type---sets its vk attribute to the corresponding keysym.

This ensures that the first conditional in your snippet above succeeds, and XTEST is used. I have not run any extensive tests on this branch however, other than ensuring that unmapped keys fail.

yannrichet commented 4 years ago

I get a new error, which might be just related to the new lines: 

# Create a display to verify that we have an X connection
DISPLAY = Xlib.display.Display()
atexit.register(DISPLAY.close)
...
def display_manager(display=DISPLAY):
...

:

Traceback (most recent call last): File "/usr/local/bin/snitch", line 11, in sys.exit(main()) File "/usr/local/lib/python3.6/dist-packages/snitch/main.py", line 42, in main results = WIN.playback(include_snapshots=True) File "/usr/local/lib/python3.6/dist-packages/snitch/ui/controller.py", line 134, in playback key_event_catcher=catcher File "/usr/local/lib/python3.6/dist-packages/snitch/player.py", line 74, in play event.execute() File "/usr/local/lib/python3.6/dist-packages/snitch/model/data/events.py", line 263, in execute kbd.type(self.text) File "/usr/local/lib/python3.6/dist-packages/pynput/keyboard/_base.py", line 466, in type self.press(key) File "/usr/local/lib/python3.6/dist-packages/pynput/keyboard/_base.py", line 386, in press self._handle(resolved, True) File "/usr/local/lib/python3.6/dist-packages/pynput/keyboard/_xorg.py", line 256, in _handle dm.keysym_to_keycode(key.vk)) File "/usr/lib/python3.6/contextlib.py", line 88, in exit next(self.gen) File "/usr/local/lib/python3.6/dist-packages/pynput/_util/xorg.py", line 76, in display_manager raise X11Error(errors) pynput._util.xorg.X11Error: [(BadValue(<Xlib.display._BaseDisplay object at 0x7fb2c24ba710>, b'\x00\x02\x17\x00\x00\x00\x00\x00\x02\x00\x84\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'), None)]

moses-palmer commented 4 years ago

That it interesting. This feature branch adds error checking where there was none before. Do you think you could provide a listing of the values passed to dm.keysym_to_keycode(key.vk)) on line 256 in pynput/keyboard/_xorg.py?