Unresponsive key listener (except modifiers) under a headless xorg server

Description

I have a use case where I want to use pynput in a headless Linux system (e.g., without a monitor or desktop manager). I was able to start a dummy xorg server and use pynput. However, the key listener is not reactive (big delay) to normal keystrokes (e.g., the callbacks are not called when I press a normal key [k, a, b, etc.]). Paradoxically, the key listener is very reactive (no delay) when a modifier key is pressed (ctr, shift, etc.). I am wondering why this happens, and if there is a way to make this work properly for the normal keys. See the example code below to reproduce this behavior.

The key listener is reactive if I ssh into the device and run the main.py script provided below as an example or if I start the desktop manager.

Thanks for any suggestions!

Platform and pynput version

Arch Linux or Ubuntu (on raspberry pi), pynput=1.7.6

To Reproduce

Login into your Linux system but do not start the desktop manager
Start xorg server (sudo X :6 -config xorg_headless.conf)
export DISPLAY=:6
python3 main.py
Press any key and then try pressing modifiers (e.g., ctr)

_xorgheadless.conf

Section "ServerFlags"
        Option "AutoAddDevices" "true"
EndSection

Section "Monitor"
        Identifier "dummy_monitor"
        HorizSync 28.0-80.0
        VertRefresh 48.0-75.0
        Modeline "1920x1080" 172.80 1920 2040 2248 2576 1080 1081 1084 1118
EndSection

Section "Device"
        Identifier "dummy_card"
        VideoRam 256000
        Driver "dummy"
EndSection

Section "Screen"
        Identifier "dummy_screen"
        Device "dummy_card"
        Monitor "dummy_monitor"
        SubSection "Display"
        EndSubSection
EndSection

main.py

import time
from pynput import keyboard

def on_press(key):
    key = str(key).strip("'")
    print(f"key pressed: {key}")

def on_release( key):
    key = str(key).strip("'")
    print(f"key released: {key}")

key_listener = keyboard.Listener(
    on_press=on_press, on_release=on_release
)

key_listener.start()

print(f"{key_listener.is_alive()} listening to key strokes...")
while True:
    time.sleep(10)

moses-palmer / pynput

Unresponsive key listener (except modifiers) under a headless xorg server #535