OpenAdaptAI / OpenAdapt

AI-First Process Automation with Large ([Language (LLMs) / Action (LAMs) / Multimodal (LMMs)] / Visual Language (VLMs)) Models
https://www.OpenAdapt.AI
MIT License
764 stars 105 forks source link

Add Linux support #144

Open Mustaballer opened 1 year ago

Mustaballer commented 1 year ago

The following functions, get_double_click_interval_seconds and get_double_click_distance_pixels in utils.py only work for Windows and Mac platforms. The goal is to add Linux support as well.


    if hasattr(get_double_click_interval_seconds, "override_value"):
        return get_double_click_interval_seconds.override_value
    if sys.platform == "darwin":
        from AppKit import NSEvent
        return NSEvent.doubleClickInterval()
    elif sys.platform == "win32":
        # https://stackoverflow.com/a/31686041/95989
        from ctypes import windll
        return windll.user32.GetDoubleClickTime() / 1000
    else:
        raise Exception(f"Unsupported {sys.platform=}")

def get_double_click_distance_pixels():
    if sys.platform == "darwin":
        # From https://developer.apple.com/documentation/appkit/nspressgesturerecognizer/1527495-allowablemovement:
        #     The default value of this property is the same as the
        #     double-click distance.
        # TODO: do this more robustly; see:
        # https://forum.xojo.com/t/get-allowed-unit-distance-between-doubleclicks-on-macos/35014/7
        from AppKit import NSPressGestureRecognizer
        return NSPressGestureRecognizer.new().allowableMovement()
    elif sys.platform == "win32":
        import win32api
        import win32con
        x = win32api.GetSystemMetrics(win32con.SM_CXDOUBLECLK)
        y = win32api.GetSystemMetrics(win32con.SM_CYDOUBLECLK)
        if x != y:
            logger.warning(f"{x=} != {y=}")
        return max(x, y)
    else:
        raise Exception(f"Unsupported {sys.platform=}")```
abrichr commented 1 year ago

Looks like this might be window-manager specific; see:

https://stackoverflow.com/questions/50868129/how-to-get-double-click-time-interval-value-programmatically-on-linux

https://bbs.archlinux.org/viewtopic.php?id=240001

https://unix.stackexchange.com/questions/45925/how-to-configure-the-double-click-behavior-in-an-x-terminal

Edit: as a workaround we can always define a DEFAULT_DOUBLE_CLICK_DISTANCE_PIXELS and DEFAULT_DOUBLE_CLICK_INTERVAL_SECONDS

abrichr commented 1 year ago

We are currently using a modified version of pynput at https://github.com/abrichr/pynput that supports distinguishing between organic and injected input. Last time I checked this was not supported on Linux, but strictly speaking it's not absolutely necessary for our purpose.

abrichr commented 1 month ago

@Timothyxxx regarding https://github.com/xlang-ai/OSWorld/issues/30

abrichr commented 1 month ago

Related: https://github.com/OpenAdaptAI/OpenAdapt/issues/631