search

OpenInterpreter / open-interpreter

A natural language interface for computers
http://openinterpreter.com/
GNU Affero General Public License v3.0
50.54k stars 4.4k forks source link

`interpreter.computer.keyboard.write` method fails when run by a macOS background service #1292

Open getvictor opened 3 weeks ago

getvictor commented 3 weeks ago

Describe the bug

I have a background macOS service (fleetd) that's trying to run a script using interpreter APIs.

It fails keyboard.write because it doesn't seem to have access to NSPasteboard.generalPasteboard

I'm not sure why it can't get a handle to the generalPasteboard. The service has System Events TCC permissions and all other requests for TCC permissions that popped up (Accessibility, Recording Screen and Audio).

Looking at the error, the method should work even without access to the generalPasteboard. interpreter should write one character at a time instead of trying to copy/paste.

Traceback (most recent call last):
  File "/Users/victor/work/migrate-to-fleet/main.py", line 44, in <module>
    interpreter.computer.keyboard.write("Profiles")
  File "/opt/homebrew/lib/python3.11/site-packages/interpreter/core/computer/keyboard/keyboard.py", line 40, in write
    self.computer.clipboard.copy(line)
  File "/opt/homebrew/lib/python3.11/site-packages/interpreter/core/computer/clipboard/clipboard.py", line 27, in copy
    pyperclip.copy(text)
  File "/opt/homebrew/lib/python3.11/site-packages/pyperclip/__init__.py", line 139, in copy_osx_pyobjc
    board.declareTypes_owner_([AppKit.NSStringPboardType], None)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'declareTypes_owner_'

Reproduce

  1. Set up Fleet server and install a Fleet agent on your macOS device.
  2. Execute a script via your Fleet agent. The script can be like: 
    #!/bin/sh
/Users/<myUsername>/myPythonScript.py

    The python script should try and use interpreter.computer.keyboard.write("Profiles")

Expected behavior

I expected interpreter.computer.keyboard.write("Profiles") to work.

Screenshots

No response

Open Interpreter version

0.2.5

Python version

3.11.9

Operating System name and version

macOS 14

Additional context

I'm trying to create a demo where I use an Open Interpreter script to remediate an IT policy failure.

KillianLucas commented 2 weeks ago

Hey @getvictor! Thanks for discovering this and making such an incredible demo!

Were you able to find a way around this? Does pyautogui work in this environment, or another library like pynput? Would be happy to swap out the keyboard controlling library to make it work with fleet.