OpenAdaptAI / OpenAdapt

Open Source Generative Process Automation (i.e. Generative RPA). AI-First Process Automation with Large ([Language (LLMs) / Action (LAMs) / Multimodal (LMMs)] / Visual Language (VLMs)) Models
https://www.OpenAdapt.AI
MIT License
995 stars 138 forks source link

Explore MacOS/Windows widgets #224

Open abrichr opened 1 year ago

abrichr commented 1 year ago

MacOS: https://developer.apple.com/documentation/widgetkit/creating-a-widget-extension Windows: https://learn.microsoft.com/en-us/samples/microsoft/windowsappsdk-samples/widgets/

Can we use these to indicate something about the current window?

What APIs/libraries are available that let us show an icon next to e.g. the minimize icon?

jesicasusanto commented 1 year ago

Here are the API/libraries that I find useful for this issue

  1. Create a Chrome Extension with Chrome Extension API:

Set up a manifest file (manifest.json) specifying the extension's details. Create an HTML file (popup.html) for the extension's user interface. b. Modify the Chrome UI:

Add the necessary icons (e.g., recording in progress, recording available, default stage) as image assets within the extension. Inject JavaScript code into the browser window to manipulate the UI. c. Detect Recording Status:

Determine whether a recording is in progress or available. Use Chrome Extension APIs (e.g., chrome.tabs) to communicate between the browser window and the extension. d. Update the UI:

When the recording status changes, update the extension's UI (e.g., change the icon next to the minimize icon) by modifying the DOM or using browser action or page action APIs.

  1. Electron: Build a desktop application using Electron and leverage the Electron API to customize the window's appearance and behavior. Electron provides methods to add custom icons and manipulate the window's UI elements.

  2. Windows (WinAPI): On Windows, we can use the Windows API (WinAPI) to create custom windows and modify their UI. We can utilize functions such as SetWindowText and SetWindowLongPtr to change the window's title and add custom icons.

abrichr commented 1 year ago

What we want is closer to #3, see https://www.rainmeter.net/ for inspiration

Code: https://github.com/rainmeter/rainmeter

Another one is xwidget: https://www.xwidget.com/xwidget.html

Probably the way to do this is to just create a window that is always on top and for whcih we can cutomize the styling. We update the position of the window to be at the top right of whatever active window the user is currently interacting with, and update the contents of the window to show different icons based on what we know about the active window and avaliable recordings.

So we need the window to support a few things:

  1. Modified appearance such that there is no close/minimize/maximize buttons and no title
  2. Transparent except for the puterbot image that we will be updating
  3. Fixed/always on top, e.g. https://github.com/JakubBlaha/KivyOnTop