Open abrichr opened 2 months ago
It's possible that pywinauto
is too slow here: https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/window/_windows.py#L90C1-L98C35.
You can use the pygetwindow
library for a faster solution. Here’s how to get the name, dimensions, and position of the active window:
import pygetwindow as gw
def get_active_window_details() -> dict:
"""
Get the name, dimensions, and position of the active window.
Returns:
dict: A dictionary containing the title, left, top, width, and height of the active window.
"""
window = gw.getActiveWindow()
if window is None:
return {}
window_details = {
'title': window.title,
'left': window.left,
'top': window.top,
'width': window.width,
'height': window.height
}
return window_details
# Example usage
if __name__ == "__main__":
active_window_details = get_active_window_details()
print(active_window_details)
This method is significantly faster than using pywinauto
and should provide the necessary details about the active window. Ensure you have pygetwindow
installed:
pip install pygetwindow
We need window.get_active_window_data
(https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/window/__init__.py#L22) to return more quickly.
Probably the simplest way to approach this is to leave existing functionality as-is, and implement a separate function that uses pygetwindow to return similar data (except for the accessibility data) as quickly as possible.
As part of this, we need a test that calls both functions (the existing one based on pywinauto
and the new one based on pygetwindow
) and compares the execution time.
There are two possible solutions:
Separate window_events containing metadata from those containing accessibility data into separate event types. When the accessibility event is received, update the already written action event to point to it (or have the accessibility event point to the action event).
Only set RECORD_WINDOW_DATA = True
during evaluation, and not during recording. Then during evaluation, we can wait for the accessibility data to come back. (This seems like a hack.)
Let's go with option number 1. This will require:
WindowEvent.data
A11yEvent
that points to the relevant WindowEvent
.record.py::record()
we need to track which WindowEvent
is waiting for its corresponding A11yEvent
so that we can set the A11yEvent.window_event_id
.
Describe the bug
During recording, if the active window changes and the user emits an
ActionEvent
before a newWindowEvent
is received, the resulting segmentation (during replay) will fail.Edit: this only happens if READ_WINDOW_DATA is True.
WindowEvents
are read in https://github.com/OpenAdaptAI/OpenAdapt/blob/main/openadapt/record.py#L713To Reproduce
TODO