louis030195 / screen-pipe

Turn your screen into actions (using LLMs). Inspired by adept.ai, rewind.ai, Apple Shortcut. Rust + WASM.
https://screenpi.pe
MIT License
81 stars 1 forks source link

getting more activity data #10

Open louis030195 opened 3 days ago

louis030195 commented 3 days ago

dont think its a priority (vision is supposed to contains all the information)

but we can get user application focus like this

To get the current active window in Rust for different operating systems (Windows, macOS, Linux), you can use different libraries tailored for each platform. Below are examples for each OS:

Windows

For Windows, you can use the windows crate to interact with the Windows API.

use windows::{
    Win32::UI::WindowsAndMessaging::{GetForegroundWindow, GetWindowTextW, GetWindowThreadProcessId},
};

pub fn get_active_window() -> (u32, String) {
    unsafe {
        let hwnd = GetForegroundWindow();

        let mut pid: u32 = 0;
        GetWindowThreadProcessId(hwnd, Some(&mut pid));

        let mut bytes: [u16; 500] = [0; 500];
        let len = GetWindowTextW(hwnd, &mut bytes);
        let title = String::from_utf16_lossy(&bytes[..len as usize]);

        (pid, title)
    }
}

macOS

For macOS, you can use the active-win-pos-rs crate.

use active_win_pos_rs::get_active_window;

fn main() {
    match get_active_window() {
        Ok(active_window) => {
            println!("active window: {:#?}", active_window);
        },
        Err(()) => {
            println!("error occurred while getting the active window");
        }
    }
}

Linux

For Linux, you can use the x11_get_windows crate to interact with the X11 windowing system.

use x11_get_windows::Session;

fn main() {
    let mut session = Session::open().expect("Error opening a new session.");
    let active_window = session.active_window().expect("Error getting the active window.");
    let title = active_window.get_title(&session.display).expect("Error getting the title of the window.");
    println!("Active window title: {:?}", title);
}

Integration Example

Here's how you might integrate the active window detection into your existing Rust code:

// ... existing code ...

fn get_active_window_title() -> String {
    #[cfg(target_os = "windows")]
    {
        let (_, title) = get_active_window();
        title
    }
    #[cfg(target_os = "macos")]
    {
        match get_active_window() {
            Ok(active_window) => active_window.title,
            Err(_) => String::from("Unknown"),
        }
    }
    #[cfg(target_os = "linux")]
    {
        let mut session = Session::open().expect("Error opening a new session.");
        session.active_window()
            .expect("Error getting the active window.")
            .get_title(&session.display)
            .expect("Error getting the title of the window.")
    }
}

// ... existing code ...

let frame_id = {
    let mut db_clone = db_frame_id_ref.lock().unwrap();
    db_clone.as_mut().unwrap().insert_frame(Some(get_active_window_title()))?
};

// ... existing code ...

This code will get the active window title and insert it into the database along with the frame. Make sure to add the necessary dependencies to your Cargo.toml:

[dependencies]
windows = "0.51" # For Windows
active-win-pos-rs = "0.8" # For macOS
x11_get_windows = "0.1" # For Linux

References

louis030195 commented 2 days ago

would be cool to track keyboard and mouse too (optionally)

that gives me idea to train models that uses your computer hhaha