Questions regarding the workings/concept, possibilities and future plans

[claudiusraphael]

Haven't tested yet, but a question in forefront: What is the application like?

What i mean is:

Does it capture the mouse-coordinates per a) Whole Desktop b) per screen c) per application Window

and whatever the answer(s) might be, why is it that way?

I am wondering for example if it would be possible to configure specific use-cases like track pointer of:

• a remote-connection/VM via RDP/Spice and similar
• a browser tab or a sub-window in a MDI app
• a toolbox/window in a MDI/.Net app, for example a VST-Synth inside a DAW

Any hints on the inner workings would be wonderful.

Also, you mentioned you are working on a next version - any blog/changelog/issue-tracker one can follow?

Last question: How Python-specific is it - meaning are you using patterns/functions that are exclusive to Python or do you think it can be ported (halfway easily) to a Rust/WebAssembly/Node/Deno-Combo?

If you feel able to answer (some of) these questions it might be a good start for a wiki?

[claudiusraphael]

Oh one more (technical) additional question:

The captured data, do you draw it directly or is it stored as raw data, so it might even be used for reproduction for example for automated testing of Apps in Virtual Machines and to enable statistical analysis over time?

[huntfx]

Hey, the tracking is done per application window if a valid executable is detected (got from the AppList.txt file), or per screen if not. Each resolution is tracked separately as raw values in a numpy array, where the image generation process will upscale everything, merge it together, convert to the RGB values and downscale again (last 2 things may be the other way around, it's been a few years since I've looked at it).

As to your specific cases:

• Remote desktop: The app window tracking only requires an executable file, so that's easily possible. eg. pcoip_client.exe: Remote Desktop
• Browser tab - It's possible but not so ideal. You can filter the executable file with a window title (you can use <*> for a wildcard, v2 will switch to regex). eg. chrome.exe[Play <*> Stadia<*>]: Google Stadia)
• Toolbox inside app - Won't be possible with my current code.

Also, you mentioned you are working on a next version - any blog/changelog/issue-tracker one can follow?

It's not issues holding me back or anything, it's just that my job involves a lot of coding, so when I finish work I don't have the motivation to do any more haha. I may post on reddit if/when it's done, but I plan to upload an executable of it for a Github release too, so I reckon the best way to follow is just click the watch button and forget about it until something pops up :)

Last question: How Python-specific is it - meaning are you using patterns/functions that are exclusive to Python or do you think it can be ported (halfway easily) to a Rust/WebAssembly/Node/Deno-Combo?

Without knowing the other languages, I would say it could be written in them as the concept itself is quite simple. The save files wouldn't be compatible though as they contain numpy arrays. A wiki post on how it works is something I'd like to do, but I'll wait until v2 is more fleshed out.

The captured data, do you draw it directly or is it stored as raw data, so it might even be used for reproduction for example for automated testing of Apps in Virtual Machines and to enable statistical analysis over time?

Even though it's raw data it can't be used this way, I designed it for visualisation and not accuracy. Since it's meant to be run for months/years on end, the older data is squished down when it reaches a certain threshold, and any new data overwrites the old data.

I added a very basic history feature as I thought it'd be interesting watching an animation, but it's only meant to store a few hours, and there's much better solutions for replaying actions anyway.

[claudiusraphael]

Wow, this is quite a complete answer to all my questions. Insightful - and i totally understand the difficulty of doing coding privately on top of doing it for business, so no excuses needed, lol. The details you gave will allow me to follow through and grasp the projects internals, so: much thanks for taking the time!

If you don't mind, i'd like to know what environment are you coding and testing in? I prefer to set up a vm/lxc with the preferred/tested host-OS and IDE used by the developer(s) of a project to get a feel for the thought-process/workflow drizzling into the resulting code. If possible please share those details as a reference-frame and feel free to close this issue.

Thank you!