Slightly confused about capturing mouse icon with screen/window

[alexciarlillo]

What is the most direct route to capturing a screen or window and including the mouse cursor? Based on the given callbacks the solution I have come up with is a little cumbersome:

• in onMouseChanged extract the mouse image into a buffer
• in onNewFrame while extracting the image i also utilize the mouse image buffer from above. if i am extracting a pixel that falls within the bounds of the mouse cursor (calculated using it's point + size) then extract the pixel values from the mouse image instead of the normal image, otherwise take the pixel from the normal image (naive approach)

on the surface this seems reasonable, but it doesn't really work for a few reasons:

• sharing data between the 2 callbacks which run on different threads presents synchronization issues.
• the logic for determining the mouse position/size is brittle and it doesn't appear to take into account monitor scaling
• i can't just blindly take the mouse pixel value, i need to combine it with the underlying captured image pixel depending on the alpha of the mouse pixel

overall this seems like not a very good approach, but I'm not seeing much better alternative. the end goal is simply to extract a single RGBA buffer of the captured screen/window including the mouse cursor. is there a simpler way that I am missing? i cannot rely on image processing libraries or anything to combine them, i need to extract this is as a pretty generic single buffer for use elsewhere (specifically I am using this buffer to draw to an HTML5 canvas).

[smasherprog]

The solution to the problem is actually simple.. You do not send a single combined image, you send the image from onnewframe, and send the mouseimages seperately. You combine them in javascript

[smasherprog]

I have done this before and faced the same problem as you. Its much better to put that logic into the client and let the client (web Browser) decide how to handle this. This way no locking needs to occur in your c++ code...

[smasherprog]

The same approach to how the library works is how you should handle the problem you are facing. The biggest reason is that if you are trying to combine into 1 big image and sent that down to the web browser you are going to send a TON of data (which is bad). This might be simpler, but its not going to be a viable program because of the volume of data.

You only should care what has changed. So when the library callsbacks occur, just forward ALL the data along to the client and let the client drawn them.. including the onFrameChanged (contains areas that have changed).

[alexciarlillo]

I am not really "sending" a lot of data to the browser. I am fortunately able to use a shared Buffer (via node-addon-api) and have a dedicated shared memory region I can read from in JS and write to from C++. That is why just injecting the cursor image into this pre-existing shared memory region seemed ideal, rather than having to set up another such region just for the cursor itself and then correctly map the coordinates into the canvas (again because I am also scaling the final resulting image in C++ to meet streaming video constraints, e.g. scaling down a 4k image to 720p). So again, if I could get the cursor data into the original un-scaled buffer, the rest of the pipeline I have setup will continue to "just work" with minimal necessary changes.

I may opt to go a different route though and fork/patch the library to just draw the cursor directly from the various frame processors. On OSX this was as simple as setting capturesCursor to true and I am working on windows implementations now.

edit: thank you for the quick response and answering my question though! This is a wonderful project.