Windows desktop capture performance

[bp2008]

Hi. First of all, thank you so much for building MeshCentral. I just discovered it today, and I am in awe of what you have accomplished. In 2016, I started a much less ambitious project with remarkably similar goals and design choices, but I never got very far due to a lack of time and motivation. I am thrilled to see so many of my goals fully realized by MeshCentral, and so much more. With such active development too! ❤️❤️❤️

So, I set up a MeshCentral server in an Ubuntu 20.04 VM (which went fine following instructions for 18.04) and installed the agent software on two Windows machines. Something I noticed right away is that the desktop capture performance is (I'm sorry) terrible. To be specific, I mean the "fast" frame rate is quite slow, it makes the local mouse pointer flicker, and it consumes significant CPU time when nothing is changing on screen. Based on earlier issues (#1237, #1208, #357) it seems you are open to improving all of this.

The solution is to use DXGI Desktop Duplication which is an API available in Windows 8 and newer. This API tells you about regions of the screen that have moved and changed in a very efficient manner. The frame rate is limited mainly by your encoding, data streaming, and decoding pipeline. When I used this several years ago, I was able to achieve 60+ FPS when dragging a small window around on a 4K desktop. I would send the moved region information straight to the browser where there are HTML5 canvas operations to easily handle moving image data regions. For the changed regions, I'd encode them with libjpeg-turbo and have the browser decode them and draw them in the appropriate spots on the canvas. This jpeg approach won't handle full screen video amazingly well, but for most desktop usage this approach performs pretty well. To do any better I suspect it would be necessary to use a hardware-accelerated video encoder.

Some caveats about DXGI Desktop Duplication that I recall from when I was messing with it:

• Requires Windows 8 and newer
• It failed when no user was logged in, forcing me to fall back to other capture methods. I'd have my agent program occasionally try to restart the DXGI code so that the full efficiency and frame rate could be restored after a user login.

I'm not sure, maybe I should have posted this in the MeshAgent repo?

[Ylianst]

Bryan has been working on that exact improvement. MeshCentral started a long time ago and we used to support WindowsXP. So, yes, it's going to be nice if we can improve video capture at some point.

[TerAnYu]

Hello! Are you planning a change for Windows? At the moment, I am experiencing a performance problem when connecting to a windows agent, very slowly receiving/transmitting cursor movement. Changing quality from 40% to 5% does not improve the situation.

[krayon007]

Yes it's still on my roadmap. I actually put it on my OKR to get it done in the next few quarters. Which windows version are you running? I know the current method uses a bit of CPU, but you shouldnt be receiving cursor updates slowly...

[TerAnYu]

@krayon007, i use different versions: Winfows 10 Pro 21H2 (19044.2130) Windows 11 Pro - 21H2 (22000.1098)

video: left - web client; right - mesh agent Are on the same network, webrtc is enabled, the server is on the internet.

[bp2008]

It looks like when you moved the cursor on the left machine, it was moving smoothly and its movements were being reflected smoothly on the right machine. The only performance problem I see there is the low frame rate in the web client, which unfortunately is still normal for the Windows mesh agent.

Based on what I remember from looking at the source code 2 years ago, the primary bottleneck was probably the desktop capture method being used. That is why I recommended DXGI duplication above as a different capture method. DXGI is extremely fast at desktop capture. DXGI also tells you about the regions of the desktop that moved or changed, so the mesh agent won't need to waste time looking for changes in every video frame anymore (if it is even bothering to do that, I don't remember).

[TerAnYu]

Thanks for the answer! Yes, the pointer moves smoothly on both sides, even at 64 kbps. The web client settings were set to: Quality - 40% Scaling - 100% Frame rate-Fast

As far as I understand, it is impossible to switch the image capture mode?

[bp2008]

Correct. As far as I know, there only is one mode implemented and it is quite slow.

[TerAnYu]

Have a workaround for the current problem or wait for your solution?