search

Ylianst / MeshAgent

MeshAgent used along with MeshCentral to remotely manage computers. Many variations of the background management agent are included as binaries in the MeshCentral project.
https://meshcentral.com
223 stars 87 forks source link

Capture hardware accelerated fullscreen applications #20

Open marco-vogt opened 5 years ago

marco-vogt commented 5 years ago

Hey,

i know this is not a trivial task but have you guys thought about trying to capture fullscreen opengl/vulkan/directx applications?

For our use case, on most remote machines there is a opengl fullscreen application running pretty much 24/7. When connecting to the machines via remote control, we always only see the desktop "underneath" the opengl application which causes a lot of confusion and is not very pleasent to work with.

Ive looked around a bit on how things like that could be implemented and i found out that its not very easy to implement but its possible. You basically inject your code into the graphics api dll of the running application and intercept certain function calls and capture the buffer at the right time.

This is how teamviewer and several other remote control softwares are doing it. (also Fraps and OBS work like that)

Like i said, i know this is not something that can be done in a couple of days but i was just wondering if thats something you thought about implementing into MeshCentral?

Cheers

krayon007 commented 5 years ago

Not sure about windows, but in Linux TeamViewer installs a custom video driver to intercept the video buffer, which is something we explicitly are trying to avoid having to do.

A colleague of mine write some code to do remote screen capture on Linux that didn't require X but I found that required specific graphics hardware and was incompatible with how graphics was virtualized on VMs. This sounds similar to what you are talking about, as his code had to intercept the buffer in the same way after the OS wrote to the graphics hardware.

It could be worth investigating again, but I'm not sure this is possible without relying on a custom driver to fetch the buffer after the GPU pipeline.

jsastriawan commented 5 years ago

I have tried to write framebuffer scrapper using libdrm for Linux and it works well on driver that support framebuffer readback (i915) but not on vmware svga.

https://github.com/jsastriawan/MeshAgent

I have not look at it for quite sometime. Feel free to continue working on it.

Regards,

Joko

marco-vogt commented 5 years ago

I have no idea how it works under linux...i just described the windows way of achieving this.

I think it would be rather straight forward on windows. Once one graphics api is successfully implemented, the others should be no problem. but its not a thing to make in a couple of days, sure.

But anyway, I just wanted to know if this is something you guys are planning to implement but as you said you want to avoid doing this, its fine.

If youre curious how this is done in windows, this should give you an idea. https://github.com/bakwc/LibQtScreen

marco-vogt commented 4 years ago

If youre still interested, i just found out about the Windows Desktop Duplication API. Its pretty easy to implement but it only supports windows 8 and higher.