Open totaam opened 11 years ago
Screenscraping may be the only feasible way to get 3D apps to work: Since the hardware-accelerated 3D rendering commands typically bypass the GDI layer, most screen scrapers (WinVNC, LiveMeeting, GoToMeeting, etc.) cannot pick up the 3D rendering area of Windows applications, or they will not detect when the 3D area has been updated.
(from DRC @virtualgl-users)
Generic Window API links:
Window Event Hooks:
WH_SHELL
(The function receives notifications of Shell events from the system.): ShellProc callback function, in particular:HSHELL_REDRAW
HSHELL_WINDOWACTIVATED
, HSHELL_WINDOWCREATED
, HSHELL_WINDOWDESTROYED
and HSHELL_WINDOWREPLACED
, which we could tie with the list of rectangles if we keep the hwnd at hand.WH_CBT
(The system calls this function before activating, creating, destroying, minimizing, maximizing, moving, or sizing a window):Example code:
r10002 uses Pillow's ImageGrab module to capture the screen on win32, which is already much faster.
It is still quite slow: timeit clocks it at about 100ms for 1080p on a mid-range system, which is a lot better than GTK, but has room for improvement:
tobytes
- which makes a copy, we could avoid this by going pure native via pywin32 and managing the target buffer's lifetime ourselves (and ideally, straight into nvenc's input buffer - tall order!)BGRX
format so that we can use nvenc... (we could use whatever RGB format is native to win32 and just add the necessary kernel to nvenc instead, see #558)Some links:
The code in r10002 only captured one screen, this is fixed in r10026, see #637#comment:11 for details. Will backport.
r10028 should help speed things up a bit by re-using the same bitmap and context objects until the display settings change. But this is still not good enough: it takes about 100ms for copying 4 megapixels (2 x 1080p screens). About 65% of this time is spent in BitBlt, and the rest is in GetBitmapBits. The BitBlt performance with Aero enabled is just poor. Disabling aero is a temporary workaround.
We want to stay on the GPU if we can (#365), especially if we're going to use the GPU for encoding (ie: nvenc) - the difference is in the order of a factor 100!
More info:
This will have to do for now. (nvenc needs fixing first anyway... see #389)
r11482 added a basic systray for the shadow server which only shows: "about", "exit" and "close menu".
namedpipes.patch
(12.7 KiB)work in progress named pipes support
Moving the named pipes to #1150.
Keyboard issues in #1099.
Stumbled upon this info python-win32: Global Window Messages: What you probably want is a WH_CBT hook so that you can catch WM_ACTIVATE and WM_DEACTIVATE. There is some code that does something similar in pyAA apparently.
See my original question to python-win32 mailing list: https://mail.python.org/pipermail/python-win32/2009-September/009584.html And Win32 WH_CBT Hook - close window before it is created, WH_CBT hook, using SetWindowsHookEx to monitor windows events, KeyboardHookProc in DLL doesn't do anything when called from python
This is going to take some effort and will require Windows 8 or later and probably building things with MSVC 2013 (not the "free" express version..) for the newer API: IDXGIOutputDuplication: The IDXGIOutputDuplication interface accesses and manipulates the duplicated desktop image.
Printing may be possible using redmon: The RedMon port monitor redirects a special printer port to a program
For sound see: wasapisink
r12394 will use directsoundsrc
by default for win32 shadow servers
- we can disable compositing by calling: DwmEnableComposition. This will fix screen capture on Vista and later.
- maybe we should switch to using GetDIBits instead of GetBitmapBits, as per: Desktopmagic screengrab_win32 (ctypes), GetDIBits to PIL image example
- good directx example: Doing readback from Direct3D textures and surfaces, this is probably the right way of hooking up NVENC #558
- good IDXGIOutput1::DuplicateOutput example: RPI-GPU-rdpClient wddm.h
- ffmpeg option: gdigrab, directshow
A good solution for solving shadow performance issues is #1317 (requires an nvidia card...)
If we go down the directx route, with mingw #678 these may be helpful / needed:
g++ dxtest.cpp -o dxtest.exe -I/dxsdk/include -L/dxsdk/lib -DUNICODE -ld3d9
SetWindowsHookEx
.GetFrontBufferData
...So, by injecting the DLL into every process, we could get individual windows as surfaces.
If we are willing to require windows 8 or later, the solution is likely to be the Desktop Duplication API:
Some links on the logon stuff, which we need for running as a service:
For audio:
Added basic NVIDIA Capture SDK support using basic "copy to system memory" API. Screen capturing + downloading to system RAM at 4K takes less than 30ms, so we should be able to do 30fps at 4K (given enough bandwidth / fast encoder).
Much more can be done with this API:
Some scripts that could be useful - and not just for shadow servers:
from comtypes.client import CreateObject
o=CreateObject('WbemScripting.SWbemLocator')
s=o.ConnectServer('.', 'root\\cimv2')
res=s.ExecQuery('SELECT * FROM Win32_DisplayConfiguration')
print([int(x.Properties_['BitsPerPel'].Value) for x in res])
Could also be useful for detecting virtualbox:
print([(x.Properties_['DeviceName'].Value) for x in res])
[u'VirtualBox Graphics Adapter']
Now for making a service:
Moving the system service to #1527.
Python3 #1568 is making progress, this may help.
For capturing the screen with directx, a good example is found in the gstreamer source: winscreencap.
See also multi-monitor handling as windows: #1805
NVFBC is now deprecated on windows-10: #1317#comment:11.
DDA example: nvEncDXGIOutputDuplicationSample
Links:
See also:
Windows 10 Screen capture example code looks simple enough. Using Windows.Graphics.Capture from a Cython C / C++ context will be harder.
The current shadow server for win32 sort of works but we need to improve: