alex-ong / NESTrisOCR

OCR for statistics in NESTris
24 stars 7 forks source link

Upgrade to single-screenshot with multi-thread splitting source Image #13

Closed alex-ong closed 5 years ago

alex-ong commented 5 years ago

Currently: Thread 1-n: Take screenshot of part of screen, process score etc.

This can result in a new frame being generated between the threads, e.g. Score changes but Lines doesn't.

New:

Thread 1: Take screenshot every 1/60
Threads 2-n: Take relevant slice for score etc.

Currently MacOS does this internally by the screenshot class providing sub-images.

We can write a wrapper around the Windows screenshot class to achieve a similar image, and add an API call nextScreenshot() to go to the next screenshot.

alex-ong commented 5 years ago

After this change, we could attempt to use D3DScreenshot API, as well as using multiprocessing shared memory to reduce pickling costs in python 3.8

python 3.8: multiprocessing can now use shared memory segments to avoid pickling costs between processes

D3DScreenshot (requires 64 bit windows): https://github.com/SerpentAI/D3DShot

This would mean:

Thread 1: Take screenshots and put them in a buffer.  Change screenshot in buffer every time NextScreenshot() is called.

Thread 2-n: Use SharedMemorySegment to get sub arrays out.

One side effect of using D3DScreenshot is Win64 support only, which would mean making a 32bit vs 64 bit executable / headaches. Performance needs to be tested first.

alex-ong commented 5 years ago

Did some testing on D3DScreenshot. It is not faster than BitBlt unless you are capturing 1920x1080+

img

Res min perceptual avg max
d3d 600x600, on 1920x1080 0.015 0.015 0.016
bitblt 600x600, on 1920x1080 0.002 0.002 0.005
d3d 1920x1080 0.010 0.015 0.016
bitblt 1920x1080 0.013 0.015 0.018
d3d 3840x2160 0.014 0.030 0.100
bitblt 3840x2160 0.055 0.060 0.300

Since the normal use case is a window size around 600x600 (1/4 of the screen at 1920x1080), bitblt seems to be the clear winner for now.

alex-ong commented 5 years ago

Yobi9 has implemented single capture then split.