Status

[Arnold1]

Hi, nice project. could you tell me what is the status of this project? what already works, what does not work and what will be developed in future?

what is the difference to this method? https://www.codeproject.com/Tips/1116253/Desktop-screen-capture-on-Windows-via-Windows-Desk

[smasherprog]

Everything works, cross platform: win xp and up, linux, macox all monitors or window capturing.

[smasherprog]

The difference is my code is cross platform code....

[Arnold1]

i want to test it on my mac... how to build/run it?

also, could you quickly explain how you record a frame every 33ms (for 30hz)? how stable is the recording rate?

[smasherprog]

you can build the example code to see how it works. You just need cmake

[Arnold1]

could you quickly explain how you record a frame every 33ms (for 30hz)? how stable is the framerate? can you also record with 60hz?

[smasherprog]

Its all in the example code https://github.com/smasherprog/screen_capture_lite/blob/master/Example/Screen_Capture_Example.cpp

[smasherprog]

just run it and put some breakpoints in it

[Arnold1]

i mean explain on a higher level than reading code...

[Arnold1]

can you record with a framerate of 33ms (=30Hz) for 10 min without a pause and store it on the disk in parallel?

[smasherprog]

absolutely

[Arnold1]

great! i run your Example. How to store the recording to the disk in parallel?

i see you have that in the example, isnt it to slow to store each frame to the disk?

 // auto imgbuffer(std::make_unique<unsigned char[]>(size));
                // ExtractAndConvertToRGBA(img, imgbuffer.get(), size);
                // tje_encode_to_file(s.c_str(), Width(img), Height(img), 4, (const unsigned char*)imgbuffer.get());

[smasherprog]

i provide code in the example project for how to save as a jpeg and png files for each frame. Tha code is slow and beyond the scope of this library. The only reason for this library is to capture RAW data of the monitor or windows as fast as possible and leave the rest up to you.

[Arnold1]

i tested using the following. i noticed that it not always records the same size of the screen? is that expected?

int main()
{
    std::cout << "Starting Capture Demo/Test" << std::endl;
    std::cout << "Testing captured monitor bounds check" << std::endl;
    auto goodmonitors = SL::Screen_Capture::GetMonitors();
    for (auto &m : goodmonitors) {
        std::cout << m << std::endl;
        assert(isMonitorInsideBounds(goodmonitors, m));
    }
    auto badmonitors = SL::Screen_Capture::GetMonitors();

    for (auto m : badmonitors) {
        m.Height += 1;
        std::cout << m << std::endl;
        assert(!isMonitorInsideBounds(goodmonitors, m));
    }
    for (auto m : badmonitors) {
        m.Width += 1;
        std::cout << m << std::endl;
        assert(!isMonitorInsideBounds(goodmonitors, m));
    }
    std::cout << "Changing the cpature rate to 33 milli second" << std::endl;
    createframegrabber();
    framgrabber->setFrameChangeInterval(std::chrono::milliseconds(33));
    std::this_thread::sleep_for(std::chrono::seconds(10));
}

void createframegrabber()
{
    realcounter = 0;
    onNewFramecounter = 0;
    framgrabber =
        SL::Screen_Capture::CreateCaptureConfiguration([]() {
            auto mons = SL::Screen_Capture::GetMonitors();
            std::cout << "Library is requesting the list of monitors to capture!" << std::endl;
            for (auto &m : mons) {
                // capture just a 512x512 square...  USERS SHOULD MAKE SURE bounds are
                // valid!!!!
                /*
                            m.OffsetX += 512;
                            m.OffsetY += 512;
                            m.Height = 512;
                            m.Width = 512;
                */
                std::cout << m << std::endl;
            }
            return mons;
        })
            ->onFrameChanged([&](const SL::Screen_Capture::Image &img, const SL::Screen_Capture::Monitor &monitor) {
                // std::cout << "Difference detected!  " << img.Bounds << std::endl;
                auto r = realcounter.fetch_add(1);
                auto s = std::to_string(r) + std::string("MONITORDIF_") + std::string(".jpg");
                auto size = RowStride(img) * Height(img);

                auto imgbuffer(std::make_unique<unsigned char[]>(size));
                ExtractAndConvertToRGBA(img, imgbuffer.get(), size);
                tje_encode_to_file(("recording/" + s).c_str(), Width(img), Height(img), 4, (const unsigned char*)imgbuffer.get());
            })
            ->onNewFrame([&](const SL::Screen_Capture::Image &img, const SL::Screen_Capture::Monitor &monitor) {

                auto r = realcounter.fetch_add(1);
                auto s = std::to_string(r) + std::string("MONITORNEW_") + std::string(".jpg");

                auto size = RowStride(img) * Height(img);
                // auto imgbuffer(std::make_unique<unsigned char[]>(size));
                // ExtractAndConvertToRGBA(img, imgbuffer.get(), size);
                // tje_encode_to_file(s.c_str(), Width(img), Height(img), 4, (const unsigned char*)imgbuffer.get());
                if (std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::high_resolution_clock::now() - onNewFramestart).count() >=
                    1000) {
                    std::cout << "onNewFrame fps" << onNewFramecounter << std::endl;
                    onNewFramecounter = 0;
                    onNewFramestart = std::chrono::high_resolution_clock::now();
                }
                onNewFramecounter += 1;
            })
            ->onMouseChanged([&](const SL::Screen_Capture::Image *img, const SL::Screen_Capture::Point &point) {

                auto r = realcounter.fetch_add(1);
                auto s = std::to_string(r) + std::string(" M") + std::string(".png");
                if (img) {
                    // std::cout << "New mouse coordinates  AND NEW Image received." << " x= " << point.x << " y= " <<
                    // point.y << std::endl;
                    // lodepng::encode(s,StartSrc(*img), Width(*img), Height(*img));
                }
                else {
                    // std::cout << "New mouse coordinates received." << " x= " << point.x << " y= " << point.y << " The
                    // mouse image is still the same
                    // as the last" << std::endl;
                }

            })
            ->start_capturing();

    framgrabber->setFrameChangeInterval(std::chrono::milliseconds(100));
    framgrabber->setMouseChangeInterval(std::chrono::milliseconds(100));
}

here the output:

./screen_capture_example 
Starting Capture Demo/Test
Testing captured monitor bounds check
Id=2077750397 Index=0 Height=1800 Width=2880 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
Id=2077750397 Index=0 Height=1801 Width=2880 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
Id=2077750397 Index=0 Height=1800 Width=2881 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
Changing the cpature rate to 33 milli second
Library is requesting the list of monitors to capture!
Id=2077750397 Index=0 Height=1800 Width=2880 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
onNewFrame fps2
onNewFrame fps12
onNewFrame fps12
onNewFrame fps13
onNewFrame fps13
onNewFrame fps13
onNewFrame fps13
onNewFrame fps13
onNewFrame fps12

size frame 1: 2880 × 1800 size frame 2: 1280 × 1024 size frame 3: 1280 × 768 size frame 4: 512 × 256 ...

also why is the frame rate 14-15 fps for that setting? in this test i removed the part which stores each frame to disk:

./screen_capture_example 
Starting Capture Demo/Test
Testing captured monitor bounds check
Id=2077750397 Index=0 Height=1800 Width=2880 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
Id=2077750397 Index=0 Height=1801 Width=2880 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
Id=2077750397 Index=0 Height=1800 Width=2881 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
Changing the cpature rate to 33 milli second
Library is requesting the list of monitors to capture!
Id=2077750397 Index=0 Height=1800 Width=2880 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
onNewFrame fps14
onNewFrame fps14
onNewFrame fps15
onNewFrame fps15
onNewFrame fps14
onNewFrame fps14
onNewFrame fps14
onNewFrame fps15
onNewFrame fps14

hardware/sw info:

$ uname -a
Darwin MM-MAC-3270 15.6.0 Darwin Kernel Version 15.6.0: Tue Apr 11 16:00:51 PDT 2017; root:xnu-3248.60.11.5.3~1/RELEASE_X86_64 x86_64

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.11.6
BuildVersion:   15G1510

$ system_profiler SPHardwareDataType
Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: MacBookPro11,5
      Processor Name: Intel Core i7
      Processor Speed: 2.5 GHz
      Number of Processors: 1
      Total Number of Cores: 4
      L2 Cache (per Core): 256 KB
      L3 Cache: 6 MB
      Memory: 16 GB

[smasherprog]

onNewFrame will give you the entire monitor each time it is called onFrameChanged will give you the change in the monitor.

[smasherprog]

So you can get the first entire monitor capture, then use the onframechanged to get only the differences afterwards. This will lessen the amount of data you are capturing.

[Arnold1]

ok i see. but why is the fps only around 15 (for a 33ms timer)? even when i dont store to disk?

[smasherprog]

if you are using the above code snippet, you are saving the onframechanged images to disk. Try commenting that code out. Also, check the folder where the executable is, I bet you have alot of images there :)

[Arnold1]

my results for the following code. any idea why it only shows 15 fps?

./screen_capture_example 
Starting Capture Demo/Test
Testing captured monitor bounds check
Id=2077750397 Index=0 Height=1800 Width=2880 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
Id=2077750397 Index=0 Height=1801 Width=2880 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
Id=2077750397 Index=0 Height=1800 Width=2881 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
Changing the cpature rate to 33 milli second
Library is requesting the list of monitors to capture!
Id=2077750397 Index=0 Height=1800 Width=2880 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
onNewFrame fps14
onNewFrame fps14
onNewFrame fps15
onNewFrame fps15
onNewFrame fps14
onNewFrame fps14
onNewFrame fps14
onNewFrame fps15
onNewFrame fps14

void createframegrabber()
{
    realcounter = 0;
    onNewFramecounter = 0;
    framgrabber =
        SL::Screen_Capture::CreateCaptureConfiguration([]() {
            auto mons = SL::Screen_Capture::GetMonitors();
            std::cout << "Library is requesting the list of monitors to capture!" << std::endl;
            for (auto &m : mons) {
                // capture just a 512x512 square...  USERS SHOULD MAKE SURE bounds are
                // valid!!!!
                /*
                            m.OffsetX += 512;
                            m.OffsetY += 512;
                            m.Height = 512;
                            m.Width = 512;
                */
                std::cout << m << std::endl;
            }
            return mons;
        })
            ->onFrameChanged([&](const SL::Screen_Capture::Image &img, const SL::Screen_Capture::Monitor &monitor) {
                // std::cout << "Difference detected!  " << img.Bounds << std::endl;
                auto r = realcounter.fetch_add(1);
                auto s = std::to_string(r) + std::string("MONITORDIF_") + std::string(".jpg");
                auto size = RowStride(img) * Height(img);
            })
            ->onNewFrame([&](const SL::Screen_Capture::Image &img, const SL::Screen_Capture::Monitor &monitor) {

                auto r = realcounter.fetch_add(1);
                auto s = std::to_string(r) + std::string("MONITORNEW_") + std::string(".jpg");

                auto size = RowStride(img) * Height(img);
                // auto imgbuffer(std::make_unique<unsigned char[]>(size));
                // ExtractAndConvertToRGBA(img, imgbuffer.get(), size);
                // tje_encode_to_file(s.c_str(), Width(img), Height(img), 4, (const unsigned char*)imgbuffer.get());
                if (std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::high_resolution_clock::now() - onNewFramestart).count() >=
                    1000) {
                    std::cout << "onNewFrame fps" << onNewFramecounter << std::endl;
                    onNewFramecounter = 0;
                    onNewFramestart = std::chrono::high_resolution_clock::now();
                }
                onNewFramecounter += 1;
            })
            ->onMouseChanged([&](const SL::Screen_Capture::Image *img, const SL::Screen_Capture::Point &point) {

                auto r = realcounter.fetch_add(1);
                auto s = std::to_string(r) + std::string(" M") + std::string(".png");
                if (img) {
                    // std::cout << "New mouse coordinates  AND NEW Image received." << " x= " << point.x << " y= " <<
                    // point.y << std::endl;
                    // lodepng::encode(s,StartSrc(*img), Width(*img), Height(*img));
                }
                else {
                    // std::cout << "New mouse coordinates received." << " x= " << point.x << " y= " << point.y << " The
                    // mouse image is still the same
                    // as the last" << std::endl;
                }

            })
            ->start_capturing();

    framgrabber->setFrameChangeInterval(std::chrono::milliseconds(100));
    framgrabber->setMouseChangeInterval(std::chrono::milliseconds(100));
}

[smasherprog]

There is some jitter that I am noticing with the library that ill take a look into right now

[Arnold1]

are you able to test for mac osx? let me know if i can help you testing stuff...

[smasherprog]

I just tested on my mac and turns out there is no problem. The jitter I was experiencing on my windows machine is normal. On my mac I capture at about 24-29 fps. Try only defining the OnNewFrame Callback, not the changed callback.

[smasherprog]

Let me know if you find a bug... closing issue

[Arnold1]

@smasherprog i want to move the img to a thread for further processing:

here the example: https://gist.github.com/Arnold1/21d8df4d660cfd6e7e964d6217f94054

any idea what could be wrong? running into: Segmentation fault: 11

can i just move SL::Screen_Capture::Image &img to the queue? the queue is a safe queue (protected by a mutex + cond variable)

it works fine when onFrameChanged is not commented out... but as you said onFrameChanged shows down the fps...

[smasherprog]

The library owns the data, so you have to copy it out if you want to do anything with it. if you attempt to read or write to the data after the call to OnNewFrame or OnFrameChanged this is a race condition and anything can happen.

[smasherprog]

breaking up the work for finding the differences between frames will be slower if you try to do it in a different thread. The cost of the data copy + allocation is greater than the cost you are payying when defining the OnFrameChanged callback.

The cost is around 1-4 ms per callback for the onframechanged on my 4k monitor.

[Arnold1]

oh ok. how to copy it most efficiently and pass it to the queue?

[smasherprog]

You can look at these EXTRACT functions in the library or write your own https://github.com/smasherprog/screen_capture_lite/blob/master/src/ScreenCapture.cpp#L15

[smasherprog]

if you want to look at the mac capturing code, it could use some improvements im sure https://github.com/smasherprog/screen_capture_lite/blob/master/src/ios/CGFrameProcessor.cpp

[Arnold1]

i dont try to find the diff between frames, i just want to save them on disk with 30fps...

if you say the overhead to copy is 1-4ms. how to best move the raw image to the thread for processing (e.g. save to disk)? allocate a big ring buffer, the capture code writes to a ring buffer and the thread reads from the ring buffer?

i figured out that the overhead to copy the data is much bigger: around 20ms:

auto start = std::chrono::high_resolution_clock::now();

auto imgbuffer(std::make_unique<unsigned char[]>(size));
SL::Screen_Capture::Extract(img, imgbuffer.get(), size);
queue.enqueue(new data(size, Width(img), Height(img), std::move(imgbuffer)));

std::cout << std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::high_resolution_clock::now() - start).count() << std::endl;

[smasherprog]

Yeah allocating large buffers can be costly. If you reuse the memory that will help. The actual copy operation is as fast as its going to get with the extract function: it uses memcpy if possible, otherwise it will do a row by row memcpy, which should be fast as well.

You should see how much time your enque function takes.

[smasherprog]

also make sure you are comparing release code -O2 at least in your timings

[Arnold1]

ok i added -O2:

here the measuremets:

auto imgbuffer(std::make_unique<unsigned char[]>(size)); takes (ms): 10, 3, 9, 8, 8

SL::Screen_Capture::Extract(img, imgbuffer.get(), size); takes (ms): 11, 11, 11, 11, 11

queue.enqueue(new data(size, Width(img), Height(img), std::move(imgbuffer))); takes (ms): 0, 0, 0, 0, 0

how to reuse buffer when the thread takes the buffer?

with cmake -D CMAKE_CXX_FLAGS="-O2" . i get the following results:

$ ./screen_capture_example 
Starting Capture Demo/Test
Testing captured monitor bounds check
Id=2077750397 Index=0 Height=1800 Width=2880 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
Id=2077750397 Index=0 Height=1801 Width=2880 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
Id=2077750397 Index=0 Height=1800 Width=2881 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
Changing the cpature rate to 33 milli second
Library is requesting the list of monitors to capture!
Id=2077750397 Index=0 Height=1800 Width=2880 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
onNewFrame fps29
onNewFrame fps30
onNewFrame fps30
onNewFrame fps30
onNewFrame fps30
onNewFrame fps30
onNewFrame fps30
onNewFrame fps29
onNewFrame fps30

[smasherprog]

There are no good options. . . .

[Arnold1]

sorry i have to correct myself. here are the results:

$ ./screen_capture_example 
Starting Capture Demo/Test
Testing captured monitor bounds check
Id=2077750397 Index=0 Height=1800 Width=2880 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
Id=2077750397 Index=0 Height=1801 Width=2880 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
Id=2077750397 Index=0 Height=1800 Width=2881 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
Changing the cpature rate to 33 milli second
Library is requesting the list of monitors to capture!
Id=2077750397 Index=0 Height=1800 Width=2880 OffsetX=0 OffsetY=0 Name=Monitor 2077750397
onNewFrame fps23
onNewFrame fps24
onNewFrame fps24
onNewFrame fps24
onNewFrame fps24
onNewFrame fps24
onNewFrame fps23
onNewFrame fps24
onNewFrame fps22

what do you mean by there are no good options?

the 11ms to allocate a buffer is really high...i wonder how to lower it and reuse the buffer between the callback and thread....

[smasherprog]

processing/copy/saving this amount of data is hard to do and you just have to find the best solution that works for you.

[smasherprog]

macs have their refresh rate set to 30hz , which explains why you are capturing at that rate

[Arnold1]

in best case i would get fps30 when you look here?

onNewFrame fps23
onNewFrame fps24
onNewFrame fps24
onNewFrame fps24
onNewFrame fps24
onNewFrame fps24
onNewFrame fps23
onNewFrame fps24
onNewFrame fps22

[Arnold1]

what pixel format does Extract return? https://github.com/smasherprog/screen_capture_lite/blob/master/src/ScreenCapture.cpp#L15

[smasherprog]

Extract returns the raw data which is BGRA.. Alpha is always unused and may contain garbage.

[Arnold1]

if you are interested i modified my code to be more performant, i can output a video file: https://gist.github.com/Arnold1/57508cf245cf77be458049ca3fdf10d4

i implemented my own SL::Screen_Capture::ExtractAndConvertToBGR which seems slower than SL::Screen_Capture::Extract. when i see the code if (RowPadding(img) == 0) ... than it only needs 1 memcpy, which is super fast?

is there a way to remove A from Extract? or is there a way to remove it while capturing it?

[smasherprog]

When capturing the desktop across all three platforms: win, linux and mac they all naively seem to use BGRA for performance reasons. So, i capture the data as they do for the same reason. Therefore, alpha cannot be removed.

[smasherprog]

Additionally, I see you are saving as a bmp. You can save the alpha to disk as unused data. The file size is larger, but this is faster.

[Arnold1]

does it capture A on osx? which line do you refer to?

i want to use Extract but you said A is wrong... maybe i should remove the A in the write thread...hm

[smasherprog]

this is the function that does the capturing for mac https://github.com/smasherprog/screen_capture_lite/blob/master/src/ios/CGFrameProcessor.cpp#L22

[Arnold1]

"Additionally, I see you are saving as a bmp. You can save the alpha to disk as unused data. The file size is larger, but this is faster." which line do you refer to?

when i use SL::Screen_Capture::Extract and remove the alpha channel later on in the write thread, that should be ok?

[smasherprog]

https://gist.github.com/Arnold1/57508cf245cf77be458049ca3fdf10d4#file-main-cpp-L137

Your not using that.. never mind.

[Arnold1]

want to use SL::Screen_Capture::Extract (because its super fast) but you said A is wrong... could i remove A or set it to some value inside the write thread?

[Arnold1]

any idea why this might not work? you said Extract returns BGRA so it should be of size Width*Height*4 ?

unsigned char* buffer = d.img.get(); // contains data from SL::Screen_Capture::Extract

for(int i = 0; i < d.Height; i++) {
    for(int j = 0; j < d.Width; j++) {
        buffer[i*d.Width + j + 3] = 255;
    }
}

out_capture.write(cv::Mat(d.Height, d.Width, CV_8UC4, buffer));

[smasherprog]

You can read this for why https://www.gamedev.net/forums/topic/494153-rgb-is-actually-bgr/ try

buffer[i*d.Width + j] = 255;

[Arnold1]

i see you also have row padding... thats why?