--cg --vg resize: image streched

[jjsarton]

If we don't have the same aspect ratio for the camera and the video output the result is not very pretty. For this case, we can calculate the region we will scale. For example, if --cg is 640x480 and --vg is 480x480, we my calculate the region to be processed and set

cv::Rect crop_region(80, 0,480, 480);

and call

cv::resize(raw(crop_region), raw, cv::Size(vidGeo.value().first,vidGeo.value().second));

instead of

cv::resize(raw raw, cv::Size(vidGeo.value().first,vidGeo.value().second));

This allows to get a non distorted output.

[BenBE]

Can you do a PR to perform the streching/cropping based on a command line switch? TIA.

[jjsarton]

Assume that we call backscrub with the arguments: backscrub -b img.jpg -c /dev/video0 -v /dev/video3 --cg 640x480 --vg 480x480

The raw image (captured by the camera has an aspect ratio 4:3 the output image an aspect ratio of 1:1. Actually, the whole content of the raw image is resized so that the complete image fit to the wanted output size. An Other example we have a hdmi grabber the video size is 1920x1080 (16:9) we want to have an output with 640x480 (4:3). With the present feature, we will get a steched image.

This mean that a circle will, after resizing, will be an oval. The height is. for this case bigger as the width.

If we take into account only a centered area and crop first the raw image, a circle will remain a circle.

[jjsarton]

I have modified deepseg.c as follow:

added after the last variable declaration within main()

    cv::Rect crop_region(0,0,0,0);

Before the mainloop added:

    // calculate crop region, only if result always smaller
    if (expWidth != vidGeo.value().first &&
        vidGeo.value().first <= capGeo.value().first &&
        vidGeo.value().second <= capGeo.value().second
        ) {
            float sc = (float)vidGeo.value().first / capGeo.value().first;
            float st = (float)vidGeo.value().second / capGeo.value().second;
            sc = st > sc ? st : sc;
            int sz = (int)(vidGeo.value().first / sc) - capGeo.value().first;
            crop_region.x =  (sz < 0 ? -sz : sz)/2;
            sz = (int)(vidGeo.value().second / sc) - capGeo.value().second;
            crop_region.y =  (sz < 0 ? -sz : sz)/2;
            crop_region.width = capGeo.value().first - crop_region.x*2;
            crop_region.height = capGeo.value().second - crop_region.y*2;

    }

The code will only be passed if we are scaling down!

Scaling occur with the following:

        // scale to virtual camera geometry (if required)
        if (vidGeo != capGeo) {
            if ( crop_region.width == 0) {
            cv::resize(raw, raw, cv::Size(vidGeo.value().first,vidGeo.value().second));
            } else {
                raw(crop_region).copyTo(raw);
                cv::resize(raw, raw, cv::Size(vidGeo.value().first,vidGeo.value().second));
            }
        }

This seems to work well.

[jjsarton]

I have added cropping of capture and background. In order to do so I modified 4 files. See attached diff files. background.cc.txt deepseg.cc.txt libbackscrub.cc.txt libbackscrub.h.txt

[BenBE]

There are some code style issues with these patches (mostly indentation and tabs/spaces). Also I noticed some typos.

Please put those patches into a PR. This makes working and reviewing those patches much easier. Given that you already got those patches from git, it's not to hard to create and upload a branch containing those changes.

[jjsarton]

Sorry, but my English is not so good. I have submitted a PR (Problem Report or Push Request?). I hope that the space/tabs typos are no more present. Unfortunately, I have not moved the line ti.copyns = timestamp(); to the correct place. I think, that this is not a big issue for you.

[phlash]

Thanks for the effort so far @jjsarton, it's looking good :+1: .

From my understanding of the code, this will always crop both input video and background video to the output aspect ratio, when the output is smaller than the input in both dimensions, otherwise it will scale as before (including any distortions that may produce).

This is opinionated behaviour, which as @BenBE asked (https://github.com/floe/backscrub/issues/149#issuecomment-1229163444) would be better with a command line switch to enable, leaving the existing behaviour (always scaling) as the default.

[jjsarton]

I may use a switch, this is not a big problem. As far I can think about background distortion, this appeared first with the use of different aspect ratio for input and output. Above cited comment apply only on the newest version with the --cg and --vg. Therefor, I can't imagine that there is a lot of people which want a distorted background.

Actually I have a development version and I have some changes within deepscrub.cc. I have put the message (showUsage) within a function, corrected a little argument parsing, tried to fix bugs in front of the display. If a wrong argument is recognized, I print first the usage and then the error message. There are of course other modification which shall avoid crashes.

A lot of tests are to be performed, with many inputs and outputs (camera, background, output size, …). This requires a lot of time.