for FrameIngest module

[Alvazz]

Hi, First all thank for your sharing. I tried to use the FrameIngest module in your library to convert frames in OBS to various forms of YUV, but I found that the results seemed a little unexpected. I will provide a comparison in the form of pictures, which is easier for you to understand.

The original image:

final picture:

code block:

           cv::Mat img = im::read(“path/test.png”);
           cv::UMat outputUmat;
       img.copyTo(outputUmat);
           lvk::export_yuv(outputUmat, frame);

[Crowsinc]

It seems you are passing a BGR UMat to a function that expects a YUV UMat, which is why you get the weird colors. The OBS frame can have any format, but the imported/exported UMat is always in YUV. Converting the outputUmat to YUV before exporting it should fix the issue. Please let me know how it goes!

[Alvazz]

It seems to understand what you mean, but there is a problem, I need to alphablend the foreground and background, and when alphablend is passed to img, the format of this img needs to be in Mat BGR format. The code snippet is as follows, so this alphablend code As far as it is concerned, it seems that you can't convert img to yuv

void alphaBlend(cv::Mat& foreground, cv::Mat& background, cv::Mat& alpha, cv::Mat& outImage)
{
    //Find number of pixels. 
    int numberOfPixels = foreground.rows * foreground.cols * foreground.channels();

    //Get floating point pointers to the data matrices
    float* fptr = reinterpret_cast<float*>(foreground.data);
    float* bptr = reinterpret_cast<float*>(background.data);
    float* aptr = reinterpret_cast<float*>(alpha.data);
    float* outImagePtr = reinterpret_cast<float*>(outImage.data);

    //Loop over all pixesl ONCE
    for (
        int i = 0;
        i < numberOfPixels;
        i++, outImagePtr++, fptr++, aptr++, bptr++
        )
    {
        *outImagePtr = (*fptr) * (*aptr) + (*bptr) * (1 - *aptr);
    }
}

[Alvazz]

If I put the cv::Mat& outImage after alphablend out as follows:

                cv::UMat outputUmat;
        outImage.copyTo(outputUmat);

        cv::cvtColor(outputUmat, outputUmat, cv::COLOR_BGR2YUV);

The background is normal, but the foreground seems wrong again.

[Alvazz]

                cv::UMat outputUmat;
        outImage.copyTo(outputUmat);

        cv::cvtColor(outputUmat, outputUmat, cv::COLOR_BGR2YUV);

        lvk::export_yuv(outputUmat, frame);

[Alvazz]

An additional point:

void alphaBlend(cv::Mat& foreground, cv::Mat& background, cv::Mat& alpha, cv::Mat& outImage)

The cv::Mat& foregroundparameter properties passed in by this alphaBlend are: // fore ground mat 3 channel (R,G,B) 0.~1. or 0~255

foreground.convertTo(foreground, CV_32FC3);

The cv::Mat& alphaparameter properties passed in by this alphaBlend are: // alpha(matte) 0.~1.

alpha= convertTo3Channels(alpha);

cv::Mat convertTo3Channels(const cv::Mat& binImg)
{
    cv::Mat three_channel = cv::Mat::zeros(binImg.rows, binImg.cols, CV_32FC3);
    std::vector<cv::Mat> channels;
    for (int i = 0; i < 3; i++)
    {
        channels.push_back(binImg);
    }
    merge(channels, three_channel);
    return three_channel;
}

[Alvazz]

Looking forward to your professional answer

[Alvazz]

                 void alphaBlend(cv::Mat& foreground, cv::Mat& background, cv::Mat& alpha, cv::Mat& outImage)

The format of cv::Mat &outImage finally returned by this function method is : CV_32FC3

So I did one more step:

                outImage.convertTo(ouImage, CV_8UC3);

Then

                cv::UMat outputUmat;
        outImage.copyTo(outputUmat);

        cv::cvtColor(outputUmat, outputUmat, cv::COLOR_BGR2YUV);

        lvk::export_yuv(outputUmat, frame);

[Crowsinc]

Hi. Many webcams send their video in compressed YUV formats. Have you validated that the foreground is actually in BGR, and that the image you are passing into the YUV conversion and the extract_yuv function is actually what you expect from your code? You could check by saving the output image to a file or using imshow, although the latter can problems with OBS.

I should also mention that nothing so far has indicated an issue with the frame ingest or LiveVisionKit.

[Alvazz]

You are right! Thanks.