Color patches after cropping video

[ritesh94]

Android Version - v5 (lollipop API 21) Device - Samsung Galaxy J2

Issue Cropping video while recording using camera2 API outputs color patched video in Samsung Galaxy J2.

I tested in other models (Moto g3 (API 23), Micromax canvas (API 25), Samsung Galaxy A5 (API level 24) and its working correctly.

Gradle dependencies:

compile(group: 'org.bytedeco', name: 'javacv-platform', version: '1.4.1') {
        exclude group: 'org.bytedeco.javacpp-presets'
    }
    compile group: 'org.bytedeco.javacpp-presets', name: 'ffmpeg', version: 3.4.1-1.4
    compile group: 'org.bytedeco.javacpp-presets', name: 'ffmpeg', version: 3.4.1-1.4, classifier: 'android-arm'

ImageReader format: ImageFormat.YUV_420_888 FFmpegFrameFilter format: avutil.AV_PIX_FMT_NV21

Below is the code I am using to process/convert frames using imageReader from YUV_420_888 to NV21 format

public static byte[] imageToByteArray(Image image) {
        byte[] data = null;
        if (image.getFormat() == ImageFormat.JPEG) {
            Image.Plane[] planes = image.getPlanes();
            ByteBuffer buffer = planes[0].getBuffer();
            data = new byte[buffer.capacity()];
            buffer.get(data);
            return data;
        } else if (image.getFormat() == ImageFormat.YUV_420_888) {
            data = YUV_420_888toNV21(image);

                    //NV21toJPEG(YUV_420_888toNV21(image),image.getWidth(), image.getHeight());
        }
        return data;
    }

    private static byte[] YUV_420_888toNV21(Image image) {
        byte[] nv21;
        ByteBuffer yBuffer = image.getPlanes()[0].getBuffer();
        ByteBuffer uBuffer = image.getPlanes()[1].getBuffer();
        ByteBuffer vBuffer = image.getPlanes()[2].getBuffer();

        int ySize = yBuffer.remaining();
        int uSize = uBuffer.remaining();
        int vSize = vBuffer.remaining();

        nv21 = new byte[ySize + uSize + vSize];

        //U and V are swapped
        yBuffer.get(nv21, 0, ySize);
        vBuffer.get(nv21, ySize, vSize);
        uBuffer.get(nv21, ySize + vSize, uSize);

        return nv21;
    }

[saudet]

Try to use FFmpeg to do the format conversion instead of doing it manually.

[ritesh94]

@saudet What format should I mention in FFmpeg -pix_fmt yuv420p or '-pix_fmt NV21' ?

Following are the currently mentioned formats: ImageReader format: ImageFormat.YUV_420_888 FFmpegFrameFilter format: avutil.AV_PIX_FMT_NV21

[saudet]

YUV_420_888 might be equivalent to FFmpeg's AV_PIX_FMT_YUV420P, yes, so start with that. And if that doesn't work, there are a lot of YUV 4:2:0 formats, so try them all: https://github.com/bytedeco/javacpp-presets/blob/master/ffmpeg/src/main/java/org/bytedeco/javacpp/avutil.java#L2714

[ritesh94]

@saudet tried using ffmpeg for conversion but no luck.

ImageReader:

final ByteBuffer buffer = mImage.getPlanes()[0].getBuffer();
                    byte[] bytes = new byte[buffer.remaining()];
                    buffer.get(bytes, 0, bytes.length);*/

                    ((ByteBuffer) yuvImage.image[0].position(0)).put(bytes);

                    long t = 1000 * (System.currentTimeMillis() - startTime);
                    if (t > mFrameRecorder.getTimestamp()) {
                        mFrameRecorder.setTimestamp(t);
                    }

                    mFrameFilter.push(yuvImage);
                    Frame frame2;
                    while ((frame2 = mFrameFilter.pull()) != null) {
                        mFrameRecorder.record(frame2, mFrameFilter.getPixelFormat());
                    }

Setting FFmpegFrameFilter format: mFrameFilter.setPixelFormat(avutil.AV_PIX_FMT_YUV420P);

Setting Imagereder Format: mImageReader = ImageReader.newInstance(mImageSize.getWidth(), mImageSize.getHeight(), ImageFormat.YUV_420_888, 1);

FFmpeg command:

filters.add(String.format(Locale.getDefault(), "crop=w=%d:h=%d:x=0:y=0", imageWidth, imageWidth));
filters.add("format=pix_fmts=yuv420p");
String filtersString = TextUtils.join(",", filters); 

[saudet]

Please try the other pixel formats as well. It might not be AV_PIX_FMT_YUV420P.