Green screen when recording encoded bytes(Mediacodec.Callback)

[1goresha]

Hi every one. I use android Camera2 API and i try to record encoded bytes received in Mediacodec.Callback :

inner class EncoderCallback : MediaCodec.Callback() { @SuppressLint("RestrictedApi") override fun onInputBufferAvailable(codec: MediaCodec, index: Int) { Log.i(LOG_TAG, "_input_buffer_isready") }

    @SuppressLint("RestrictedApi", "SuspiciousIndentation")
    override fun onOutputBufferAvailable(
        codec: MediaCodec,
        index: Int,
        info: MediaCodec.BufferInfo
    ) {
        outPutByteBuffer = encoder?.getOutputBuffer(index)

        val outDate = ByteArray(info.size)
        outPutByteBuffer?.get(outDate)

        if (isFfmpegRecorderStarted) {
            val frame = Frame(1920, 1080, Frame.DEPTH_UBYTE, 2)
            if (info.size != 0) {
                frame.type = Frame.Type.VIDEO
                frame.data = outPutByteBuffer
                ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_NV21)
            }
        }
        encoder?.releaseOutputBuffer(index, false)
    }

    @SuppressLint("RestrictedApi")
    override fun onError(codec: MediaCodec, e: MediaCodec.CodecException) {
        Log.i(LOG_TAG, "Error: $e")
    }

    override fun onOutputFormatChanged(codec: MediaCodec, format: MediaFormat) {
        //  Log.i(LOG_TAG, "encoder output format changed: " + format);
    }
}

And initialization FFmpegFrameRecorder and start it is here:

fun startRec() { if (isRecording.compareAndSet(false, true)) {

                mCurrentFile = StorageUtils.createPhoneVideoFileName(context) + ".avi"

                FFmpegLogCallback.setLevel(AV_LOG_TRACE)
                FFmpegLogCallback.set()

                startTime = System.currentTimeMillis()
                ffmpegRecorder = FFmpegFrameRecorder(
                    mCurrentFile,
                    1920,
                    1080
                )
                ffmpegRecorder!!.frameRate = 30.0
                ffmpegRecorder!!.videoCodec = avcodec.AV_CODEC_ID_MPEG4
                ffmpegRecorder!!.format = "avi"
                ffmpegRecorder!!.pixelFormat = avutil.AV_PIX_FMT_YUV420P
                ffmpegRecorder!!.videoBitrate = 6000
                ffmpegRecorder!!.start()
                isFfmpegRecorderStarted = true
    }
}

It records green screen with some white horizontal lines at top of screen, can you explain me what i do wrong, thank you guys for this great lib...

[saudet]

That probably means the pixel formats don't match. You'll need to figure what pixel format your data is in.

[1goresha]

thx for quick reply, should i change pixel format rigth here (ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_NV21))?

[saudet]

Yes, but you need to put the right one.

[1goresha]

Is it can be possible that the reason is not in pix_format? I used all this pix_format and unfortunately with no result :

ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_UYVY422) // ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_BGR565) // ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_ABGR) // ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_YUVJ444P) // ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_YUVJ422P) // ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_YUVJ420P) // ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_YUV420P16LE) // ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_MEDIACODEC) // ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_NV16) // ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_YUV420P) // ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_YUYV422) // ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_RGB565) // ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_RGB555) // ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_RGB444) // ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_RGBA) // ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_NV21) // ffmpegRecorder?.record(frame, avutil.AV_PIX_FMT_NV12)

can the reason be in bytebuffer :

frame.data = outPutByteBuffer

??

thx

[1goresha]

logs:

W/System.err: Debug: detected 4 logical cores W/System.err: Debug: [mpeg4 @ 0xb40000734b836c00] intra_quant_bias = 0 inter_quant_bias = -64 W/System.err: Debug: [file @ 0xb40000734b9427a0] Setting default whitelist 'file,crypto,data' W/System.err: Debug: [avi @ 0xb40000734b84ff00] reserve_index_space:0 master_index_max_size:256 W/System.err: Debug: [avi @ 0xb40000734b84ff00] duration_est:36000.000, filesize_est:0.0GiB, master_index_max_size:256 W/System.err: Info: Output #0, avi, to '/storage/0148-2150/Android/data/com.example.svidetel_server/files/Movies/VID_PHONE_20230219_164033.avi': W/System.err: Info: Metadata: W/System.err: Info: ISFT : W/System.err: Info: Lavf59.27.100 W/System.err: Info: W/System.err: Info: Stream #0:0 W/System.err: Debug: , 0, 1/30 W/System.err: Info: : Video: mpeg4, 1 reference frame (FMP4 / 0x34504D46), yuv420p, 1920x1080 (0x0), 0/1, q=2-31, 6 kb/s W/System.err: Info: , W/System.err: Info: 30 fps, W/System.err: Info: 30 tbn W/System.err: Info:

[saudet]

It sounds like you're trying to write already encoded data with FFmpegFrameRecorder. You'll need to create packets manually and use recordPacket() in that case...

[saudet]

Duplicate of #665

[1goresha]

Ok, thx. Can you provide me some examples how to create packets? It would help me so much. And as i know i have to start with context( ffMpegFrameRecorder.start(context) ) , but i can't understand how get this context. Thx for help at advance!

[saudet]

AVPacket is part of the API for FFmpeg, so if there's anything about that it's going to be over there: http://ffmpeg.org/documentation.html

[saudet]

BTW, FFmpeg 6.0 now supports hardware accelerated encoding using MediaCodec, so we don't need to use its API directly anymore. Please give it a try with the snapshots: http://bytedeco.org/builds/

[usernamety]

怎么使用MediaCodec 进行硬件加速编码