Custom VideoView - memory issues

[dostalleos]

Description

I'm trying to apply color filter to the incoming video. I created my custom ColorFilterVideoView which extends VideoView. My video view only manipulate with frame data and create new frame to be passed to original the VideoView. You can find my whole implementation bellow. Color filter itself works well and I'm able to apply it to the incoming video. But memory consumption of the app is growing really fast up to 1 GB and then app freezes. I assume that original frame is not consumed or released, but I haven't found a way how to fix it.

Steps to Reproduce

1. Add ColorFilterVideoView (you can find it bellow) to the quickstartKotlin module.
2. Replace VideoView with ColorFilterVideoView in content_video.xml
3. Run the quickstartKotlin
4. Observe memory consumption in the profiler

Code

package com.twilio.video.quickstart.kotlin

import android.content.Context
import android.util.AttributeSet
import com.twilio.video.VideoView
import tvi.webrtc.JavaI420Buffer
import tvi.webrtc.VideoFrame
import java.nio.ByteBuffer

class ColorFilterVideoView : VideoView {

    constructor(context: Context) : super(context)
    constructor(context: Context, attrs: AttributeSet?) : super(context, attrs)

    override fun onFrame(frame: VideoFrame?) {
        frame?.let {
            handleYuvBuffer(it)
        } ?: super.onFrame(frame)
    }

    private fun handleYuvBuffer(frame: VideoFrame) {
        frame.retain()
        val i420Frame = frame.buffer.toI420()

        // Color color
        val uBuffer = ByteBuffer.allocateDirect(i420Frame.dataU.capacity())
        val vBuffer = ByteBuffer.allocateDirect(i420Frame.dataV.capacity())
        val uData = ByteArray(uBuffer.capacity()) { 99.toByte() }
        val vData = ByteArray(vBuffer.capacity()) { 99.toByte() }
        uBuffer.apply {
            put(uData)
            rewind()
        }

        vBuffer.apply {
            put(vData)
            rewind()
        }

        val output = JavaI420Buffer.wrap(i420Frame.width, i420Frame.height, i420Frame.dataY, i420Frame.strideY, uBuffer, i420Frame.strideU, vBuffer, i420Frame.strideV, null)

        super.onFrame(VideoFrame(output, frame.rotation, frame.timestampNs))
        frame.release()
    }
}

Expected Behavior

It should not consume so much memory and should not freeze or crash.

Actual Behavior

App freezes or crashes after some time, usually 1-2 min of active video. For local video it takes little bit longer until app freezes. You can join to the same room with another device to make freeze faster.

Reproduces how Often

Every time

Video Android SDK

6.0.0-beta1

Android API

30, 29

Android Device

Pixel 3, Samsung Tab A

[aaalaniz]

Hey @dostalleos

Great issue! I have filed an issue on our backlog and @Alton09 will investigate.

Thanks!

[Alton09]

Hi @dostalleos. It looks like releasing the i420Frame in your example above fixes the memory leak. We also noticed some improvements when only initializing the u and v buffers once the first time the onFrame function is called. Here's what our modifications look like:

package com.twilio.video.quickstart.kotlin

import android.content.Context
import android.util.AttributeSet
import com.twilio.video.VideoView
import tvi.webrtc.JavaI420Buffer
import tvi.webrtc.VideoFrame
import java.nio.ByteBuffer

class ColorFilterVideoView : VideoView {

    constructor(context: Context) : super(context)
    constructor(context: Context, attrs: AttributeSet?) : super(context, attrs)

    private var uBufferCache: ByteBuffer? = null
    private var vBufferCache: ByteBuffer? = null

    override fun onFrame(frame: VideoFrame?) {
        frame?.let {
            handleYuvBuffer(it)
        } ?: super.onFrame(frame)
    }

    private fun handleYuvBuffer(frame: VideoFrame) {
        frame.retain()
        val i420Frame = frame.buffer.toI420()
        initializeBufferCache(i420Frame)
        val output = JavaI420Buffer.wrap(i420Frame.width, i420Frame.height, i420Frame.dataY, i420Frame.strideY, uBufferCache, i420Frame.strideU, vBufferCache, i420Frame.strideV, null)
        val newFrame = VideoFrame(output, frame.rotation, frame.timestampNs)
        super.onFrame(newFrame)
        i420Frame.release()
        frame.release()
    }

    private fun initializeBufferCache(i420Frame: VideoFrame.I420Buffer) {
        if(uBufferCache == null) {
            uBufferCache = ByteBuffer.allocateDirect(i420Frame.dataU.capacity()).apply {
                val uData = ByteArray(capacity()) { 99.toByte() }
                put(uData)
                rewind()
            }
        }
        if(vBufferCache == null) {
            vBufferCache = ByteBuffer.allocateDirect(i420Frame.dataV.capacity()).apply {
                val vData = ByteArray(capacity()) { 99.toByte() }
                put(vData)
                rewind()
            }
        }
    }
}

[dostalleos]

Hello @Alton09, thank you for you response and help. It works well and it fixed memory issues I had.

Also thank you for your suggested improvement. I had to change condition little bit to : if(uBufferCache?.capacity() != i420Frame.dataU.capacity()), because video resolution can be changed.

Unfortunately I have different issue now. I'm experiencing some artifacts in the remote video (not in the local one). Please see my video for the reference. It's captured on Samsung Galagy Tab A. It's also happening on my Pixel 3, but not that much.

This issue is connected to the releasing of the i420Frame. If I don't release i420Frame frame, it works without artifacts, but it leaks 🤔

Do you have an idea what could be wrong?

https://user-images.githubusercontent.com/2716434/102791849-eb3a5480-43a7-11eb-874d-e6cd88ca454b.mp4

[dostalleos]

Hello @Alton09

I've probably found final solution 🎉 The key thing is to release i420Frame in the releaseCallback of the new output buffer and release output not i420Frame after calling super.onFrame().

Final solution:

class ColorFilterVideoView : VideoView {

    constructor(context: Context) : super(context)
    constructor(context: Context, attrs: AttributeSet?) : super(context, attrs)

    private var uBufferCache: ByteBuffer? = null
    private var vBufferCache: ByteBuffer? = null

    override fun onFrame(frame: VideoFrame?) {
        frame?.let {
            handleYuvBuffer(it)
        } ?: super.onFrame(frame)
    }

    private fun handleYuvBuffer(frame: VideoFrame) {
        frame.retain()
        val i420Frame = frame.buffer.toI420()

        initializeBufferCache(i420Frame)

        val output = JavaI420Buffer.wrap(i420Frame.width, i420Frame.height, i420Frame.dataY, i420Frame.strideY, uBufferCache, i420Frame.strideU, vBufferCache, i420Frame.strideV) { i420Frame.release() }

        super.onFrame(VideoFrame(output, frame.rotation, frame.timestampNs))

        output.release()
        frame.release()
    }

    private fun initializeBufferCache(i420Frame: VideoFrame.I420Buffer) {
        if (uBufferCache?.capacity() != i420Frame.dataU.capacity()) {
            uBufferCache = ByteBuffer.allocateDirect(i420Frame.dataU.capacity()).apply {
                val uData = ByteArray(capacity()) { 99.toByte() }
                put(uData)
                rewind()
            }
        }
        if (vBufferCache?.capacity() != i420Frame.dataV.capacity()) {
            vBufferCache = ByteBuffer.allocateDirect(i420Frame.dataV.capacity()).apply {
                val vData = ByteArray(capacity()) { 99.toByte() }
                put(vData)
                rewind()
            }
        }
    }
}

Do you think that this is the correct solution? I'm not able to reproduce "artifacts" and memory issues with this solution and everything seems to be ok now.

[Alton09]

Nice! Great find @dostalleos. The releaseCallback appears to be the right approach to release the i420frame. Glad to hear that it fixes the memory leak and video artifact issues. I'll close this issue since it has been resolved, but feel free to open a new issue as needed.

[kostyabakay]

@dostalleos I had absolutely the same problem but in pure WebRTC and your comments helped me to solve my issue. Thank you!