Video Capture Mirroring

[kodypeterson]

Version

N/A

Environment that reproduces the issue

N/A

Use case description

Recently, I drafted a PR for the api.video ios library to support the ability to mirror the video image https://github.com/apivideo/api.video-ios-live-stream/pull/11

The hope was to come here and do the same, but I think it may not be possible - But was curious if you had any thoughts on a solution.

My initial attempt was to setMatrix on the previewSurface as well as the encoderSurface. But, this requires locking the canvas, which I was unable to accomplish while the preview is active. Here is the Mirror class I was messing around with. The new param cameraCapture is passed in when the CameraSettings class gets instantiated.

class Mirror(private val context: Context, private val cameraController: CameraController, private val cameraCapture: CameraCapture) {
    private val notMirrored: Matrix = Matrix()
    private val mirrored: Matrix = Matrix()
    private var _isMirrored = false;

    init {
        notMirrored.setScale(0F, 0F)
        mirrored.setScale(-1F, 1F)
    }

    var isMirrored: Boolean
        get() {
            return _isMirrored
        }

        set(newValue) {
            if (cameraCapture.previewSurface == null) return

            val canvas: Canvas = cameraCapture.previewSurface!!.lockCanvas(Rect())

            if (newValue) {
                canvas.setMatrix(mirrored)
            } else {
                canvas.setMatrix(notMirrored)
            }

            cameraCapture.previewSurface!!.unlockCanvasAndPost(canvas)
        }
}

Most solutions seem to talk about modifying the actual SurfaceView or TextureView, but this library seems to rely on Camera2 API to handle the surfaces (targets) directly.

Curious if you have any additional ideas on how to implement the ability to mirror (flip) the image.

Proposed solution

No response

Alternative solutions

No response

[kodypeterson]

@ThibaultBee Just wanted to bump this to see if you had any thoughts

[ThibaultBee]

Hi, Sorry I was off for last couple of weeks.

Why are you trying to mirror the camera? What surface (preview or stream)? Is there something wrong with how StreamPack handles the orientation? I have seen a fork where someone try to mirror the video. 🤔 Is this related with https://github.com/ThibaultBee/StreamPack/issues/73?

[kodypeterson]

We found this only happens with the front camera. Our end goal is to have the preview surface stay as-is but the stream be mirrored. Reason being is that our use case is more users using the front camera to record themselves do an activity. When watching your self, the way the surface is as of now is what we would expect to see. But, as a viewer of the stream it feels wrong.

This screenshot is a great example. As a streamer, you would expect the screen to not be readable. But, as a viewer you would expect to see it correct as if you were looking at it, in this case mirrored.

[ThibaultBee]

What version are you using? I am pretty confident it won't fix your issue but I made few changes on the main branch in March which haven't been released yet. And in the master branch, the issue might be in the OrientationProvider or the CodecSurface of the VideoMediaCodecEncoder.

Finally, IMO it is a bug! 🐛

[kodypeterson]

@ThibaultBee Thanks for the info - I will likely not be able to dig back into until late next week. Will keep you posted on what I can find.

[ThibaultBee]

Hi,

I started to work on this. See PR https://github.com/ThibaultBee/StreamPack/pull/85

Unfortunately, I am not satisfied by the result yet because the during the transition from back to front camera (or front to back), there is a small amount of time were the mirror is applied on the wrong frame. I would like to mirror only at the exact frame.

[ThibaultBee]

Ok, it seems better. Could you test https://github.com/ThibaultBee/StreamPack/pull/85?

[ThibaultBee]

Released in 2.6.0. Please reopen the issue if it is not fixed.