Enhancement: Add LateUpdate-based ScreenInput Option for Capturing VR Footage

[BasilDavid]

Issue: I have been using the NatCorder library in combination with the Google Cardboard XR Plugin to capture VR footages. However, I encountered an issue where the screen recorder was capturing single-view footages instead of capturing the VR black mask as desired.

Proposed Enhancement: I have implemented a new class, LateScreenInput, that captures frames using LateUpdate, allowing me to capture the VR black mask successfully. This addition provides users with more flexibility when recording VR content, as it captures the VR mask, while the ScreenInput can still capture non-VR views. I believe this enhancement could be valuable for users who need to capture VR content with the Cardboard XR Plugin.

Code Changes: I've made the following changes in the LateScreenInput class:

• Removed the WaitForEndOfFrame yield and replaced it with LateUpdate-based frame capturing.
• Added a resolution parameter to customize the recording resolution.
• Resized the captured frame to match the desired resolution.

Code Comparison: Below is a comparison of the original ScreenInput class and my modified LateScreenInput class.

Original ScreenInput class:

/* 
*   NatCorder
*   Copyright (c) 2022 NatML Inc. All Rights Reserved.
*/

namespace NatML.Recorders.Inputs {

    using System;
    using System.Collections;
    using UnityEngine;
    using Clocks;

    /// <summary>
    /// Recorder input for recording video frames from the screen.
    /// Unlike the `CameraInput`, this recorder input is able to record overlay UI canvases.
    /// </summary>
    public sealed class ScreenInput : IDisposable {

        #region --Client API--
        /// <summary>
        /// Control number of successive camera frames to skip while recording.
        /// This is very useful for GIF recording, which typically has a lower framerate appearance.
        /// </summary>
        public int frameSkip;

        /// <summary>
        /// Create a video recording input from the screen.
        /// </summary>
        /// <param name="recorder">Media recorder to receive video frames.</param>
        /// <param name="clock">Recording clock for generating timestamps.</param>
        public ScreenInput (IMediaRecorder recorder, IClock clock = default) : this(TextureInput.CreateDefault(recorder), clock) { }

        /// <summary>
        /// Create a video recording input from the screen.
        /// </summary>
        /// <param name="input">Texture input to receive video frames.</param>
        /// <param name="clock">Recording clock for generating timestamps.</param>
        public ScreenInput (TextureInput input, IClock clock = default) {
            this.input = input;
            this.clock = clock;
            this.frameDescriptor = new RenderTextureDescriptor(input.frameSize.width, input.frameSize.height, RenderTextureFormat.ARGB32, 0);
            // Start recording
            attachment = new GameObject("NatCorder ScreenInputAttachment").AddComponent<ScreenInputAttachment>();
            attachment.StartCoroutine(CommitFrames());
        }

        /// <summary>
        /// Stop recorder input and release resources.
        /// </summary>
        public void Dispose () {
            GameObject.Destroy(attachment.gameObject);
            input.Dispose();
        }
        #endregion

        #region --Operations--
        private readonly TextureInput input;
        private readonly IClock clock;
        private readonly RenderTextureDescriptor frameDescriptor;
        private readonly ScreenInputAttachment attachment;
        private int frameCount;

        private IEnumerator CommitFrames () {
            var yielder = new WaitForEndOfFrame();
            for (;;) {
                // Check frame index
                yield return yielder;
                if (frameCount++ % (frameSkip + 1) != 0)
                    continue;
                // Capture screen
                var frameBuffer = RenderTexture.GetTemporary(frameDescriptor);
                if (SystemInfo.graphicsUVStartsAtTop) {
                    var tempBuffer = RenderTexture.GetTemporary(frameDescriptor);
                    ScreenCapture.CaptureScreenshotIntoRenderTexture(tempBuffer);
                    Graphics.Blit(tempBuffer, frameBuffer, new Vector2(1, -1), Vector2.up);
                    RenderTexture.ReleaseTemporary(tempBuffer);
                } else
                    ScreenCapture.CaptureScreenshotIntoRenderTexture(frameBuffer);
                // Commit
                input.CommitFrame(frameBuffer, clock?.timestamp ?? 0L);
                RenderTexture.ReleaseTemporary(frameBuffer);
            }
        }

        private sealed class ScreenInputAttachment : MonoBehaviour { }
        #endregion
    }
}

Modified LateScreenInput class:

namespace NatML.Recorders.Inputs
{
    using System;
    using System.Collections;
    using UnityEngine;
    using Clocks;

    /// <summary>
    /// Recorder input for recording video frames from the screen.
    /// Unlike the `CameraInput`, this recorder input is able to record overlay UI canvases.
    /// </summary>
    public sealed class LateScreenInput : IDisposable
    {
        #region --Client API--

        /// <summary>
        /// Control number of successive camera frames to skip while recording.
        /// This is very useful for GIF recording, which typically has a lower framerate appearance.
        /// </summary>
        public int frameSkip;

        /// <summary>
        /// Create a video recording input from the screen.
        /// </summary>
        /// <param name="recorder">Media recorder to receive video frames.</param>
        /// <param name="clock">Recording clock for generating timestamps.</param>
        /// <param name="resolution"></param>
        public LateScreenInput(IMediaRecorder recorder, IClock clock = default) : this(
            TextureInput.CreateDefault(recorder),
            clock)
        {
        }

        /// <summary>
        /// Create a video recording input from the screen.
        /// </summary>
        /// <param name="input">Texture input to receive video frames.</param>
        /// <param name="clock">Recording clock for generating timestamps.</param>
        public LateScreenInput(TextureInput input, IClock clock = default)
        {
            this.input = input;
            this.clock = clock;
            screenDescriptor = new RenderTextureDescriptor(Screen.width, Screen.height,
                RenderTextureFormat.ARGBHalf, 0);
            resizeDescriptor = new RenderTextureDescriptor(input.frameSize.width, input.frameSize.height,
                RenderTextureFormat.ARGBHalf, 0);
            // Start recording
            attachment = new GameObject(@"NatCorder ScreenInputAttachment").AddComponent<ScreenInputAttachment>();
            //attachment.StartCoroutine(CommitFrames());
            attachment.CommitFrames = CommitFrames;
        }

        /// <summary>
        /// Stop recorder input and release resources.
        /// </summary>
        public void Dispose()
        {
            GameObject.DestroyImmediate(attachment.gameObject);
            input.Dispose();
        }

        #endregion

        #region --Operations--

        private readonly TextureInput input;
        private readonly IClock clock;
        private readonly RenderTextureDescriptor screenDescriptor;
        private readonly RenderTextureDescriptor resizeDescriptor;
        private readonly ScreenInputAttachment attachment;
        private int frameCount;

        public void CommitFrames()
        {
            // Check frame index
            if (frameCount++ % (frameSkip + 1) != 0)
                return;
            // Capture screen
            var frameBuffer = RenderTexture.GetTemporary(screenDescriptor);
            if (SystemInfo.graphicsUVStartsAtTop)
            {
                var tempBuffer = RenderTexture.GetTemporary(screenDescriptor);
                ScreenCapture.CaptureScreenshotIntoRenderTexture(tempBuffer);
                Graphics.Blit(tempBuffer, frameBuffer, new Vector2(1, -1), Vector2.up);
                RenderTexture.ReleaseTemporary(tempBuffer);
            }
            else
                ScreenCapture.CaptureScreenshotIntoRenderTexture(frameBuffer);

            // Resizing
            var resizedFrameBuffer = RenderTexture.GetTemporary(resizeDescriptor);
            Graphics.Blit(frameBuffer, resizedFrameBuffer);
            Debug.LogError(resizedFrameBuffer.width + "   " + resizedFrameBuffer.height);
            // Commit
            input.CommitFrame(resizedFrameBuffer, clock?.timestamp ?? 0L);
            RenderTexture.ReleaseTemporary(frameBuffer);
            RenderTexture.ReleaseTemporary(resizedFrameBuffer);
        }

        private sealed class ScreenInputAttachment : MonoBehaviour
        {
            public Action CommitFrames;

            private void LateUpdate()
            {
                CommitFrames();
            }
        }

        #endregion
    }
}

How to Use: To use the LateScreenInput class for capturing VR content with the Cardboard XR Plugin, follow these steps:

• Instantiate a LateScreenInput object, passing your media recorder and desired resolution as parameters. Additionally, make sure to create a RealtimeClock for timestamp generation.

Example Usage:

// Create a RealtimeClock for timestamp generation
clock = new RealtimeClock();

// Calculate the recording resolution based on your aspect ratio
float aspectRatio = (float)Screen.width / Screen.height;
int recordWidth = (int)(Resolution * aspectRatio);
recordWidth = recordWidth % 2 == 0 ? recordWidth : recordWidth - 1;

// Instantiate an MP4Recorder with your resolution and desired frame rate (e.g., 30 FPS)
recorder = new MP4Recorder(recordWidth, Resolution, 30);

// Instantiate a LateScreenInput with the recorder, resolution, and clock
lateScreenInput = new LateScreenInput(recorder, clock);

Additional Information:

Operating System: Windows 10 x64
Unity Editor Version: 2020.3.30f1
NatCorder Version: 1.9.1
Cardboard XR Plugin Version: 1.21.0

Expected Behavior:

With this enhancement, users should be able to capture VR footages that include the VR black mask when using the Cardboard XR Plugin, in addition to the existing non-VR view capture functionality.

[olokobayusuf]

Hey @BasilDavid , thanks for the excellently detailed issue. I think this is a great feature request, though I think how I'd prefer solving is a bit different.

I think it would be better for the VideoKitRecorder input to allow users to set the input to be used. That way instead of us adding the LateScreenInput, you can attach any arbitrary input you'd like. I have to figure out a clean API to do this, but I'll run it by you.

I'd like to get this into the next update.

[BasilDavid]

Hey @olokobayusuf , Thank you for your quick response. I appreciate your willingness to consider my request.

I agree that allowing users to specify the input source for VideoKitRecorder is a great idea, providing more flexibility. I look forward to seeing the clean API you'll create for this feature.

If you need any feedback or assistance, please feel free to reach out. I'm ready to help.

[olokobayusuf]

@BasilDavid following up on this old thread, but this was implemented a while ago in the upcoming VideoKit update (0.0.18 alpha):

MediaRecorder recorder = ...
ScreenSource source = new ScreenSource(recorder, useLateUpdate: true);