Unity Freezes During Asynchronous Operations with MediaRecorder (Prepare on awake)

[BasilDavid]

Description:

I am encountering an issue where Unity iOS app freezes during Prepare on awake asynchronous operation, specifically when using the MediaRecorder class for video processing. Despite using async/await and offloading tasks to background threads, Unity's main thread still freezes, which impacts the overall performance of the application.

Steps to Reproduce:

I took the VideoKitRecorder.PrepareEncoder method for testing on a new script

1. Implement the following code in a Unity project.
2. Run the project and observe the iOS build freezing during execution.

Code Example:

    void Awake()
    {
        PrepareEncoder();
    }

    private static async Task PrepareEncoder()
    {
        // Check platform // CHECK // WebGL
        Stopwatch stopwatch = new Stopwatch();
        stopwatch.Start();
        if (!MediaRecorder.CanCreate(MediaFormat.MP4))
            return;
        stopwatch.Stop();
        UnityEngine.Debug.LogFormat("[VideoKit] {0} executed in {1} milliseconds", "CanCreate", stopwatch.ElapsedMilliseconds);

        // Create recorder
        var width = 1280;
        var height = 720;

        stopwatch.Restart();
        var clock = new FixedClock(30);
        stopwatch.Stop();
        UnityEngine.Debug.LogFormat("[VideoKit] {0} executed in {1} milliseconds", "FixedClock(30)", stopwatch.ElapsedMilliseconds);

        stopwatch.Restart();
        var recorder = await MediaRecorder.Create(MediaFormat.MP4, width: width, height: height, frameRate: 30);
        stopwatch.Stop();
        UnityEngine.Debug.LogFormat("[VideoKit] {0} executed in {1} milliseconds", "MediaRecorder.Create", stopwatch.ElapsedMilliseconds);

        // Commit empty frames
        stopwatch.Restart();
        using var pixelData = new NativeArray<byte>(
            width * height * 4,
            Allocator.Persistent,
            NativeArrayOptions.ClearMemory
        );
        stopwatch.Stop();
        UnityEngine.Debug.LogFormat("[VideoKit] {0} executed in {1} milliseconds", "new NativeArray", stopwatch.ElapsedMilliseconds);

        var format = PixelBuffer.Format.RGBA8888;

        for (var i = 0; i < 3; ++i)
        {
            stopwatch.Restart();
            using var pixelBuffer = new PixelBuffer(width, height, format, pixelData, timestamp: clock.timestamp);
            stopwatch.Stop();
            UnityEngine.Debug.LogFormat("[VideoKit] {0} : {1} executed in {2} milliseconds", i, "new PixelBuffer", stopwatch.ElapsedMilliseconds);

            stopwatch.Restart();
            recorder.Append(pixelBuffer);
            stopwatch.Stop();
            UnityEngine.Debug.LogFormat("[VideoKit] {0} : {1} executed in {2} milliseconds", i, "recorder.Append", stopwatch.ElapsedMilliseconds);
        }

        // Finish and delete
        stopwatch.Restart();
        var asset = await recorder.FinishWriting();
        stopwatch.Stop();
        UnityEngine.Debug.LogFormat("[VideoKit] {0} executed in {1} milliseconds", "recorder.FinishWriting", stopwatch.ElapsedMilliseconds);

        stopwatch.Restart();
        File.Delete(asset.path);
        stopwatch.Stop();
        UnityEngine.Debug.LogFormat("[VideoKit] {0} executed in {1} milliseconds", "File.Delete", stopwatch.ElapsedMilliseconds);
    }

Observed Results:

[VideoKit] CanCreate executed in 0 milliseconds
[VideoKit] FixedClock(30) executed in 0 milliseconds
[VideoKit] MediaRecorder.Create executed in 2519 milliseconds
[VideoKit] new NativeArray executed in 0 milliseconds
[VideoKit] 0 : new PixelBuffer executed in 0 milliseconds
[VideoKit] 0 : recorder.Append executed in 9903 milliseconds
[VideoKit] 1 : new PixelBuffer executed in 0 milliseconds
[VideoKit] 1 : recorder.Append executed in 1 milliseconds
[VideoKit] 2 : new PixelBuffer executed in 0 milliseconds
[VideoKit] 2 : recorder.Append executed in 0 milliseconds
[VideoKit] recorder.FinishWriting executed in 84 milliseconds
[VideoKit] File.Delete executed in 0 milliseconds

Problem:

• Freezing Issue: Despite the PrepareEncoder method being async, Unity freezes, particularly during the MediaRecorder.Create and recorder.Append operations.
• Unexpected Timings: The operations MediaRecorder.Create and recorder.Append take an unusually long time (2519 ms and 9903 ms respectively), which might contribute to the freezing.
• Asynchronous Execution: Using Task.Run or other methods to offload work to background threads does not resolve the issue.

Expected Behavior:

• The method should run asynchronously without causing the build to freeze.
• The operations should execute more efficiently, with asynchronous handling ensuring the main thread remains responsive.

Environment:

Unity Version: 2022.3.40f1 Editor Platform: macOS Target Platform: iOS VideoKit Library Version: 0.0.20

Request for Help:

• Potential Fixes: Any insights into how to prevent the freezing or optimize the operations would be appreciated.
• Thread Safety: Clarifications on whether MediaRecorder.Create and recorder.Append should be handled differently in an async context would be helpful.

[olokobayusuf]

Potential Fixes: Any insights into how to prevent the freezing or optimize the operations would be appreciated.

@BasilDavid this is a known issue, and is the reason why prepareOnAwake exists. When an application first uses a hardware encoder, the system has to provision hardware resources. This results in a noticeable hiccup which is app-wide, and can't be worked around by using a background thread (see this question). The prepareOnAwake setting exists to hide this hiccup in app initialization, so there isn't a noticeable hiccup during the app's execution. After this first recording, all subsequent recordings will never show this hiccup.

Thread Safety: Clarifications on whether MediaRecorder.Create and recorder.Append should be handled differently in an async context would be helpful.

Media recorders can be used from different threads. You just have to synchronize accesses using a lock.