Capture duration incorrect for fragmented recording when CPU is high

[mmhmmh]

Hi,

This is a very useful library. Thanks to all who contributed.

While testing, I did find a weird issue when the fragmented recording option is turned on.

Basically if for any reason the capture loop is not at the specified framerate, the total duration will drift shorter resulting in sync issues.

One way to reproduce is to start the capture and then ran a CPU stress tool and capture exactly 5 mins of the desktop. You will see that the result video duration, when opened in the VLC, will say exactly 5 min, but when dragged to chrome web browser it will say something like 4:45 so about 15s short.

This behavior does not appear if the fragmented option is off.

I dumped the frames of the video and confirmed that the duration is indeed short.

I looked at the code and it looks like it is just using the MFCreateFMPEG4MediaSink instead of MFCreateMPEG4MediaSink so I have no clue how the behavior is different.

Can anyone share some insight on how to approach debugging this issue?

[sskodje]

It only happens when you stress the cpu? Do you use gpu or cpu encoding?

[mmhmmh]

I have only tested stressing the CPU because that's the most common scenario. It is using GPU accelerated encoding.

[mmhmmh]

@sskodje Hi, just wondering if you were able to reproduce this issue.

[sskodje]

Unfortunately i have not had any time to investigate it, as i've spent the time i have trying to get the next release out. Can you check if the issue is still present in this version?

https://github.com/sskodje/ScreenRecorderLib/actions/runs/1478803218

[jamesmundy]

Thanks for creating such a great library! I think I am experiencing the same issue here, or at least similar. I'm trying to screen record on a quite resource constrained VM (no GPU also) and I noticed that the end of my videos were being cut off. CPU utilization peaks at 100% during the recording process. I've played around to see what can be done to get the full video.

Before my config code looked like this:

VideoEncoderOptions = new VideoEncoderOptions
                {
                    Bitrate = 8000 * 1000,
                    Framerate = 60,
                    IsFixedFramerate = true,
                    //Currently supported are H264VideoEncoder and H265VideoEncoder
                    Encoder = new H264VideoEncoder
                    {
                        BitrateMode = H264BitrateControlMode.CBR,
                        EncoderProfile = H264Profile.Main,
                    },
                    //Fragmented Mp4 allows playback to start at arbitrary positions inside a video stream,
                    //instead of requiring to read the headers at the start of the stream.
                    //IsFragmentedMp4Enabled = true,
                    //If throttling is disabled, out of memory exceptions may eventually crash the program,
                    //depending on encoder settings and system specifications.
                    IsThrottlingDisabled = false,
                    //Hardware encoding is enabled by default.
                    IsHardwareEncodingEnabled = true,
                    //Low latency mode provides faster encoding, but can reduce quality.
                    IsLowLatencyEnabled = false,
                    //Fast start writes the mp4 header at the beginning of the file, to facilitate streaming.
                    IsMp4FastStartEnabled = false
                },

which cut off the end. Like this I no longer get the cut off (though CPU is still at 100%).

VideoEncoderOptions = new VideoEncoderOptions
                {
                    Bitrate = 4000 * 1000,
                    // Reduce framerate. Original: 60. New: 30
                    Framerate = 30,
                    //IsFixedFramerate = true,
                    //Currently supported are H264VideoEncoder and H265VideoEncoder
                    Encoder = new H264VideoEncoder
                    {
                        BitrateMode = H264BitrateControlMode.UnconstrainedVBR,
                        EncoderProfile = H264Profile.Main,
                    },
                    //Fragmented Mp4 allows playback to start at arbitrary positions inside a video stream,
                    //instead of requiring to read the headers at the start of the stream.
                    //IsFragmentedMp4Enabled = true,
                    //If throttling is disabled, out of memory exceptions may eventually crash the program,
                    //depending on encoder settings and system specifications.
                    IsThrottlingDisabled = false,
                    //Hardware encoding is enabled by default.
                    IsHardwareEncodingEnabled = true,
                    //Low latency mode provides faster encoding, but can reduce quality.
                    IsLowLatencyEnabled = false,
                    //Fast start writes the mp4 header at the beginning of the file, to facilitate streaming.
                    IsMp4FastStartEnabled = false
                },

My initial thought was that perhaps the video processing is lagging behind and when I call stop the buffer is not yet empty and the processing hasn't yet caught up? Could this be the case?

The same code runs fine and captures the full video locally where my PC isn't resource constrained.

Appreciate your work!

[sskodje]

The library and ecoder works fully async, so when you end a recording, it will call Finalize() on the IMFSinkWriter, which will write the remaining buffer to the video encoder, which then calls an event when it is finished.

Are you sure your own code is waiting until either OnRecordingComplete or OnRecordingFailed is returned? If you have a timeout for example, it could kill the process before it is finished. If the resources are constrained and the buffer is full, it could take a little while for the encoder to finish up.

It could also probably help to turn on logging to a file with "trace" details, to see what happens internally.

[jamesmundy]

@sskodje thanks for replying. Yes, I am waiting for the complete, with this code:

var tcs = new TaskCompletionSource<bool>();
_recorder.OnRecordingComplete += (s, e) => tcs.SetResult(true);

_recorder.Stop();

// Wait for the OnRecordingComplete event to be triggered
await tcs.Task;

This is the log I see with an error.

2024-06-21 15:04:21.039 [INFO]  [RecordingManager.cpp     |         operator (): 284] >> Starting recording task
2024-06-21 15:04:21.058 [ERROR] [OutputManager.cpp        | InitializeVideoSink: 454] >> BAD HR: hr=0x80004002, error is: No such interface supported
2024-06-21 15:04:42.828 [INFO]  [OutputManager.cpp        |   FinalizeRecording: 133] >> Cleaning up resources
2024-06-21 15:04:42.831 [INFO]  [OutputManager.cpp        |   FinalizeRecording: 134] >> Finalizing recording
2024-06-21 15:04:52.269 [INFO]  [RecordingManager.cpp     |         operator (): 306] >> Exiting recording task

So there is an error with no interface being supported but I have IsHardwareEncodingEnabled set to false.

Update I have upgraded the VM to have more CPU & Memory and can get a good quality video which is the desired length with this:

VideoEncoderOptions = new VideoEncoderOptions
{
    Bitrate = 3500 * 1000,
    // Reduce framerate. Original: 60. New: 30
    Framerate = 60,
    //IsFixedFramerate = true,
    //Currently supported are H264VideoEncoder and H265VideoEncoder
    Encoder = new H264VideoEncoder
    {
        BitrateMode = H264BitrateControlMode.UnconstrainedVBR,
        EncoderProfile = H264Profile.Main,
    },
    //Fragmented Mp4 allows playback to start at arbitrary positions inside a video stream,
    //instead of requiring to read the headers at the start of the stream.
    IsFragmentedMp4Enabled = true,
    //If throttling is disabled, out of memory exceptions may eventually crash the program,
    //depending on encoder settings and system specifications.
    IsThrottlingDisabled = false,
    //Hardware encoding is enabled by default.
    IsHardwareEncodingEnabled = false,
    //Low latency mode provides faster encoding, but can reduce quality.
    IsLowLatencyEnabled = false,
    //Fast start writes the mp4 header at the beginning of the file, to facilitate streaming.
    IsMp4FastStartEnabled = true
},

but the error is still there unfortunately:

2024-06-21 17:21:28.574 [INFO]  [RecordingManager.cpp     |         operator (): 284] >> Starting recording task
2024-06-21 17:21:28.584 [ERROR] [OutputManager.cpp        | InitializeVideoSink: 454] >> BAD HR: hr=0x80004002, error is: No such interface supported
2024-06-21 17:21:36.370 [INFO]  [RecordingManager.cpp     |   ~RecordingManager: 116] >> Media Foundation shut down
2024-06-21 17:21:41.839 [INFO]  [OutputManager.cpp        |   FinalizeRecording: 133] >> Cleaning up resources
2024-06-21 17:21:41.840 [INFO]  [OutputManager.cpp        |   FinalizeRecording: 134] >> Finalizing recording
2024-06-21 17:21:42.627 [INFO]  [RecordingManager.cpp     |         operator (): 306] >> Exiting recording task

[sskodje]

InitializeVideoSink: 454] >> BAD HR: hr=0x80004002, error is: No such interface supported

This error means means it fails to add hardware acceleration to a texture transform because there are no hardware devices. It shouldn't affect anything.

If you set IsFragmentedMp4Enabled = false, does the resulting video become corrupt? Fragmented mp4 may continue to work even when the video encoder errors or abruptly stops, so it's a way to check if the encoder properly finalizes or not.

[jamesmundy]

The files have never been corrupted and I just tried it with the fragmentation turned off and the video recorded fine. For now I'm going to use a bigger VM as that produces a higher quality and less stuttery video without the missing end but that is definitely occurring. I'm using a basic Azure B2 instance with 2vCPUs and 8GB RAM so it is quite constrained! I'd be happy to help debug but I have a workaround for now.