added PRIORITIZE_H264 env

mohan43u commented 5 months ago

while recording, mirotalk selects vp9,vp8 then goes to h264. most of the modern browsers support vp9/vp8, but vp9/vp8 becomes issue when postprocessing, hardware accelerated encoding/decoding is limited for vp9/vp8, so this env will prioritize h264/aac before moving to other possibilities.

miroslavpejic85 commented 5 months ago

Hey @mohan43u!

Thank you for the pull request. I'd like to provide some insights into why we currently employ the existing codecs.

The choice of the best codec for recording depends on various factors such as browser support, video quality requirements, and compatibility with the intended playback devices. Each codec has its own advantages and disadvantages. Here's a brief overview of the current MiroTalk P2P options and why we put it in this order:

/**
 * Get MediaRecorder MimeTypes
 * @returns {boolean} is mimeType supported by media recorder
 */
function getSupportedMimeTypes() {
    const possibleTypes = [
        'video/webm;codecs=vp9,opus',
        'video/webm;codecs=vp8,opus',
        'video/webm;codecs=h264,opus',
        'video/mp4;codecs=h264,aac',
        'video/mp4',
    ];
    return possibleTypes.filter((mimeType) => {
        return MediaRecorder.isTypeSupported(mimeType);
    });
}

video/webm;codecs=vp9,opus
- Supported by Chrome, Edge, Opera, Vivaldi, and Brave.
- Uses the VP9 video codec and Opus audio codec.
- Provides good video compression and quality.

video/webm;codecs=vp8,opus
- Supported by Chrome, Edge, Firefox, Opera, Vivaldi, and Brave.
- Uses the VP8 video codec and Opus audio codec.
- Similar to VP9 but with slightly less efficient compression.
- Firefox prefers VP8 over VP9 for WebM.

video/webm;codecs=h264,opus
- Supported by Chrome, Edge, Opera, Vivaldi, and Brave.
- Uses the H.264 video codec and Opus audio codec.
- H.264 is a widely supported video codec, but it may not provide as efficient compression as VP9.

video/mp4;codecs=h264,aac
- Supported by Safari
- Uses the H.264 video codec and AAC audio codec.
- Similar to the video/mp4 option but explicitly specifying the codecs.

video/mp4
- Supported by Safari.
- Uses the H.264 video codec and AAC audio codec.
- H.264 is a common and widely supported codec for MP4, suitable for Safari and many other platforms.

For us the broad compatibility is a priority, using a combination like video/webm;codecs=vp9,opus or video/webm;codecs=vp8,opus would be a good choice as they are supported by many modern browsers. In Safari the recording using video/mp4 as per default so not need to force it to video/mp4;codecs=h264,aac.

Our approach of starting with VP9/VP8 and then moving to H.264 as a fallback is a reasonable strategy. This way, you can take advantage of the potentially better compression efficiency of VP9/VP8 when it's supported, and smoothly transition to H.264 when needed for compatibility and performance reasons.

Web Browsers: Most modern web browsers, such as Google Chrome, Mozilla Firefox, Microsoft Edge..., support the playback of WebM files directly in the browser. You can simply open a WebM file using your browser, and it should play.

Online Video Players: There are online video players that support WebM playback. Websites like VLC Web Player and Online Video Converter allow you to upload and play WebM files directly through your web browser.

Cloud Storage Services: If your WebM files are stored in cloud storage services like Google Drive or Dropbox..., you can use their built-in media players to play WebM files.

Media Players with WebM Support: Standalone media players like VLC also support WebM playback. You can download and install these players on your computer to play WebM files locally.

The media format video/mp4;codecs=h264,aac is predominantly compatible only with Safari. If prioritize active, the other browsers will use video/webm;codecs=h264,opus. It's worth noting that while this format is widely supported, it may not offer the same level of compression efficiency as VP9.

Some considerations

Browser Support:
- H.264 (AVC): Widely supported across various browsers, including Chrome, Edge, Safari, and others.
- VP8/VP9: Supported in Chrome, Edge, Opera, Vivaldi, and Brave, but not in Safari.
Compression Efficiency:
- H.264 (AVC): Generally provides good compression efficiency, especially for high-motion videos.
- VP8/VP9: Known for better compression efficiency, especially at lower bitrates. VP9, in particular, is designed to offer higher quality at lower bitrates compared to H.264.
Quality:
- H.264 (AVC): Provides good video quality, but VP9 may outperform it in certain scenarios, especially at lower bitrates.
- VP8/VP9: Known for delivering high-quality video, especially in scenarios where bandwidth is limited.
Patent and Licensing Issues:
- H.264 (AVC): Some users may be concerned about potential patent and licensing issues associated with H.264.
- VP8/VP9: Royalty-free and open-source, which can be advantageous for certain projects.
File Size:
- H.264 (AVC): Files may be larger compared to VP8/VP9 at the same quality level.
- VP8/VP9: Generally provides better compression, leading to smaller file sizes for a given quality level.

PR purpose

Correct me if I wrong:

VP9/VP8: Potential for better compression efficiency but may pose challenges with hardware acceleration and post-processing.

H.264 (AVC): Widely supported, efficient, and often benefits from optimized hardware encoding/decoding.

By prioritizing H.264/AAC - H.264/OPUS as the default and fallback option, you strike a balance between compatibility, efficiency, and post-processing considerations. This strategy is particularly relevant in environments where hardware support for VP9/VP8 may not be consistent or optimal.

Question

Can you please describe the advantages you gain by prioritizing the H.264/AAC - H.264/OPUS codec in your usage scenario?

Thank you again and have a great weekend. Miroslav

mohan43u commented 5 months ago

Can you please describe the advantages you gain by prioritizing the H.264/AAC - H.264/OPUS codec in your usage scenario?

Me and my friend were on remote call discussing about one book, so we throught of recording the discussion to later upload in my friend's youtube channel. He started the recording, the discussion went almost 1 hour. We stopped the recording and got the recorded file in vp8/opus (he used firefox).

The problem started when we try to reduce the length and to cut unwanted part of discussion. I don't know why, but the recorded video didn't have timestamp. so I have to first put some timestamp on both audio and video streams, ffmpeg refused to put timestamp when I used -codec:a copy and -codec:v copy. So, ~800MB file has to be re processed.

Since the video is vp8, I dont have a system which is capable of doing hardware accelerated decoding and encoding for this codec, so I have to convert the entire ~800MB file into h264/aac with timestamp. This took almost ~2 hours (on i5 11th gen laptop). once I converted to h264, cutting that long video and scaling the chunks to 1920x1080 become very easy as I can use -hwaccel vaapi with scale_vaapi and -codec:v h264_vaapi to process those chunks. rendering the final video took only few seconds as the chunks are already scaled up to fullHD and framerates are set properly with the help of hardware acceleration.

So, I thought of making the browser to record the source video in h264 so that post processing will be much more easier on systems (like mine) which have good hardware acceleration support only for h264. so this patch happened.

miroslavpejic85 commented 5 months ago

Thank you for the detailed explanation, much appreciated!

So, I thought of making the browser to record the source video in h264 so that post processing will be much more easier on systems (like mine) which have good hardware acceleration support only for h264. so this patch happened.

Even when prioritize H.264, Firefox (browser used by your friend) does not provide support for it. To resolve this, alternative browsers such as Chrome, Edge, Opera, Vivaldi, and Brave are recommended. By enforcing this specification on these browsers, they will utilize video/webm;codecs=h264,opus, but be cautious, as Safari is the only browser that supports the slightly altered video/mp4;codecs=h264,aac (video/mp4).

In the scenario where recording is done using video/webm;codecs=h264, the inclusion of Opus codec remains beneficial for post-processing as outlined in the earlier described situation? Additionally, in long recordings (1h+) the file is downloaded as previously?

Thank you so much!

mohan43u commented 5 months ago

In the scenario where recording is done using video/webm;codecs=h264, the inclusion of Opus codec remains beneficial for post-processing as outlined in the earlier described situation?

Yes, converting audio frames are easier in CPU compare to video frames, converting video frames requires much more processing power (we need lot of CPU cores or we need hw acceleration which uses GPU).

To resolve this, alternative browsers such as Chrome, Edge, Opera, Vivaldi, and Brave are recommended

Yes, I have to make sure we use chromium based browsers if we are planning to record the remote sessions. Its unfortunate Firefox refuse to implement h264/opus in webm container https://bugzilla.mozilla.org/show_bug.cgi?id=1563140 .

miroslavpejic85 commented 5 months ago

Merged, Thank you 4 the contribution.

miroslavpejic85 commented 5 months ago

Hi @mohan43u,

We have relocated the option to prioritize H.264 in the recording settings and added also codecs field in the recording info. Now, everyone is free to customize their preferences and choose H.264 without being compelled to use it universally through the .env file, as was the case before.

commit

miroslavpejic85 / mirotalk