[HELP] Generated MP4 files does not work on Whatsapp Web

[TiagoSilvaPereira]

Hello, I hope you are well!

I'm using canvas-record to record my canvas and save a video to MP4 using AVC1 codec (which internally uses mp4-muxer to generate the mp4 file). The mp4 generated files works well, but seems they don't work with Whatsapp Web (I think this is a problem with codecs, although the recommended codec to use with Whatsapp is ACV1).

I'm still not sure what causes this problem, I've already tried several profile, level, latencyMode, etc. settings. The MP4 videos are generated correctly, and I can watch them through the video player or in the browser, but when I send them via Whatsapp Web, a preview is not generated, and the videos are only sent as attached files.

I will attach two examples of similar videos. The first one works on Whatsapp (it was converted from webm to mp4 on a remote server), and the second does not (it was generated directly in mp4 using canvas-record).

First video, converted from webm to mp4 using ffmpeg on a server (works on whatsapp web): https://github.com/dmnsgn/canvas-record/assets/11933789/80b6745b-cce3-42fa-aa98-bd8670a95d50

Second video, converted using webcodecs on the browser using canvas-record (does not work on whatsapp web): https://github.com/dmnsgn/canvas-record/assets/11933789/1531e1a7-59be-4418-b079-2e71477bea20

Seems the only relevant setting that is different between those videos is the Codec ID, which is mp41 for video 1 and mp42 for video 2. But I really don't know how to change this using mp4-muxer. I tried to locate it inside the library code and options, but without success.

My question is: Is there any way to change the video from mp42 to mp41 before saving it, so that I can make it work with Whatsapp?

Some informations about both videos:

Visual comparison

Video 1 - Ffmpeg

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '.\video1.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.29.100
  Duration: 00:00:06.56, start: 0.000000, bitrate: 6494 kb/s
  Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 898x584 [SAR 1:1 DAR 449:292], 6492 kb/s, 16 fps, 16 tbr, 16384 tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]

Video 2 - Webcodecs

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '.\video2.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: mp42isom
    creation_time   : 2023-10-11T18:34:44.000000Z
  Duration: 00:00:06.57, start: 0.000000, bitrate: 2158 kb/s
  Stream #0:0[0x1](und): Video: h264 (Baseline) (avc1 / 0x31637661), yuv420p(progressive), 954x604, 2155 kb/s, 60 fps, 60 tbr, 90k tbn (default)
    Metadata:
      creation_time   : 2023-10-11T18:34:44.000000Z
      vendor_id       : [0][0][0][0]
      encoder         : JVT/AVC Coding

[Vanilagy]

Thanks for the report! I'll look into this when I find the time, which should be this weekend :) It's obviously not good if files can't be sent over WhatsApp Web, as that's quite a common place to share videos these days. It usually means there's something still sus about the files I output.

However, what I can say is that I do find it weird how the major brand infers to mp42 - don't think that string appears in my library. Have you played around with the ftyp box in src/box.ts? That's where all these brands are listed.

[TiagoSilvaPereira]

Hi @Vanilagy thank you very much for your attention. I'll check this ftyp setting to see if it works

[Vanilagy]

After doing some experiments, I've come to the conclusion that there's something fundamentally wrong with the mp4-muxer-encoded video file you sent - not just a codec or brand mismatch. In the macOS Finder, the two video files already appear different:

And the two behave very differently regarding seeking, too. The left (FFmpeg) version seeks just fine, while the right version is all glitchy:

https://share.cleanshot.com/ZWDW86RM

So I think the video was not created correctly by mp4-muxer - my initial guess would be that canvas-record is doing something wrong.

To verify, I actually reencoded the video using mp4-muxer myself, by playing back the correct video, recording it and piping it into the muxer.

The code I used to generate the file

```html ```

This is the resulting video: https://github.com/Vanilagy/mp4-muxer/assets/1696106/16d2c5a2-ce12-46ae-b023-8da6b552505e

This video plays perfectly on my machine, just like the FFmpeg version does. Can you test sending this version over WhatsApp Web?

---

Anyway, all this makes me believe canvas-record is not using this library fully correctly and therein lies the bug. Are you affiliated with canvas-record at all and know where I should look in the code, or should I dig around a bit myself?

[Vanilagy]

Okay, after a quick inspection of the code in canvas-record, it looks pretty correct to me - maybe this is a writing-related bug with this library. I'm not quite sure how canvas-record works, but it seems like it provides two modes of writing the file - one "in-browser" (which then probably offers a download button), and one that writes to disk directly. Which one of these did you test?

[TiagoSilvaPereira]

Hi @Vanilagy thank you very much for spending your time looking at this issue.

At first, I thought the problem was in the canvas-record library. I tried everything without success. Then it occurred to me to try to upload a video generated directly with mp4-muxer from the Demo page (https://vanilagy.github.io/mp4-muxer/demo), and the same problem occurred. It was at this point that I decided to open an issue here.

Also, I just tested the video you sent, but it remains the same behavior (no preview on WhatsApp web).

About your question, I'm using the default option from canvas-record which downloads the video directly from Browser using mp4-muxer.

Best regards, Tiago

[Vanilagy]

I think I've found the issue! The files generated by mp4-muxer place the moov atom, which contains all the metadata, at the end of the file. This is a common practice because it greatly simplifies writing the file, but some pieces of software check only the start of the file for metadata. Placing the moov atom at the start of the file is known as "Fast Start" in the MP4 space.

Here's your video reencoded using mp4-muxer with the moov atom at the start (with hacked-together code): https://github.com/Vanilagy/mp4-muxer/assets/1696106/af10d575-8609-4736-99ad-1cf6d6d2a919

Perfectly embeds and plays in WhatsApp Web for me:

Anyway, seems like I'll need to add a Fast Start option to this library, which will take a bit of work. I'll get back to you when there's progress. If you have the capacity, a donation would be greatly appreciated - but obviously not required :) https://ko-fi.com/vanilagy

[TiagoSilvaPereira]

@Vanilagy thank you very much for checking this, I tested it here, and it worked well on my WhatsApp Web too

I'll get back to you when there's progress. If you have the capacity, a donation would be greatly appreciated - but obviously not required :)

It would be a pleasure

[Vanilagy]

Thank you so much for your generous donation! <3

---

After a good amount of work, I've released v3.0.0 - as you can see in the updated README, there's now a fastStart option which can be used to place metadata at the start of the file. This is also now enabled by default in the demo, so please try using the demo now and dragging it into WhatsApp Web.

What's funny is that during the implementation I've actually found the true cause for why the files didn't use to work on WhatsApp Web: It had nothing to do with Fast Start, it was actually that the mdat box which holds all media data was using the "large size" format, which uses an additional 8 bytes to specify its size. I do this to support files larger than 4 GiB. Whatever parser WhatsApp was using, it did not support this larger box size definition even though it is clearly included in the base spec. In the new update, I've improved on this behavior too: I now only use the large box size when the box is larger than 4 GiB.

You should talk to the creator of canvas-record to make sure they update their version of mp4-muxer to integrate with all the new features.

[TiagoSilvaPereira]

Hi @Vanilagy thank you very much, your work is incredible. I just tried the demo page and it is working 👏 👏 👏 I'll create a fork of canvas-record and recompile it using the new version of mp4-muxer.