Evi/mp3 support

[erwanvivien]

I made a repro that you can test with yarn && npm run dev

output.tar.gz also available here: https://github.com/erwanvivien/canvaskit-question/tree/evi/mp4-from-saved-chunks

The code is this

import "./style.css";

import {
  ArrayBufferTarget as MP4ArrayBufferTarget,
  Muxer as MP4Muxer,
} from "mp4-muxer";

// The file contains your MP3 chunks
import packets from "./mp3chunk.json";

const base64toUint8Array = (base64: string): Uint8Array => {
  const binaryString = window.atob(base64);
  const bytes = new Uint8Array(binaryString.length);
  for (let i = 0; i < binaryString.length; i++) {
    bytes[i] = binaryString.charCodeAt(i);
  }

  return bytes;
};

const main = async () => {
  const target = new MP4ArrayBufferTarget();
  const mp4muxer = new MP4Muxer({
    target,
    fastStart: "in-memory",
    video: undefined,
    audio: {
      codec: "mp3",
      numberOfChannels: 2,
      sampleRate: 48000,
    },
  });

  let isFirst = true;
  for (const packet of packets) {
    const encodedAudioChunk = new EncodedAudioChunk({
      type: isFirst ? "key" : "delta",
      data: base64toUint8Array(packet.data),
      duration: Number(packet.duration) * 1e6,
      timestamp: Number(packet.timestamp) * 1e6,
    });
    isFirst = false;

    mp4muxer.addAudioChunk(encodedAudioChunk);
  }

  mp4muxer.finalize();

  const blob = new Blob([target.buffer], { type: "video/mp4" });
  const url = URL.createObjectURL(blob);

  console.log(url);
};

main();

with this file for mp3chunk.json

mp3chunk.json

[Vanilagy]

Thanks for the PR. Please give the PR a proper title and description first, then I'll check it out!

[erwanvivien]

I still need time to fix things, it seems like audio exported file doesn't work on QuickTime player

Edit: It doesn't work for an FFMpeg exported file too 🫡

[erwanvivien]

I think the PR is in fact OK, but after testing different players, some of them play the audio some do not

On my device (MacOS 12):

• 🔶 Slack (will play it once, then might not play it again, though Slack does some reencoding)
• ❌ QuickTime (after and including MacOS 13 it works)
• ✅ Chrome
• ✅ Instagram upload
• ❌ Twitter upload

I know that for a long time QuickTime was not able to play MP3 frames in an MP4, see https://stackoverflow.com/a/28778982 https://stackoverflow.com/a/34394517 https://old.reddit.com/r/ffmpeg/comments/c8z17w/comment/esuur48/

======

The only problem I have is that the "mp3 " box seems weird to me, when I encode with FFMpeg using liblame for the MP3 encoding, I'm getting the normal stsd with mp4a boxes 🤔 And when I try that with your already existing boxes it fails miserably

======

I will not be pushing the PR as I was not aware that AAC encoding was available on all Chrome platforms. I only referred to this Chromium code to see wether or not AAC encoding was possible

In fact I will close the PR and open a new one without the MP3 commit (e68c0b6bcdd30350a272d8b9c6d6b407c556e8de) just to push the few made improvements