search

mattdesl / mp4-h264

[project suspended] MP4 + H264 encoding for the browser with WASM
MIT License
223 stars 12 forks source link

readme

:skull: :warning:

Project Suspended

It has come to my attention that MPEG LA often goes after any distributed software (free / OSS / commercial) that encodes H264 video, and charges huge royalties on it. I don't wish to navigate this legal landscape for this module, so all of the binary and source code surrounding the H264 encoding has now been removed from this project, and the development of the project is suspended until further notice.

However:

(Update: See mp4-wasm for a possible solution that does not depend on an H264 encoder.)

What follows is the previous version of the repository, with all the H264 WASM/C/C++ source code and binary files removed.

mp4-h264

Standalone H264 encoder and MP4 muxer compiled with Emscripten into WASM.

✨ Live Demo (redacted)

Current Features:

Possible Future Features:

Contents

Example in a Web Browser

You can import the module directly as an ES Module using <script type="module"> in your HTML. You can use unpkg for easy access, like so:

// Import latest
import loadEncoder from "[redacted]";

(async () => {
  // Load the WASM module first
  const Encoder = await loadEncoder();

  // Create a new encoder interface
  const encoder = Encoder.create({
    width: 256,
    height: 256,
    fps: 30
  });

  // Encode each frame, typically in an animation loop
  for (... each frame ...) {
    const rgba = /* get some Uint8Array of flat RGBA data */;
    encoder.encodeRGB(rgba);
  }

  // Finish encoder to get back uint8 MP4 data
  const mp4 = encoder.end();

  // Optionally convert to URL for <video> tag, download, etc...
  const url = URL.createObjectURL(new Blob([mp4], { type: "video/mp4" }));
  ...
})();

See ./test/simple.html for a working example, or a live demo on CodePen.

If you want to host your own files, copy the following from the build directory: mp4-encoder.js, mp4-encoder.wasm, and optionally mp4-encoder.simd.wasm (if you plan to use it). Save them in a folder (e.g. vendor), then you can import the JavaScript directly:

import loadEncoder from './vendor/mp4-encoder.js`;

Example with SIMD (Browser Only)

Ideally you should use the SIMD version for faster encoding, if the browser supports it (only Chrome at the time of writing, and only via a flag). In Chrome, go to about:flags and turn on #enable-webassembly-simd, then reboot.

In code, you need to detect SIMD — if you try to load SIMD WASM in an unsupported browser you might run into crashes or errors. You can use wasm-feature-detect for this.

import { simd } from "https://unpkg.com/wasm-feature-detect?module";
import loadEncoder from "[redacted]";

(async () => {
  // Detect SIMD
  const isSIMD = await simd();

  // Load the WASM module
  const Encoder = await loadEncoder({
    simd: isSIMD
  });
})();

Example with Node.js

The default main export of this module is a Node.js compatible WASM build, so this works with newer versions of Node.js. Note that there is no SIMD build for Node.js at the moment.

const loadEncoder = require("[redacted]");

(async () => {
  const Encoder = await loadEncoder();
  // ... same API as web ...
})();

Muxing Only

Let's say you already have H264 data and just want to use this to mux it into an MP4 container, then you'll want to use the lower-level Direct API like so:

const loadEncoder = require('[redacted]');
const fs = require('fs');

(async () => {
  const Encoder = await loadEncoder();
  const stream = fs.createWriteStream('file.mp4');

  const mux = Encoder.create_muxer({
    width,
    height,
    // sequential outputs for simpler 'file write'
    sequential: true,
  }, write);

  for (let chunk of readChunksOfH264FromSomewhere()) {
    // malloc() / free() a pointer
    const p = Encoder.create_buffer(chunk.byteLength);
    // set data in memory
    Encoder.HEAPU8.set(chunk, p);
    // write NAL units with AnnexB format
    // <Uint8Array [startcode] | [NAL] | [startcode] | [NAL] ...>
    Encoder.mux_nal(mux, p, chunk.byteLength);
    Encoder.free_buffer(p);
  }

  // this may trigger more writes
  Encoder.finalize_muxer(mux);

  function write (pointer, size, offset) {
    const buf = Encoder.HEAPU8.slice(pointer, pointer + size);
    stream.write(buf);
    return 0;
  }
})();

See ./test/node-mux.js for a full example.

WebCodecs

If your environment supports WebCodecs, you can use it to achieve much faster encoding (i.e. 3 times faster). This is pretty similar to using "Mux Only", see the ./test/webcodecs.html demo for details.

At the time of writing, WebCodecs is behind a command line flag on Chrome Canary:

API Structure

There are two APIs exposed by this module:

Simple API

promise<Encoder> = loadEncoder(config = {})

Loads the WASM encoder and returns the Encoder (web assembly) module. Options:

(async () => {
  const Encoder = await loadEncoder({
    getWasmPath (fileName, scriptDir, isSIMDRequested) {
      return '/my-wasm-files/' + fileName;
    }
  });
})();

encoder = Encoder.create(opt = {})

Creates a new encoder interface with options:

encoder.encodeRGB(pixels)

Encode a frame of RGB(A) pixels, a flat Uint8Array or Uint8ClampedArray. This will use the same stride that you specified when creating the encoder (default 4). Use a stride of 3 if you only have flat RGB bytes, or a stride of 4 if you have RGBA bytes (which is what you get from <canvas> APIs).

This will convert RGB to YUV natively (in WASM) in order to write a single H264 frame of video.

encoder.encodeYUV(yuv)

Encodes an image that is already in YUV layout, saving the need to do the RGB > YUV conversion step.

This may be useful if you already have YUV data, or if you are using a hardware solution to convert RGB to YUV, or if you are multithreading and your renderer thread is idling (in which case you may benefit in memory and performance by converting RGB to YUV in the renderer thread, and then sending that to the encoder thread).

See ./test/util/RGBAtoYUV.js for an example of how to go from RGB to YUV from JavaScript.

uint8 = encoder.end()

Ends the encoding and frees any internal memory used by this encoder interface, returning a final Uint8Array which contains the full MP4 file in bytes. After calling end(), you can no longer use this interface, and instead you'll have to create a new encoder.

uint8 = encoder.memory()

Alias for accessing Encoder.HEAPU8 - note this storage may change as memory is allocated, so you should always access it directly via encoder.memory() rather than storing it as a variable.

ptr = encoder.getRGBPointer() (experimental)

Allocates if necessary, then returns a pointer into the encoder's internal uint8 memory for an RGB(A) buffer (created using stride option).

This might be useful if you have an API that writes pixels directly into an existing buffer with an offset, such as WebGL2's readPixels, to avoid a second data copy.

const ptr = encoder.getRGBPointer();

... render each frame ...
  // read pixels directly into WASM memory
  gl.readPixels(... options ..., encoder.memory(), ptr);
  // trigger an encode from memory
  encoder.encodeRGBPointer();

ptr = encoder.getYUVPointer() (experimental)

Allocates if necessary, then returns a pointer into the encoder's internal uint8 memory that can be used for faster YUV image transfer. See above.

encoder.encodeRGBPointer() (experimental)

Assuming RGB(A) bytes have already been written into the memory heap at the RGB(A) pointer location, it will then encode those bytes directly.

encoder.encodeYUVPointer() (experimental)

Assuming YUV bytes have already been written into the memory heap at the YUV pointer location, it will then encode those bytes directly.

Direct API

Instead of returning an interface with functions, the direct API returns pointers into Encoder.HEAPU8 memory, and then all functions act on the pointer into the encoder's struct.

const outputs = [];

const settings = {
  sequential: true,
  width: 256,
  height: 256
};

function write (ptr, size, offset) {
  // take a chunk of memory and push it to stack
  // offset is ignored as we are using { sequential }
  const buf = Encoder.HEAPU8.slice(ptr, size);
  outputs.push(buf);
  return 0; // 0 == success, 1 == error
}

const encoder_ptr = Encoder.create_encoder(settings, write);

const stride = 4; // stride of our RGBA input
const rgb_ptr = Encoder.create_buffer(width * height * stride);
const yuv_ptr = Encoder.create_buffer((width * height * 3) / 2);

for (.. each frame ..) {
  const pixels = /* Uint8Array */;
  Encoder.HEAPU8.set(pixels, rgb_ptr);
  Encoder.encode_rgb(encoder_ptr, rgb_ptr, stride, yuv_ptr);
}

// finish & free memory
Encoder.finalize_encoder(encoder_ptr);
Encoder.free_buffer(rgb_ptr);
Encoder.free_buffer(yuv_ptr);

const mp4File = Buffer.concat(outputs);

Tips for Speed

Tips for File Size vs. Quality

The default setting produces high quality video, but with very high file sizes (note: the defaults may change as this module is still a WIP).

If you are concerned about filesize, you should set the kbps parameter instead of using a constant quantizationParameter (which defaults to 10, i.e. the best quality).

First, determine what you think would be an acceptable output size based on your video. Let's say you're encoding a 10 second animation and want it to be around 10MB in file size.

const duration = 10; // in seconds
const sizeInMBs = 10; // desired output size
const sizeInKilobits = sizeInMBs * 8000; // conversion

// this is the value passed into the encoder
const kbps = sizeInKilobits / duration;

const encoder = Encoder.create({
  width,
  height,
  fps,
  kbps
})

This should produce a video file around 10 MB, with a quality to match. To improve quality, you can try to lower the { speed } option, or lower the upper bound of { qpMax } from 51 to something smaller (although doing so may increase file size beyond your desired amount).

More Advanced Examples

See the ./test folder for examples including SIMD, multi-threading, fast pixel access, and more.

TODOs

Contributing

If you think you can help, please open an issue.

Building from C/C++ Source

NOTE: this has now been removed from the project

Currently you need Emscripten with SIMD supported. Then modify the paths in ./src/mp4-encoder/build.sh to your emcc build. Then, from this package's folder, run:

./src/mp4-encoder/build.sh

Credits

This was originally based on h264-mp4-encoder by Trevor Sundberg, but it's been modified quite a bit to reduce the size (~1.7MB to ~150KB), use a different architecture for faster encoding and streamed writing, and use different C libraries (minimp4 instead of libmp4v2).

Also see minimp4 and minih264 by @lieff, which are the lightweight C libraries powering this module.

License

MIT, see LICENSE.md for details.