WebAssembly encoder under the binary license?

[tangobravo]

In modern browsers, Javascript is able to create in-memory file objects which can then be saved to a user's device or shared via the WebShare API. Web pages can also access video from the user's cameras or generate video-like content from WebGL entirely client side.

The Media Recorder API exposes encoders to the browser, but this is not currently implemented and working on all browsers, and codec and format support is not universal.

With a WebAssembly (wasm) build of an h.264 encoder (and mp4 muxer) it would be possible for webpages to generate and save universally-viewable mp4 videos to user's devices entirely client-side. This has bandwidth, privacy, and availability advantages when compared to a server-side transcoding solution.

The biggest downside to this approach is the patent licensing requirements for h.264; downloading a web assembly build of an encoder is likely to be considered as distribution of an encoding product and would therefore incur licensing fees for every visitor to the site. For most free-to-access sites this licensing requirement prevents this approach from being commercially viable.

If a web assembly build was contributed to openh264, would Cisco consider providing and hosting binary web assembly builds under the openh264 binary license? I believe a web page would be able to comply with the terms of that binary license (including attribution and an ability to disable the feature) and this could unlock universal client-side mp4 encoding on the web, which I think would be of great value to the web development community.

If Cisco are open to providing binary distribution for a wasm build then I would happy to help contribute to the implementation.

[mattdesl]

I'd like to add support to this as well — I managed to get a very fast client-side H264 encoder working with WASM + SIMD + JS, but unfortunately had to suspend the project due to licensing concerns (see mp4-h264).

Fast and robust client-side encoding could enable a lot of new types of web applications, for example user-generated MP4/H264 files (video editors, conversion & CG tools, etc). Currently, it's nearly impossible to achieve H264 encoding across browsers without a costly ffmpeg backend.

If openh264 could include a few .wasm binaries generated by Emscripten (such as an encoder, decoder, and combined build), this would help alleviate the issue and allow for more H264 applications on the web.

@tangobravo – maybe you've already considered this, but if Cisco were to support such an initiative, how would the wasm files actually be distributed / downloaded? It seems like it would have to be fetched() at runtime (from JavaScript, in the client) directly from a Cisco CDN, to ensure that developers aren't downloading and re-distributing the wasm binaries from their own websites.

[tangobravo]

Yup, Cisco would need to "distribute" the module so it would need to come from their CDN. That's how the native OpenH264 distributions work AFAIK (the browsers that use it don't include it in their download but get it from Cisco when they are first started I think).

This repo seems pretty inactive - my hope was if someone contributed build scripts and any necessary modifications to the code for a wasm build then Cisco would agree to host the resulting .wasm and .js files in the same way they do for the native builds. Without any comment from them so far though, I'm not hugely motivated to spend time working on this.

Some unrelated info on other aspects of client-side video encoding:

AAC seems a little easier - limiting to AAC-LC which was spec'ed in 1997 means any relevant patents have now expired. Most actual AAC content is AAC-LC anyway, and RedHat have maintained a fork of an AAC library with all non-LC features stripped out: https://src.fedoraproject.org/rpms/fdk-aac-free

It's another few years until patent expiration for h264 unfortunately.

In Chromium-based browsers it's possible to get h264 encoding from MediaEncoder, just in a webm container. Re-muxing in js/wasm should be possible to convert that to mp4 without needing to ship an h264 implementation at all. There are also options to get callbacks with the raw h264 data with the "Insertable Media using Streams" API. More info on all those Chromium-specific options on this bug: https://bugs.chromium.org/p/chromium/issues/detail?id=1072056#c8

[sijchen]

my hope was if someone contributed build scripts and any necessary modifications to the code for a wasm build then Cisco would agree to host the resulting .wasm and .js files in the same way they do for the native builds.

Yes if there are PRs to fit this need we can merge that and try to generate the binaries. Not sure how much it takes to set up the build environment for WASM though.

[creativefctr]

One question I didn't understand is that if we use other encoders like OpenH264, do we still have to pay the license fees to MPEG LA?

[tangobravo]

Patent license fees would be due if you distribute an "encoding product". The idea here is if Cisco distribute the wasm binaries of OpenH264 then they are the ones who are doing the distribution, so they take on the responsibility for the applicable licensing fees.