search

peaBerberian / wasp-hls

WebAssembly-based (Rust) & in-worker HLS Media Player for the web
https://peaberberian.github.io/wasp-hls/
MIT License
124 stars 4 forks source link
hls mediasource mediasource-extensions mse player-video webassembly

readme

Wasp-hls's logo Wasp-hls's logo

🎓 Getting Started - ⏯ Demo - 📖 API documentation

Wasp-hls is an HLS media player (the library, streaming engine part, not the UI/application part which has to be built on top of it) for the web which:

  1. Relies the most possible on WebAssembly (Written in the Rust language before being compiled).

  2. Runs mostly in a Web Worker (even for media buffering when APIs are available), to reduce the influence an heavy UI can have on playback (and in some situations vice-versa).

Note that this is only a personal project as well as a proof of concept and it is still heavily in development.

What's this exactly? HLS?

Streaming protocols and HLS

To provide media contents at large scale to their customers, most streaming actors (Netflix, Amazon Prime Video, YouTube, Twitch, Canal+, Disney+ etc. you got it), rely on the few same HTTP-based adaptive bitrate streaming protocols: majoritarily MPEG-DASH and Apple's HLS (some rely on only one like Twitch with HLS, others may rely on both depending on the case).

Those protocols all have similar concepts: all expose a central file listing the characteristics of the content (for example, the video qualities, audio tracks, the available subtitles etc.) and allow a client to request the media through small chunks, each containing only the wanted quality and track for a specific time period.

de18e941-81de-482f-843d-834a4dd3aa71 Schema of how HLS basically works, found on Apple's Website which is the one behind HLS

This architecture allows to:

For HLS specifically, this so-called central file is called the "Multivariant Playlist" (a.k.a. "Master Playlist") and is in the M3U8 file format. HLS also has the concept of secondary "Media Playlists", also as M3U8 files, which describe specific tracks and qualities.

The Media Source Extensions™

To allow the implementation of such adaptive streaming media players on the web (and thus with JavaScript), a W3C recommendation was written: the Media Source Extensions™ recommendation, generally just abbreviated to "MSE".

Basically, it builds on top of the HTML5 <video> element, adding a new set of browser API accessible from JavaScript to create media buffers, called SourceBuffers, and allowing to push aforementioned small chunks of media data to it for later decoding.

The media player library

The role of an HLS media player library like this one is thus to load the Multivariant Playlist, detect which characteristics (bandwidth, quality, codecs, preferred language, accessibility etc.) are wanted and to load the right media data at the right time, then communicating it to the browser through the MSE APIs so it can be decoded.

twitch-wasp The Wasp-hls player reading a Multivariant playlist from Twitch. You can see on the top right the requests performed - mostly of media segments, and some logs on the bottom right. You can also see a cog logo before the requests' url indicating that they are all performed in a WebWorker.

This may look relatively simple at first, but there's a lot of potential optimizations to make and special cases to handle which make quasi-mandatory the need to develop separately the media streaming library (the part "understanding" the streaming protocol and pushing chunks) and the application (the part relying on the library and implementing an UI and the business logic on top).

Most people generally mean the latter when they talk about a "player", here, I'm "only" implementing the former and the application has to be developped separately.

The demo page do also implement a UI part, but this is just to showcase the library and is not actually what's exposed by this package, only the library is (though you can copy the demo's code if you want to).

What's WebWorker and WebAssembly doing with all that?

Amongst the main characteristics of this player is that it relies on a WebWorker, to run concurrently with the application, and WebAssembly, to do so optimally. It thus allows to have a theoretically efficient media player, allowing to avoid stalling in the content if the UI do some heavy lifting and vice-versa.

This is even more important when playing what is called "low-latency contents" (contents with a small-ish latency between the recording and the watch-ing, in the few seconds range) which have amongst its characteristics the fact that only very small data buffers are constructed - allowing to play closer to live but more exposed to rebuffering risk if the segment loading pipeline takes more time than expected).

Playing low-latency contents is one of the main goal of this project. On that matter as an amusing note, playing low-latency contents through a media player with a WebWorker + WebAssembly combination is exactly what Twitch is doing, though their players isn't open-source. This one is (it's also able to play Twitch contents if you succeed to work-around their CORS policy, only not with low-latency for now)!

Even without taking into account low-latency contents, I consider WebAssembly to be particularly adapted for an adaptive media player:

Generating an HLS content

Because this is just the player part, the HLS content has to be prepared - through what we call the packaging step - separately by using what's called a "packager".

For example the shaka-packager is a relatively easy to use packager. With it and FFmpeg, you are armed with the right tools to produce HLS contents.

Why starting this project?

I'm currently working as the lead developper of another, featureful adaptive media player library, the open-source RxPlayer so this is not something totally out of the blue.

The reasons why I started this project are mainly:

What's done

It already has a lot of features but there's still some left work:

Type of contents:

Worker-related features:

Adaptive BitRate:

Request Scheduling:

Media demuxing:

MSE API and buffer handling:

Tracks:

Miscellaneous:

Playlist tags specifically considered (unchecked ones are mainly just ignored, most of them are not needed for playback):

Setup

If you want to contribute or build the Wasp-hls locally, you will need to have nodejs and rust installed.

Then you need to install node dependencies by calling in your shell:

# Install all node dependencies (needs npm, generally installed with nodejs)
npm install

You also need to add the rust wasm32 target and some rust dependencies:

# Add wasm32 target (needs rustup that you most likely took with rust)
rustup target add wasm32-unknown-unknown

# Add wasm-bindgen CLI (needs cargo, generally installed with rust)
cargo install wasm-bindgen-cli

# Optionally, you may also need clippy, for checking Rust code mistakes
rustup component add clippy

Build

The building of the Wasp-hls player may be performed by module, if you just updated one area of the code (the Rust code for example), or as a whole.

The Rust (WebAssembly) code

To build only the Rust code in src/rs-core/ to its destination WebAssembly file (build/wasp_hls_bg.wasm), you can run any of the following commands:

# Build in debug mode, which leads to a bigger file and slower code, but is
# more useful and quicker to build when developping
npm run build:wasm

# Build in release mode, which is the actual delivered result
npm run build:wasm:release

The TypeScript Worker code

To build only the Worker code in src/ts-worker to its destination JavaScript file (build/worker.js), you can run any of the following commands:

# Build in debug mode, which leads to a bigger file though much easier to debug
npm run build:worker

# Build in release mode, which is the actual delivered minified result
npm run build:worker:release

The Main (API) code

To build only the code running in the main thread present in src/ts-main to its destination JavaScript file (build/main.js), you can run any of the following commands:

# Build in debug mode, which leads to a bigger file though much easier to debug
npm run build:main

# Build in release mode, which is the actual delivered minified result
npm run build:main:release

The Demo

To build only the demo application showcasing the Wasp-hls player, whose code is present in the (demo/) directory, to its destination JavaScript file (build/demo.js), you can run any of the following commands:

# Build in debug mode, which leads to a bigger file though much easier to debug
npm run build:demo

# Build in debug mode, with a "watcher" rebuilding each time one of its files
# changes
npm run build:demo:watch

# Build in release mode, which is the actual delivered minified result
npm run build:demo:release

Then to perform your tests, you generally want to serve the demo. You can do so with:

npm run serve

Combinations

If you just want to build the whole Wasp-hls player code, without the demo, you may call:

npm run build:all

If you want to build all that code AND the demo:

npm run build:all && npm run build:demo

Though what you most likely want to do here is build the full code used by the demo to perform your tests, here just write:

npm run build:all:demo

That last script bypass the generation of the build/main.js file, as the demo file (build/demo.js) already includes the content of that file anyway.

To build everything in release mode, for an actual release or for tests in production conditions, write:

# WebAssembly + Worker + Main in release mode
npm run build:release

# If you also want the demo in release mode
npm run build:demo:release

The Documentation

The documentation, written in doc/ may also be built to its final directory (build/doc), through the following command:

npm run doc

It may then be served, so it can be read on a web browser, through:

npm run serve

And then requesting the /doc path.

Update the code

You're welcome to read the code which should be hopefully documented enough and readable enough to dive into. The source code of the player is in the src directory, if you would prefer to work on the demo, it's in the demo directory, as for the documentation, it's in the doc directory.

Check the code

To check the TypeScript types of TypeScript files and their code style with the eslint package, you can run:

# Check all TypeScript files in the project
npm run check

# OR, check only the Worker code
npm run check:worker

# OR, check only the Main code
npm run check:main

# OR, check only the Demo code
npm run check:demo

To check the Rust code, with clippy, you can run:

# Check all Rust files in the project
npm run clippy

You also might want to format automatically the code before commiting:

# Format all TypeScript, JavaScript, Markdown and HTML files in the project
npm run fmtt

# Format all Rust code
npm run fmtr

# Do both
npm run fmt