🎓 Getting Started
-
⏯ Demo
-
đź“– API documentation
Wasp-hls is an HLS media player (the library, streaming engine part, not the UI/application part which has to be built on top of it) for the web which:
Relies the most possible on WebAssembly (Written in the Rust language before being compiled).
Runs mostly in a Web Worker (even for media buffering when APIs are available), to reduce the influence an heavy UI can have on playback (and in some situations vice-versa).
Note that this is only a personal project as well as a proof of concept and it is still heavily in development.
To provide media contents at large scale to their customers, most streaming actors (Netflix, Amazon Prime Video, YouTube, Twitch, Canal+, Disney+ etc. you got it), rely on the few same HTTP-based adaptive bitrate streaming protocols: majoritarily MPEG-DASH and Apple's HLS (some rely on only one like Twitch with HLS, others may rely on both depending on the case).
Those protocols all have similar concepts: all expose a central file listing the characteristics of the content (for example, the video qualities, audio tracks, the available subtitles etc.) and allow a client to request the media through small chunks, each containing only the wanted quality and track for a specific time period.
Schema of how HLS basically works, found on Apple's Website which is the one
behind HLS
This architecture allows to:
For HLS specifically, this so-called central file is called the "Multivariant
Playlist" (a.k.a. "Master Playlist") and is in the M3U8
file
format.
HLS also has the concept of secondary "Media Playlists", also as M3U8
files,
which describe specific tracks and qualities.
To allow the implementation of such adaptive streaming media players on the web (and thus with JavaScript), a W3C recommendation was written: the Media Source Extensions™ recommendation, generally just abbreviated to "MSE".
Basically, it builds on top
of the HTML5 <video>
element, adding a new set of browser API accessible from
JavaScript to create media buffers, called SourceBuffer
s, and allowing to push
aforementioned small chunks of media data to it for later decoding.
The role of an HLS media player library like this one is thus to load the Multivariant Playlist, detect which characteristics (bandwidth, quality, codecs, preferred language, accessibility etc.) are wanted and to load the right media data at the right time, then communicating it to the browser through the MSE APIs so it can be decoded.
The Wasp-hls player reading a Multivariant playlist from Twitch. You can see
on the top right the requests performed - mostly of media segments, and some
logs on the bottom right. You can also see a cog logo before the requests' url
indicating that they are all performed in a WebWorker.
This may look relatively simple at first, but there's a lot of potential optimizations to make and special cases to handle which make quasi-mandatory the need to develop separately the media streaming library (the part "understanding" the streaming protocol and pushing chunks) and the application (the part relying on the library and implementing an UI and the business logic on top).
Most people generally mean the latter when they talk about a "player", here, I'm "only" implementing the former and the application has to be developped separately.
The demo page do also implement a UI part, but this is just to showcase the library and is not actually what's exposed by this package, only the library is (though you can copy the demo's code if you want to).
Amongst the main characteristics of this player is that it relies on a WebWorker, to run concurrently with the application, and WebAssembly, to do so optimally. It thus allows to have a theoretically efficient media player, allowing to avoid stalling in the content if the UI do some heavy lifting and vice-versa.
This is even more important when playing what is called "low-latency contents" (contents with a small-ish latency between the recording and the watch-ing, in the few seconds range) which have amongst its characteristics the fact that only very small data buffers are constructed - allowing to play closer to live but more exposed to rebuffering risk if the segment loading pipeline takes more time than expected).
Playing low-latency contents is one of the main goal of this project. On that matter as an amusing note, playing low-latency contents through a media player with a WebWorker + WebAssembly combination is exactly what Twitch is doing, though their players isn't open-source. This one is (it's also able to play Twitch contents if you succeed to work-around their CORS policy, only not with low-latency for now)!
Even without taking into account low-latency contents, I consider WebAssembly to be particularly adapted for an adaptive media player:
Such software generally handle large objects (not only talking about media segments here). The relatively few controls JavaScript offers in terms of memory management can be problematic (I've seen GC-pressure related issues on another player I work on for example).
With WebAssembly and a source language with enough control over memory management, GC-linked issues disappear (because no GC :p!) and the application's memory usage is much more explicit and controlled, which is usually what we want.
Even if there's a lot of input/output and thus the need to interact with JavaScript, the performance profile is completely different than your usual JS front-end application: very few DOM access here, a lot of time is spent on parsing/transmuxing and other logic that can be completely implemented in WebAssembly.
Because this is just the player part, the HLS content has to be prepared - through what we call the packaging step - separately by using what's called a "packager".
For example the shaka-packager is a relatively easy to use packager. With it and FFmpeg, you are armed with the right tools to produce HLS contents.
I'm currently working as the lead developper of another, featureful adaptive media player library, the open-source RxPlayer so this is not something totally out of the blue.
The reasons why I started this project are mainly:
to see how IO-heavy logic (like we have here with many requests, media segments streaming, playback observation, network metrics etc.) using web APIs only exposed to JavaScript could be conjugated with WebAssembly and Rust.
to experiment with a big-enough WebAssembly and Web Worker-based library: how should it interact with applications written in JavaScript?
to learn more about the HLS streaming protocol, for which the RxPlayer does not provide first class support (yet?)
to see how I would write a complex beast like a Media player if restarting from scratch (though the situation is very different here, due to the difference in the language used).
to see if there's any real performance and/or memory related advantage (or disadvantage) in relying on WebAssembly for the core logic of a media player, in various situations (multiple players on the same page, large 4k segments, web workers, mse-in-worker).
to play with a Web Worker-based media player, and find out its influence in terms of API definition, synchronization difficulties, performance issues etc.
to work on and improve my Rust skills
It already has a lot of features but there's still some left work:
Type of contents:
EXT-X-PROGRAM-DATE-TIME
tag in media playlist)Worker-related features:
Adaptive BitRate:
Request Scheduling:
Media demuxing:
MSE API and buffer handling:
SourceBuffer
s)QuotaExceededError
after pushing segments (when low
on memory).
This is generally not needed as the browser should already handle some kind of
garbage collection but some platforms still may have issues when memory is
constrained.
Priority: lowTracks:
Miscellaneous:
Playlist tags specifically considered (unchecked ones are mainly just ignored, most of them are not needed for playback):
If you want to contribute or build the Wasp-hls locally, you will need to have nodejs and rust installed.
Then you need to install node dependencies by calling in your shell:
# Install all node dependencies (needs npm, generally installed with nodejs)
npm install
You also need to add the rust wasm32 target and some rust dependencies:
# Add wasm32 target (needs rustup that you most likely took with rust)
rustup target add wasm32-unknown-unknown
# Add wasm-bindgen CLI (needs cargo, generally installed with rust)
cargo install wasm-bindgen-cli
# Optionally, you may also need clippy, for checking Rust code mistakes
rustup component add clippy
The building of the Wasp-hls player may be performed by module, if you just updated one area of the code (the Rust code for example), or as a whole.
To build only the Rust code in src/rs-core/
to its destination WebAssembly
file (build/wasp_hls_bg.wasm
), you can run any of the following commands:
# Build in debug mode, which leads to a bigger file and slower code, but is
# more useful and quicker to build when developping
npm run build:wasm
# Build in release mode, which is the actual delivered result
npm run build:wasm:release
To build only the Worker code in src/ts-worker
to its destination JavaScript
file (build/worker.js
), you can run any of the following commands:
# Build in debug mode, which leads to a bigger file though much easier to debug
npm run build:worker
# Build in release mode, which is the actual delivered minified result
npm run build:worker:release
To build only the code running in the main thread present in src/ts-main
to
its destination JavaScript file (build/main.js
), you can run any of the
following commands:
# Build in debug mode, which leads to a bigger file though much easier to debug
npm run build:main
# Build in release mode, which is the actual delivered minified result
npm run build:main:release
To build only the demo application showcasing the Wasp-hls player, whose code
is present in the (demo/
) directory, to its destination JavaScript file
(build/demo.js
), you can run any of the following commands:
# Build in debug mode, which leads to a bigger file though much easier to debug
npm run build:demo
# Build in debug mode, with a "watcher" rebuilding each time one of its files
# changes
npm run build:demo:watch
# Build in release mode, which is the actual delivered minified result
npm run build:demo:release
Then to perform your tests, you generally want to serve the demo. You can do so with:
npm run serve
If you just want to build the whole Wasp-hls player code, without the demo, you may call:
npm run build:all
If you want to build all that code AND the demo:
npm run build:all && npm run build:demo
Though what you most likely want to do here is build the full code used by the demo to perform your tests, here just write:
npm run build:all:demo
That last script bypass the generation of the build/main.js
file, as the demo
file (build/demo.js
) already includes the content of that file anyway.
To build everything in release mode, for an actual release or for tests in production conditions, write:
# WebAssembly + Worker + Main in release mode
npm run build:release
# If you also want the demo in release mode
npm run build:demo:release
The documentation, written in doc/
may also be built to its final directory
(build/doc
), through the following command:
npm run doc
It may then be served, so it can be read on a web browser, through:
npm run serve
And then requesting the /doc
path.
You're welcome to read the code which should be hopefully documented enough and
readable enough to dive into. The source code of the player is in the src
directory, if you would prefer to work on the demo, it's in the demo
directory, as for the documentation, it's in the doc
directory.
To check the TypeScript types of TypeScript files and their code style with
the eslint
package, you can run:
# Check all TypeScript files in the project
npm run check
# OR, check only the Worker code
npm run check:worker
# OR, check only the Main code
npm run check:main
# OR, check only the Demo code
npm run check:demo
To check the Rust code, with clippy, you can run:
# Check all Rust files in the project
npm run clippy
You also might want to format automatically the code before commiting:
# Format all TypeScript, JavaScript, Markdown and HTML files in the project
npm run fmtt
# Format all Rust code
npm run fmtr
# Do both
npm run fmt