AirPlay module: Needs Apple HLS livestream transcoding to extend range of media AppleTVs can play

[ghost]

We've been porting @butterproject over to @watson's brilliant airplay-protocol and bonjour libraries.

As part of that, I discovered a serious AppleTV streaming problem that affects both WebTorrent and Butter (which uses WebTorrent for streaming).

The issue is being tracked at @butterproject's repo, so to avoid duplication I'll point everyone to the comment with the most relevant summary:

https://github.com/butterproject/butter-desktop/issues/528#issuecomment-267760909

In short: The AppleTVs support an extremely narrow range of containers, along with very specific H264 profiles and audio codecs. It isn't possible to just tell it to stream a ".mkv" file that WebTorrent serves. There needs to be transcoding in the middle. Otherwise the AppleTV fails to stream the video.

Solution (volunteers welcome): Both projects need a suitable Apple HLS (HTTP Live Streaming) proxy to sit in-between WebTorrent and the AppleTV. Such a proxy should be based on FFmpeg (because re-encoding will be needed), and should take care of:

• Generating the main .m3u8 playlist, using the correct machine IP for all URLs (given by the caller to the library), so that we can be sure to bind to the correct network interface. The AppleTV is very picky about IPs.
• Converting the external subtitle file (.srt, .ass, etc) to WebVTT instead.
• Running a small HTTP server which serves the converted subtitle.
• Embedding the URL to the WebVTT subtitle in the .m3u8 playlist, along with the URL to the FFmpeg .ts media chunk sub-playlist.
• Running FFmpeg to transcode all non-H264 video and non-AAC audio to H264/AAC. But if the media codecs are already supported, tell it to use "codec:copy" to avoid wasting CPU. Most media simply need a container change.
• Telling FFmpeg to chunk everything into MPEG2TS Transport Stream (.ts) containers.
• Handling the communication between WebTorrent <-> FFmpeg <-> AppleTV, such as the ability for the AppleTV to seek in the file and have it pass through to WebTorrent to change the position in the stream (perhaps FFmpeg's server handles this communication for us).

Tons of projects have achieved all of this before us, so I know there are reference implementations all over the web (and perhaps even a working Node implementation). As this is a pretty big project, I am listing it on both trackers. I am going on a long trip and won't be around, and the other Butter devs aren't AppleTV users. So volunteers are welcome if anyone is interested in fixing AppleTV support in WebTorrent.

[feross]

Thanks for cross-posting here. If such a project is undertaken, it would definitely benefit us as well. Using ffmpeg would sadly require compiling the app on each platform we want to support -- requiring a lot of VMs to do a single release, but that's acceptable if it solves this problem.

@SteveJobzniak Is it possible to use FFmpeg on a video file that is not completely downloaded? Otherwise, we lose the ability to stream to the Apple TV.

[ghost]

@feross I agree that it's really unfortunate to need FFmpeg. This is one of those times when Apple's walled garden approach really hurts us. They could have supported MKV and DLNA streaming and proper subtitles, but they insisted on AirPlay and MP4/MOV/AppleHLS. "Not invented here" comes to mind. And also "Please buy all your media via our iTunes store instead of streaming your own".

Using ffmpeg would sadly require compiling the app on each platform we want to support -- requiring a lot of VMs to do a single release, but that's acceptable if it solves this problem.

At least for Windows and Mac, it's possible to just tell the user to grab the pre-compiled statically linked snapshot binaries: https://ffmpeg.org/download.html and to place them in the appropriate folder. And then we can just call "ffmpeg.exe" or "ffmpeg" based on whether WebTorrent is running on Mac or Windows. And Linux also has static builds listed, but I'm not sure how well those work since they require certain kernel versions.

The main issue to overcome is ensuring that the user's FFmpeg binary has been built with libx264 and libfdk_aac (the H264 and best AAC encoders), so that we can generate proper output for the AppleTV whenever transcoding is needed. I think it's enough to just call the ffmpeg binary once without any arguments, and look for --enable-libx264 and --enable-libfdk-aac in the "Configuration:" output it will show us.

Is it possible to use FFmpeg on a video file that is not completely downloaded? Otherwise, we lose the ability to stream to the Apple TV.

Yeah. FFmpeg can use live-streams as input and Apple HLS as output. Here's an example from a mailing list which turned up on Google:

https://ffmpeg.org/pipermail/ffmpeg-user/2013-November/018558.html

And there's more info here:

https://trac.ffmpeg.org/wiki/StreamingGuide

It seems like it would be possible to do the following:

• Start FFmpeg, with input set to WebTorrent's livestream server address. And output set as m3u8 and HLS in a folder.
• Do the other things I listed above to generate the master playlist, the WebVTT file, provide HTTP-access to the FFmpeg HLS .m3u8/.ts output folder, etc.
• Serve the master playlist via a Node HTTP server, and tell the AppleTV to connect to that.
• The AppleTV will then load the master playlist, load the WebVTT subtitle, and start streaming from the FFmpeg HLS playlist.

The main unsolved questions are:

• Can the user scrub through the video on their AppleTV if we do the above? Or will FFmpeg just run through and convert the whole input stream from start to finish without any user control?
• What's the latency like? Something like 10 seconds to seek would be acceptable. A minute to seek wouldn't.

If there's trouble with the above approach, perhaps we can tell FFmpeg (perhaps via "FFserver"?) to generate an output .mp4 stream URL with embedded subtitles instead, which should allow the AppleTV to do seeking in the single-file livestream. But that may come with its own problems, since Apple's terrible .mp4 container is an insane container format which puts the metadata at the end of the file and requires the metadata to list all of the keyframe binary offsets in the entire file, so it may not be possible to live-generate such a container without having the complete input file.

It's possible to solve all of this somehow, but I haven't got the time to research it any further. I need to leave and can't stay around. So if someone is interested in helping WebTorrent with this AppleTV issue then contributors are very welcome. There are tons of projects who have already achieved this type of FFmpeg HLS transcoding for the AppleTV, so it doesn't seem that difficult.

Most of the desktop apps that let you stream via AirPlay use FFmpeg internally, and they all let the AppleTV control the playback via the AppleTV remote with very low latency (~8 seconds perhaps), so it looks like it is possible to have low-latency seeking ability in the stream. :-) That's the good news. The bad news is that we need some hero with the available time and energy to do the remaining research and implementation. ;-)

Until then, AppleTV users will have to deal with basically not being able to stream anything since most content is in .mkv format. So if someone out there is an AppleTV user with some programming ability, please don't hesitate to help out! :-)

[feross]

@SteveJobzniak Thanks for the excellent write-up! Waiting for our hero...

[theoephraim]

One more potential issue to watch for is quality. I tried using https://www.strem.io/ as a solution to stream torrent files directly to airplay. It did work (and I assume they are doing something similar to what is described). However I have to say the video/audio quality was terrible - and I checked the original file and it was fine.

I'm going to try to dig into this and see how tough it is.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.