Needs new Airplay library for Apple TV 4 and subtitle support

[aitte2]

The airplay module in Butter is very, very bare-bones: src/app/lib/device/airplay.js (no subtitles, no scrubbing support, etc)

I can see that Butter is currently using "airplay-js": https://github.com/guerrerocarlos/node-airplay-js

This "airplay-js" library cannot find my AppleTV4, and others are saying that Butter/PT has never been reliable at Airplay. The library doesn't seem very good. It has lots of unanswered, years-old tickets about being unable to find devices.

Here are some potential replacement options: https://www.npmjs.com/browse/keyword/airplay

Either way, please keep this ticket open. The current airplay library in Butter is awful and will sooner or later need replacing or patching, with something reliable at finding devices.

[amilajack]

zfkun/node-airplay relies on a global installation of ffmpeg so I dont think it would be considered. benvanik/node-airplay seems like the better choice.

[aitte2]

@amilajack I had a look at the NPM airplay list, and saw that the @watson guy is super prolific, documenting every AirPlay variant and creating lots of libraries for clients and servers. A true AirPlay genius. His latest release was 4 months ago. And he seems entirely JS-based (no native code), by implementing mDNS and stuff in JS. He even wrote "airharvest" to check for differences in AirPlay versions, so he seems very dedicated to doing the job right!

He has a couple of things of great interest:

https://www.npmjs.com/package/airplay-protocol (a complete implementation of speaking to an AirPlay device) https://github.com/watson/airplay-protocol https://www.npmjs.com/package/bonjour (a bonjour/mDNS implementation in JS, for finding AirPlay devices) https://github.com/watson/bonjour

And, he has a complete implementation of a program that finds an AirPlay device and streams to it: https://www.npmjs.com/package/airplayer https://github.com/watson/airplayer

The airplayer code is very simple and uses his bonjour and airplay-protocol packages. That code could easily be hacked apart and implemented in Butter.

My vote is for @watson to jump on board in this discussion and explain his projects. Seems like they could be integrated very nicely into Butter, and they may solve all the AirPlay issues that Butter currently has.

[amilajack]

@aitte2 Do you have an airplay device to test on? If you do, can you try testing out the module to see if it works. I've had terrible experiences with these kinds of modules

[aitte2]

@amilajack I have an Apple TV 4. I'll get the airplayer package and see how it works out.

[amilajack]

Great! Also I updated my comment ^

[aitte2]

@amilajack Nice! I just noticed that the "airplayer" package is both a CLI app and a programming module. Check the small code example at the top: https://www.npmjs.com/package/airplayer

The "list" object is the bonjour discovery object, and the "player" object corresponds to airplay-protocol. So no need to rip apart "airplayer". Instead, we can just depend on "airplayer" and let it wrap the bonjour and airplay-protocol details if they ever change in the future.

So Butter just needs to do this:

var airplayer = require('airplayer')
var airplayBonjourDiscovery = airplayer()

airplayBonjourDiscovery.on('update', function (player) {
    console.log('Found new AirPlay device:', player.name)
    // now just use the "airplay-protocol" player object, such as: player.play(url)
})

So I tried the code above, and this is the result: "Found new AirPlay device: Apple TV Bubbles"

That's better than Butter/PT ever did for me with the old libraries! Now on to the streaming test... I'll get a video file and try it.

[aitte2]

Wow! I am streaming the "I Am Legend" 1080p trailer to my Apple TV! I can even pause and scrub on the Apple TV to control the playback via the Apple remote control! And it sends "preview screenshots" of the scrub position to the Apple TV, so that I can see where the scrubbing will land!

This library kicks ass and is written entirely in JS. @watson you are a genius.

Looks like we have found our replacement for AirPlay support in Butter now!

[amilajack]

Sounds good! Also what i love about this module is that it has tests. Don't think i've seen other casting modules with tests. Also thinking of making a PR to add Appveyor support (for windows).

[amilajack]

Also on a side note, are you willing to test an experimental pct client of mine: http://github.com/amilajack/popcorn-desktop-experimental

[aitte2]

@amilajack Sorry I can't try popcorntime due to legality issues with my ISP. Your GUI looks very clean though!

I am working on implementing @watson's airplay-protocol and bonjour modules in butter-desktop. The "airplayer" wrapper was a bit too thick, and obscured details. So I am using the modules directly instead.

[losh11]

+1

Older versions of popcorn-time, every 1 out of 20 sessions would have issues with AirPlay streaming where the session would time out in the middle of streaming. This would mean that the user would not be able to scrub back to the point where the session abruptly ended - using the remote to forward the streamed video would sometimes slowly forward (at x1) when it should forward at max speed.

I would be willing to try your client @amilajack but I don't have an Apple TV4, but I do have a TV3. Let me know if I can help in testing.

[aitte2]

Alright... I've used the new libraries to massively improve AirPlay on Butter/PT, and it can now detect the AirPlay devices perfectly, and can start streaming to them. But there are other, deeper issues in Butter/PT. For that reason, I'll contribute the finished code as-is, as well as a list of what needs to be done, so that others can carry on the work:

Step 1: Edit "package.json" to swap to the new library.

Remove the awful "airplay-js" as a dependency. Add the following dependencies instead:

"airplay-protocol": "^2.0.2",
"bonjour": "^3.3.1",

Step 2: Replace all contents of "src/app/lib/device/airplay.js" with the following code instead. This new code is feature-complete and is ready to be used. It handles all functions that the old code tried to handle, but this one actually works. Backup link: http://pastebin.com/Et6xmaLw

(function (App) {
    'use strict';

    var Bonjour = require('bonjour'),
        AirPlayProtocol = require('airplay-protocol');

    var bonjourBrowser = new Bonjour(),
        collection = App.Device.Collection;

    var makeID = function (bonjourService) {
        // strong ID since there can only be 1 bonjour service per host+port
        var uniqueDeviceId = bonjourService.host + '_' + bonjourService.port;
        return 'airplay-' + Common.md5(uniqueDeviceId);
    };

    var AirPlay = App.Device.Generic.extend({
        defaults: {
            type: 'airplay',
            typeFamily: 'external'
        },

        _makeID: makeID,

        initialize: function (attrs) {
            this.device = attrs.device;
            this.attributes.id = this._makeID(this.device.service);
            this.attributes.name = this.device.service.name;
            this.attributes.address = this.device.service.host;
        },

        play: function (streamModel) {
            var self = this;
            var url = streamModel.get('src');
            this.device.player.play(url, function (err, res, body) {
                if (err) {
                    // FIXME: optionally do something on failure to play
                }
            });
        },

        stop: function () {
            this.device.player.stop();
        }
    });

    // start scanning for AirPlay services and their status changes
    win.info('Scanning: Local Network for AirPlay servers');
    var airplayBrowser = bonjourBrowser.find({ type: 'airplay' });

    // AirPlay service answered our search query or broadcasts going up
    airplayBrowser.on('up', function (service) {
        var model = collection.get({
            id: makeID(service)
        });
        if (!model) {
            // we don't already know about this host+port combo, so add it!
            var device = {
                service: service,
                player: new AirPlayProtocol(service.host, service.port)
            };

            collection.add(new AirPlay({
                device: device
            }));
        }
    });

    // AirPlay service broadcasts going down
    // this usually only happens when turning off AirPlay in the ATV settings
    airplayBrowser.on('down', function (service) {
        var model = collection.get({
            id: makeID(service)
        });
        if (model) {
            model.destroy();
        }
    });

    App.Device.AirPlay = AirPlay;
})(window.App);

Step 3: Now, about those deep problems in Butter/PT...

3a: Butter defaults to using the IPv6 address as the streaming hostname: ""[%cINFO%c] Available IPs: [\"fe80::6233:4bff:fe11:a4de\",\"10.0.1.167\"]"" Butter picks the "fe80" name. My AppleTV4 refuses to connect if I am telling it the IPv6 stream URL. So for now, I had to manually run "networksetup -setv6off Wi-Fi" to turn off IPv6 to get Butter to advertise a IPv4 streaming URL instead.

As a permanent fix, Butter must be changed to always pick an IPv4 address if one is available, since IPv4 is much more reliable and consistently working than v6 (which usually has all kinds of routing and firewall issues). This necessary change applies to all streaming methods, not just the AirPlay one. It's always safest to bind to IPv4 if available.

Here's a code-snippet which gets the IPv4 IP, if you want to use the "internal-ip" library: var ip = require('internal-ip').v4()

A proper fix would look for a IPv4 address first, and then use IPv6 only if there is no IPv4 available.

3b: Even on IPv4, something in Butter is preventing streaming from actually starting on the AppleTV4. I can see that the new libraries work perfectly (I tried them standalone, without Butter earlier, and was able to stream a 1080p movie trailer perfectly). So the problem is in Butter. The AppleTV4 tries to connect to the stream (i.e. "http://10.0.1.167:40830/") and the TV screen flashes black for half a second, but then it disconnects from the stream. That's some problem in Butter itself.

Possible reasons for the Butter problem (which would have happened with the old AirPlay library too, since both libs are unrelated to streaming and are simply telling the AppleTV to start streaming a certain HTTP URL): 1. Perhaps the Butter server needs to give the stream a filename, like "http://10.0.1.167:40830/stream.m4v"? 2. Perhaps we aren't sending proper Content-Type MIME and Content-Length headers from Butter's media server. (To see what headers should be supported by a confirmed-working, AppleTV-compatible streaming server, look at: https://github.com/watson/airplayer/blob/master/bin.js)

Step 4: After doing all of the above (switching to the new libraries, making Butter select IPv4 by default, and fixing the MIME/filename issues that are preventing a stream connection), there's just one thing left to do: Read the airplay-protocol API (https://github.com/watson/airplay-protocol) and implement more of the streaming functions if necessary, such as the ability to scrub via Butter's GUI and have the AirPlay device automatically seek there (see chromecast.js for example implementations of all those extra functions). I would have done all of that too, but since Butter's HTTP streaming server itself isn't compatible with the AppleTV4 yet, I had to stop here.

[aitte2]

So in short: Use the new libraries and the new code I posted. The new code I provided for Butter correctly discovers AirPlay devices using Bonjour (whereas the old "airplay-js" didn't see them at all!), and it perfectly sends the player play/stop commands. So it's a direct improvement against the old code.

The fact that streaming just flashes for half a second and then aborts on the AppleTV is a SEPARATE issue, which will require fixes to Butter's HTTP streaming media server itself. I have confirmed that the new airplay+bonjour libraries work perfectly when used outside of Butter. So the Butter streaming server is the next problem to tackle. And I can't help with that since I'm not involved deeply enough in this project to know where to begin. Good luck. :)

[aitte2]

@losh11 Feel free to try the new code above. The only thing that it alters is the Bonjour AirPlay device discovery (it's a very good Bonjour mDNS implementation which now actually works and finds the AirPlay devices), as well as changing the way it sends the AirPlay "start streaming a URL" command. Both are much more robust now. The new "src/app/lib/device/airplay.js" code is done and all that remains is to fix Butter's streaming server.

I've independently verified that the new library's "start streaming a URL" command works and was able to stream (via this code: https://github.com/watson/airplayer/blob/master/bin.js), so the remaining problems are totally within Butter (the IPv6 issue and the lack of proper streaming server headers).

PS: A separate issue worth mentioning here so that testers don't fall victim to it: If you're using a VPN, make sure your computer is configured to only route internet (not local) traffic over the VPN, otherwise you won't see your AppleTV via any of the libraries, since the local mDNS traffic will not reach you.

[aitte2]

Well, one step at a time... we'll get there in the end. :) Hopefully someone else with an AppleTV can take over and debug why Butter's HTTP media server isn't working. I've exceeded my 5 hours of time allotment and need to go. Hey, at least the new code finds the devices and tells them to start streaming - unlike the old code which did nothing. :)

So at least the ball is rolling now! Hopefully someone picks it up. ;) Take care!

[aitte2]

Oh and @watson, thanks a lot again for your wonderful libraries. They must have taken a lot of time to test and perfect and I can see how thorough you are! Perhaps you'd be interested in helping the Butter project figure out why their HTTP media server isn't compatible with the AppleTV4? Your own "airplayer" server project works perfectly. Getting AirPlay to work in Butter would be excellent and would make a lot of users happy.

And perhaps you also know how to send the "vtt" subtitle stream URL to the AirPlay server? If that could be achieved with/added to your library, then users in other countries will even have subtitle support. :) I know that AirPlay itself supports VTT subtitles (and even settings such as text size, font, etc), so it can be done. Perhaps it's supposed to be sent to the server via the ".property()" setter in your library?

[amilajack]

@losh11 Great! We have a getting started guide. If you dont want to install and build you can always go straight to the releases.

[watson]

Hi guys, thanks for considering using my airplay modules - much appreciated!

I'd love to help out if possible, so if any issues arise (bugs, missing features etc), please reach out 😃

[xaiki]

hey @aitte2 your work is very apreciated, whenever you feel like it please send us a PR, for your IP issues it looks like the solution should be that the airplay2 module should have an option to list v4 addresses first (instead of forcing all of butter to use v4 by default), @watson what's your take on this ?

[watson]

@xaiki It's been a few months since I played with this my self, so I can't remember all the details off the top of my head. But I just had quick look at the code posted above and it seems ok.

Regarding subtitles: Currently the airplayer module streams raw video to the Apple TV and implements seeking using HTTP range queries. As far as I know, the only way to support subtitles is to package the video inside HLS, which is a bit more work. I haven't looked into VTT, but I'd imagine it would still require HLS as the wrapper.

I'll see if I have time to look into adding support for it soon.

[xaiki]

ok, i see, the issue is a lot of code added by @ddaf in lib/device/generic.js that will mangle your ips, and try to find the 'best fiting one'... you can correct that again into your play function, but it probably should go onto an device method so that you can override the selection.

[watson]

I had a look at adding subtitles yesterday, and this is my findings so far:

Adding subtitles to an mp4 container

It seems to be possible to add subtitles to mp4 containers without using HLS/m3u8. The mp4 container format consists of "boxes" (previously known as atoms). One of these boxes can contain subtitles (multiple subtitle boxes allows for subs in different languages).

To take an external subtitles file and embed it into an mp4 container, one would need to first decode the container and then re-encode it with the subtitles box added in.

I've been trying to use the mp4-stream module to achieve this, but so far without any luck (seems too buggy).

Wrapping inside an m3u8 playlist

The most used approach is to wrap the movie in an m3u8 playlist. This allows you to add multiple bitrates of the same movie, separate audio-tracks and multiple subtitle tracks.

The downside of this approach is that it seems to require splitting up the file in multiple chunks of say 30 seconds each. This is problematic, because you need to parse the video to chop it up into these chunks. Most implementations uses ffmpeg to do this, but ffmpeg normally needs to be compiled locally on the users computer which makes this a hassle.

Also, as far as I know, m3u8 requires each segment to be mpeg2 - so mpeg4 videos needs to be converted.

In iOS 10 / macOS 10.12 / tvOS 10 an later, fMP4 and range queries is supported in m3u8 which (i think) means that we don't need to parse the video file to split it into 30 second segments and that we don't need to convert it to mpeg2 - but I have not validated any of this.

[ghost]

Pull request for aitte's changes, since he said he didn't have any more time and asked others to carry on his work. I've verified his changes and created a PR:

https://github.com/butterproject/butter-desktop/pull/437

And he's right, the current Butter streaming server doesn't yet support the Apple TV 4. But at least it finds it now and attempts to start streaming!

@watson Wow, you really followed through on your promise to research AirPlay subtitles. Thank you! <3

Out of those options, rewrapping in MP4 on-the-fly looks like the least CPU intensive method. Current M3U8's require converting to MPEG2-TS and splitting into 30 second chunks.

I haven't heard of modern M3U8 players supporting non-split MP4s and Range queries instead, but even if that's true it would still mean we need MP4 files.

So it looks like the best method is:

• Create a dummy MP4 container with one or more subtitle atoms at the start.
• Then re-wrap the video and audio data from the older container into MP4 on the fly.

I know that the "Airflow" (http://airflowapp.com) player supports subtitles and sends video without re-encoding, so I bet it uses the MP4 re-wrapping method. The author of Airflow can be found at https://www.reddit.com/user/airflow_matt/ and on the Air Video HD (http://www.inmethod.com/airvideohd/) forums, because he's the creator of AVHD too. Perhaps he would be open to telling you how he achieved subtitles over AirPlay?

@xaiki I'm not too familiar with Butter, but maybe it already contains ffmpeg? If so, perhaps we can force ffmpeg to add a subtitle stream to the output.

Also: I see what you mean about "generic.js" trying to find the closest IP. Something would need to be changed to make it prefer IPv4 for AirPlay. I can't promise that I'll look at it. I just wanted to submit the PR to see the AirPlay changes get added, but perhaps I'll do further work too. :)

[xaiki]

@SteveJobzniak we don't ship ffmpeg, will comment on your PR, thanks for picking this up !

[vankasteelj]

https://github.com/butterproject/butter-desktop/pull/466/files