ghost
commented
7 years ago (This is copied from & extended from comments in the pull request discussion, to put it all in one place):
Here are the four AppleTV4 streaming problems in Butter, in order of severity from worst to least:
1. Butter defaults to using the IPv6 address as the streaming hostname in the URL it sends to the AirPlay.play() player function:
STATUS: FIXED BY https://github.com/butterproject/butter-desktop/pull/537
"[%cINFO%c] Available IPs: [\"fe80::6233:4bff:fe11:a4de\",\"10.0.1.167\"]"
Butter picks the "fe80" name. An AppleTV refuses to connect (whatsoever) if you are telling it the IPv6 stream URL. So for now, I had to manually run "networksetup -setv6off Wi-Fi" to turn off IPv6 to get Butter to advertise an IPv4 streaming URL instead.
There are 3 ways to fix that:
- Option 1: (outside airplay.js): if deviceType == AirPlay, generate an ipv4 streaming URL to give to the device's play() function. This is hardcoded and ugly and requires knowledge of AirPlay, but it's a quick way to fix it.
- Option 2 (inside airplay.js): Rewrite the URL before we send it to the AppleTV. Basically, get the local machine's ipv4 IP, and rewrite the streaming url to replace the host part with an IPv4 address. This requires more work and is more fragile against future changes; we'd need a URL rewriting library for Javascript, to extract the protocol (http), port, and the path, and then we need a "local ipv4 address" library (like "require('network-address')" or "var ip = require('internal-ip').v4()") to insert that ipv4 as the host. And we must be totally sure that Butter (webtorrent?) has bound to the IPv4 IP and is listening on the port (and that it didn't just bind the streaming port to IPv6).
- Option 3 (outside airplay.js): Best option: Make it possible for device libraries to specify what type of URL their play() method wants, and make Butter Desktop give the library the proper type. Or both types! This was the fix chosen!
2. AppleTV4 connects to Butter and then disconnects when it sees unexpected HTTP headers:
STATUS: THE REASON HAS BEEN FOUND. WE NEED TO SUPPORT APPLE'S HTTP LIVE STREAMING. THE DEVICES SUPPORT ONLY MOV, MP4, M4V AND HLS (M3U8). ANY OTHER CONTAINERS OR UNSUPPORTED CODEC CAUSES THE SCREEN TO FLASH BLACK BRIEFLY AS IT CONNECTS AND THEN ABORTS THE ATTEMPT TO PLAY THE UNSUPPORTED MEDIA.
The AppleTV4 seems more restrictive than older generations of the AppleTV. It attempts to connect to Butter's HTTP server for streaming, and then gives up and disconnects again.
For testing, I was able to make my AppleTV4 connect and stream via a small HTTP server which has the following headers (so whatever header stuff is happening below, is stuff Butter may need to be doing too):
var server = http.createServer(function (req, res) {
var stat = fs.statSync(filename)
var size = stat.size
var range = req.headers.range
range = range && rangeParser(size, range)[0]
res.writeHead(206, {
'Access-Control-Allow-Origin': '*',
'Accept-Ranges': 'bytes',
'Content-Length': (range.end - range.start + 1),
'Content-Type': contentType,
'Content-Range': util.format('bytes %d-%d/%d', range.start, range.end, size)
})
fs.createReadStream(filename, range).pipe(res)
})AppleTV4 connects fine to that. It only fails to connect to Butter's own HTTP server (which is based on webtorrent?). It may be that Butter's server doesn't have 'Access-Control-Allow-Origin': '*', or doesn't support byte-range commands, or content-length, or perhaps it needs to provide a ".mp4" filename in the stream URL in case the AppleTV4 doesn't use MIME to check its type, or perhaps it needs to send the proper mimetype, or any amount of other possible issues. Apparently the bug is in webtorrent, as per the discussion in this comment and the ones after it: https://github.com/butterproject/butter-desktop/pull/437#issuecomment-265910439.
I haven't got time for testing. Someone with AppleTV4 hardware and a Mac can easily do a tcpdump, and then developers can take a look to see why/when the AppleTV4 is disconnecting. Instructions for tcpdump are here, it's built into OSX: https://support.apple.com/en-us/HT202013
3. No scrubbing/seeking/playback position tracking:
STATUS: NOT REALLY WORTH WORKING ON IT UNTIL AIRPLAY IN BUTTER IS FURTHER ENHANCED WITH APPLE HLS SUPPORT. BUT CONTRIBUTIONS ARE ALWAYS WELCOME IF SOMEONE WANTS TO WORK ON IT NOW.
The groundwork for it has been done by the "_scrub()" function I added to the new airplay.js, but there was an unclear mess about what functions Butter actually requires for playback control, so I left that up to some other contributor.
What needs to be implemented: All scrubbing functions, and also the updating of playback position in Butter's GUI whenever the AppleTV returns position events to tell you where the user is in the video.
All discussions about the necessary scrubbing functions are in this pull request: https://github.com/butterproject/butter-desktop/pull/437 (but no conclusive answer was reached there about what functions are really needed).
Scrubbing will be easy to do via _scrub(), and for relative scrubbing (like -30 seconds) you would have to either keep track of the AirPlay device's (AppleTV's) playback position when the device sends you events, or query the device for that info via sending scrub() without arguments (the underlying function, not my wrapper).
Implementation of the automatic playback position event tracking can be done by analyzing @watson's "airplay-protocol" events, which has events for playback position/state updates. That requires a lot more time and work than I have available so I left that for someone else.
4. No subtitle support:
STATUS: WILL BE SUPPORTED WHENEVER WE HAVE APPLE HLS SUPPORT.
Unfortunately the AirPlay protocol has no "subtitle stream" support, so it's very hard to add support for subtitles over AirPlay, especially since Butter is streaming the data live from torrents and can't do container re-wrapping before streaming, since we don't have all data.
The only ways to support subtitles on AirPlay are by putting chunked mpeg2ts files in a m3u8 playlist (aka Apple HLS - HTTP Live Streaming), or by re-wrapping the media in a mp4 container on-the-fly (the desktop app "Airflow" does this). See this comment by @watson for both techniques: https://github.com/butterproject/butter-desktop/issues/434#issuecomment-248829694
The easiest way seems to be via Apple HLS. But m3u8 requires conversion of the video data into 30-second "mpeg2ts" transport stream containers. Those containers CAN hold h264 video, but the actual problem is MAKING those containers LIVE. You can read Apple's docs here: https://developer.apple.com/streaming/ (especially this page https://developer.apple.com/library/content/documentation/NetworkingInternet/Conceptual/StreamingMediaGuide/Introduction/Introduction.html).
The bare minimum seems to be to use Apple's Live stream segmenter tools: https://developer.apple.com/library/content/documentation/NetworkingInternet/Conceptual/StreamingMediaGuide/UsingHTTPLiveStreaming/UsingHTTPLiveStreaming.html#//apple_ref/doc/uid/TP40008332-CH102-SW1
Those tools CAN accept "live stream" files as-they-become-available, but it still seems like a lot of work to hack this into Butter. It might be the EASIEST way, though. Because with m3u8, you can just assign the subtitle as a separate "track" in the .m3u8 playlist file itsef.
Another way would be to somehow write a tool which rewrites the video container live and converts it to an mp4 with an internal subtitle track (that's what apps like Airflow do). But that would require someone to engineer a universal container-decoder (ie supporting .mp4, .avi, .mkv, etc), and having a subtitle-embedder, and then a container-creator (to final .mp4). And to do that LIVE. It would be very difficult, to say the least! For that reason, I think the best chance we have is to feed the data to Apple's "live stream segmenter tools" (links above) and letting IT take care of chunking the data into mpeg2ts... But again that's yet another huge project. So unless we get massive amounts of resources and developers with AppleTVs to help us out, I do not think we will see subtitle support in the AirPlay streamer.
Edit: One of the easiest ways may be to bundle the ffmpeg binary, and to do this:
- Make ffmpeg connect to the webtorrent live stream server.
- Make ffmpeg generate the .m3u8 and mpeg2ts chunks and make IT do the work of embedding the subtitle in that stream.
- Tell the AppleTV to connect to ffmpeg.
I really don't know if that's feasible. For instance, how would we know when the AppleTV wants to scrub in the media, when it's being proxied and converted by the intermediary ffmpeg step?. But it's worth researching. Maybe ffmpeg already supports scrubbing and passing those commands on to the source (Butter) that it's reading the data from.
Well, that's it from me. Now I've braindumped all the info I know about all 4 remaining AirPlay issues. ;-) I need to go. Very busy with real life.
xaiki
team-pct
see @SteveJobzniak's comments on #437 we'd need some tcpdumps to see what's happening as nobody owns that hardware.