search

bentasker / HLS-Stream-Creator

Simple Bash Script to take a media file, segment it and create an M3U8 playlist for serving using HLS
BSD 3-Clause "New" or "Revised" License
273 stars 101 forks source link

The files created with this script will not play on Android devices #16

Closed danieladr closed 7 years ago

danieladr commented 7 years ago

I've just realized that none of my files will play on Android devices. I've spent over 3 months rendering over 1,000 videos and now it has come to my attention because none of my students are able to view the videos from their Android devices, only from iPhones and Laptops/Desktops.

bentasker commented 7 years ago

Have you got an example stream? What version of android?

My streams all play back on Android. That said, Native HLS support on Android has something of a chequered past, so you probably will see a bit of variance. Players like MX Player (and even Samsung's 'Videos') will happily handle it in either case, but whether you'll get in-browser playback will vary a bit more - you should do with Chrome browser though (depending on the underlying version of Android)

danieladr commented 7 years ago

Thanks for the reply :)

Yes, I do. It would be the video on the header of this link.

I've encoded all my videos using these settings, would you have a different recommendation?

./HLS-Stream-Creator.sh -i /Volumes/VERBO/EAD/Starling/EAD/VOCAL/LED/MP4/semana_21.* -s 10 -o /Volumes/VERBO/EAD/Starling/EAD/VOCAL/LED/HLS/semana_21 -b 272,872,1372,1872 -f

They are all playing it back from different Android versions, I have received feedback from users using from 4.1 to 7.0, most of them on the latest Android version and all on Google Chrome. The same issue has been replicated across all device brands and browsers. Basically, when you hit 'play' the time would be 00:00 and the player's screen goes black.

On iPhones, iPad, Desktops, and Laptops it would play back normally on different browsers. I tested mostly on Chrome, Safari, and Firefox.

android_example

img_5256

bentasker commented 7 years ago

Hi,

I've a theory on this, but can't look at it at the moment.

If you want to test in the meantime though, it looks like you've some access controls set up on the S3 bucket hosting the videos (I can't play back with ffplay without setting the referrer header correctly, for example). I've a feeling that android's playback doesn't actually set referrer, so it may be worth turning that off to test.

I'll try and find some time to MiTM the connection to see whether I can find out any more

Ben

danieladr commented 7 years ago

Hi, yes I do. They go like this:

{
    "Version": "2008-10-17",
    "Id": "S3BubbleSecurity",
    "Statement": [
        {
            "Sid": "S3BubbleAllow",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::starlingead/*",
            "Condition": {
                "StringLike": {
                    "aws:Referer": [
                        "https://s3bubble.com/*",
                        "https://media.s3bubble.com/*",
                        "https://ead.starlingacademy.com.br/*"
                    ]
                }
            }
        }
    ]
}

I tried removing the 'Referer' option but the problem remains :(

Do you have any suggestion on how to change the policy?

bentasker commented 7 years ago

Hmmm, ok.

Bear with me then and I'll MiTM the outgoing request so that I can see the headers (and response code etc)

bentasker commented 7 years ago

OK, so I've just MiTM'd it, and it looks like it is the referrer header (or at least, the S4 I tested from doesn't include one).

With a simple MiTM I get the following logs

    10.15.0.10  -   -   [16/Aug/2017:12:48:14 +0100]    "GET /INSTITUCIONAL/EAD/TOUR/HLS/ead.mp4_master.m3u8 HTTP/1.1"  206 220 "https://ead.starlingacademy.com.br/"   "Mozilla/5.0 (Linux; Android 4.4.2; GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"    "-" "starlingead.s3.amazonaws.com"  CACHE_- 0.399   868162.vps-10.com   starlingead.s3.amazonaws.com
    127.0.0.2   -   -   [16/Aug/2017:12:48:14 +0100]    "GET /INSTITUCIONAL/EAD/TOUR/HLS/ead.mp4_master.m3u8 HTTP/1.0"  206 220 "https://ead.starlingacademy.com.br/"   "Mozilla/5.0 (Linux; Android 4.4.2; GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"    "-" "starlingead.s3.amazonaws.com"  CACHE_- 0.399   868162.vps-10.com   starlingead.s3.amazonaws.com
    10.15.0.10  -   -   [16/Aug/2017:12:48:14 +0100]    "GET /INSTITUCIONAL/EAD/TOUR/HLS/ead.mp4_master.m3u8 HTTP/1.1"  403 254 "-" "Mozilla/5.0 (Linux; Android 4.4.2; GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"    "-" "starlingead.s3.amazonaws.com"  CACHE_- 0.340   868162.vps-10.com   starlingead.s3.amazonaws.com
    127.0.0.2   -   -   [16/Aug/2017:12:48:14 +0100]    "GET /INSTITUCIONAL/EAD/TOUR/HLS/ead.mp4_master.m3u8 HTTP/1.0"  403 243 "-" "Mozilla/5.0 (Linux; Android 4.4.2; GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"    "-" "starlingead.s3.amazonaws.com"  CACHE_- 0.340   868162.vps-10.com   starlingead.s3.amazonaws.com
    10.15.0.10  -   -   [16/Aug/2017:12:48:15 +0100]    "GET /INSTITUCIONAL/EAD/TOUR/HLS/ead.mp4_master.m3u8 HTTP/1.1"  403 254 "-" "Mozilla/5.0 (Linux; Android 4.4.2; GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"    "-" "starlingead.s3.amazonaws.com"  CACHE_- 0.503   868162.vps-10.com   starlingead.s3.amazonaws.com
    127.0.0.2   -   -   [16/Aug/2017:12:48:15 +0100]    "GET /INSTITUCIONAL/EAD/TOUR/HLS/ead.mp4_master.m3u8 HTTP/1.0"  403 243 "-" "Mozilla/5.0 (Linux; Android 4.4.2; GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"    "-" "starlingead.s3.amazonaws.com"  CACHE_- 0.504   868162.vps-10.com   starlingead.s3.amazonaws.com
    10.15.0.10  -   -   [16/Aug/2017:12:48:20 +0100]    "GET /INSTITUCIONAL/EAD/TOUR/HLS/ead.mp4_master.m3u8 HTTP/1.1"  403 254 "-" "Mozilla/5.0 (Linux; Android 4.4.2; GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"    "-" "starlingead.s3.amazonaws.com"  CACHE_- 0.342   868162.vps-10.com   starlingead.s3.amazonaws.com
    127.0.0.2   -   -   [16/Aug/2017:12:48:20 +0100]    "GET /INSTITUCIONAL/EAD/TOUR/HLS/ead.mp4_master.m3u8 HTTP/1.0"  403 243 "-" "Mozilla/5.0 (Linux; Android 4.4.2; GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"    "-" "starlingead.s3.amazonaws.com"  CACHE_- 0.343   868162.vps-10.com   starlingead.s3.amazonaws.com

Those first 2 requests do include the referer - I'm not sure why the player drops it...

See pcap http.pcap for the plaintext packet capture, you can see referrer isn't there.

Just to prove definitively whether it's definitely the issue, I configured my MiTM to inject a valid Referer header and playback started

    10.15.0.10  -   -   [16/Aug/2017:12:53:06 +0100]    "GET /INSTITUCIONAL/EAD/TOUR/HLS/ead.mp4_master.jpg?response-content-type=image%2Fjpeg&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJU4JKHMR652E4TNQ%2F20170815%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20170815T195917Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3540&X-Amz-Signature=ecc2a92048c6f92a69788f75515172aa362c0d6972004e9f373886960d3cb9b8 HTTP/1.1"   403 380 "https://ead.starlingacademy.com.br/"   "Mozilla/5.0 (Linux; Android 4.4.2; GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"    "-" "starlingead.s3.amazonaws.com"  CACHE_- 0.329   868162.vps-10.com   starlingead.s3.amazonaws.com
    127.0.0.2   -   -   [16/Aug/2017:12:53:06 +0100]    "GET /INSTITUCIONAL/EAD/TOUR/HLS/ead.mp4_master.jpg?response-content-type=image%2Fjpeg&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJU4JKHMR652E4TNQ%2F20170815%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20170815T195917Z&X-Amz-SignedHeaders=host&X-Amz-Expires=3540&X-Amz-Signature=ecc2a92048c6f92a69788f75515172aa362c0d6972004e9f373886960d3cb9b8 HTTP/1.0"   403 368 "https://ead.starlingacademy.com.br/foo"    "Mozilla/5.0 (Linux; Android 4.4.2; GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"    "-" "starlingead.s3.amazonaws.com"  CACHE_- 0.329   868162.vps-10.com   starlingead.s3.amazonaws.com
    10.15.0.10  -   -   [16/Aug/2017:12:53:12 +0100]    "GET /INSTITUCIONAL/EAD/TOUR/HLS/ead.mp4_master.m3u8 HTTP/1.1"  200 220 "-" "Mozilla/5.0 (Linux; Android 4.4.2; GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"    "-" "starlingead.s3.amazonaws.com"  CACHE_- 0.352   868162.vps-10.com   starlingead.s3.amazonaws.com
    10.15.0.10  -   -   [16/Aug/2017:12:53:12 +0100]    "GET /INSTITUCIONAL/EAD/TOUR/HLS/ead.mp4_master.m3u8 HTTP/1.1"  200 220 "-" "Mozilla/5.0 (Linux; Android 4.4.2; GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"    "-" "starlingead.s3.amazonaws.com"  CACHE_- 0.364   868162.vps-10.com   starlingead.s3.amazonaws.com
    127.0.0.2   -   -   [16/Aug/2017:12:53:12 +0100]    "GET /INSTITUCIONAL/EAD/TOUR/HLS/ead.mp4_master.m3u8 HTTP/1.0"  200 220 "https://ead.starlingacademy.com.br/foo"    "Mozilla/5.0 (Linux; Android 4.4.2; GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"    "-" "starlingead.s3.amazonaws.com"  CACHE_- 0.353   868162.vps-10.com   starlingead.s3.amazonaws.com
    127.0.0.2   -   -   [16/Aug/2017:12:53:12 +0100]    "GET /INSTITUCIONAL/EAD/TOUR/HLS/ead.mp4_master.m3u8 HTTP/1.0"  200 220 "https://ead.starlingacademy.com.br/foo"    "Mozilla/5.0 (Linux; Android 4.4.2; GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"    "-" "starlingead.s3.amazonaws.com"  CACHE_- 0.364   868162.vps-10.com   starlingead.s3.amazonaws.com
    10.15.0.10  -   -   [16/Aug/2017:12:53:13 +0100]    "GET /INSTITUCIONAL/EAD/TOUR/HLS/ead.mp4_272.m3u8 HTTP/1.1" 200 1411    "-" "Mozilla/5.0 (Linux; Android 4.4.2; GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"    "-" "starlingead.s3.amazonaws.com"  CACHE_- 0.368   868162.vps-10.com   starlingead.s3.amazonaws.com
    127.0.0.2   -   -   [16/Aug/2017:12:53:13 +0100]    "GET /INSTITUCIONAL/EAD/TOUR/HLS/ead.mp4_272.m3u8 HTTP/1.0" 200 1411    "https://ead.starlingacademy.com.br/foo"    "Mozilla/5.0 (Linux; Android 4.4.2; GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"    "-" "starlingead.s3.amazonaws.com"  CACHE_- 0.367   868162.vps-10.com   starlingead.s3.amazonaws.com
    10.15.0.10  -   -   [16/Aug/2017:12:53:14 +0100]    "GET /INSTITUCIONAL/EAD/TOUR/HLS/ead.mp4_272_00000.ts HTTP/1.1" 200 621528  "-" "Mozilla/5.0 (Linux; Android 4.4.2; GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36"    "-" "starlingead.s3.amazonaws.com"  CACHE_- 1.124   868162.vps-10.com   starlingead.s3.amazonaws.com
    127.0.0.2   -   -   [16/Aug/2017:12:53:14 +0100]    "GET /INSTITUCIONAL/EAD/TOUR/HLS/ead.mp4_272_00000.ts HTTP/1.0" 200 621528  "https://ead.starlingacademy.com.br/foo"    "Mozilla/5.0 (Linux; Android 4.4.2; GT-I9505 Build/KOT49H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile

So, yeah, you need to take a look at that ACL. I think you'll probably need to remove the entire condition block for it to start working (as otherwise there'll be nothing to match and everything will fail by default)

bentasker commented 7 years ago

Ach, Github doesn't let us attach pcaps.

Pcaps can be found here - https://filedrop.bentasker.co.uk/5/hnqxpYoZzHYN2fr3Bq2dMGOivrQsLKkpXK7:ivyvMgFS1ih5y8LnmSwtAFNpSTc_tf@6eETrnOV-1pyHMTx0_BOKi0h-0-BNx1SmOfnUxGFBELz6X6OhbR1t1UX0U=C@=mfEDdBlYreR1Rk9Y-YxAd/hls-iss-16/

danieladr commented 7 years ago

Thanks! I have just removed the entire block now, I will do some testing. I hope this resolves the issue.

I really appreciate your time and help. I will let you know if it works.

danieladr commented 7 years ago

Still the same issue... I don't know what else to do :(

I configured my bucket like this now:

{
    "Version": "2008-10-17",
    "Id": "S3BubbleSecurity",
    "Statement": [
        {
            "Sid": "S3BubbleAllow",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::starlingead/*"
        }
    ]
}
screen shot 2017-08-16 at 09 09 24 screen shot 2017-08-16 at 09 09 15
bentasker commented 7 years ago

Maybe a caching thing with AWS?

Seems to work on my S4 now without injecting a referer header

danieladr commented 7 years ago

Hey... Thanks! Looks like it is working now :)

bentasker commented 7 years ago

No worries, closing this off now