warren-bank / HLS-Proxy

Node.js server to proxy HLS video streams
http://webcast-reloaded.surge.sh/proxy.html
GNU General Public License v2.0
238 stars 68 forks source link

Fails to modify m3u8 content when server doesn't response with a "content-encoding" header #26

Closed Shepless closed 1 year ago

Shepless commented 1 year ago

Firstly, thanks for such an excellent package and all the work you have put into this @warren-bank

After spending some time (and failing!) to get my my m3u8 file to proxy correctly, I spent some time debugging it and found that if a response doesn't include a content-encoding header the response just returns empty, This empty response then stops the modify_m3u8_content function from modifying the contents to prepend the proxied URL.

I was hoping to fix it myself, but alas my node stream skills aren't very good so I have spent a few hours failing.

The issue isn't your library really, just poorly implemented server responses. But the issue occurs here in your request package

Update Things I tried when fixing myself:

warren-bank commented 1 year ago

the line of code you point to is:

if ((data instanceof Object) && (data.headers instanceof Object) && (typeof data.headers['content-encoding'] === 'string')){
  // ...
  return decoded_data
}
return data

is the issue that the server is returning the header:

content-encoding: 

so it exists.. it's a string.. but it has an empty value?

if so, would simply changing that line to also test that the value is non-empty fix your issue?

if ((data instanceof Object) && (data.headers instanceof Object) && (typeof data.headers['content-encoding'] === 'string') && data.headers['content-encoding']){
  // ...
  return decoded_data
}
return data
Shepless commented 1 year ago

@warren-bank I shall give that a try. However, I forgot that whilst debugging I found that it wasn't hitting the modify m3u8 content function at all because the m3u8 I'm using has .ts extensions for each stream. For example:

#EXTINF:
aHR0cHM6Ly90dm5vdy5iZXN.....=.ts

This makes it skip due to the is_m3u8 check in the proxy.js. I forgot when debugging that I had forced it to modify the content when I stumbled on the content-encoding issue.

warren-bank commented 1 year ago

sorry.. I answered before actually reading what was inside of that if block..

now, I see that the value of the header is tested for supported encodings.. and if the value isn't supported (or is empty).. then the decoder doesn't do anything.. so the method is harmless

warren-bank commented 1 year ago

can we just start at the beginning.. you can change the URL domains and paths.. but just give an idea of what you've done.. so I can quickly confirm that it isn't something simple and obvious.

what does the URL for the manifest look like.. that you are giving to a client.. for it to access a stream via the proxy?

http://<ip of proxy>:<port of proxy>/<base64 of manifest URL>.m3u8

warren-bank commented 1 year ago

that manifest isn't HLS..

IPTV .m3u lists of channels look a lot like HLS .m3u8 lists of video chunks.. it's confusing.. but your manifest is for IPTV, and each .ts URL points to a stream.. whereas in HLS, each .ts URL points to a file that holds about 10 seconds of video.

I don't really know the technical specifics.. only that the distinction is a constant source of aggrevation.

Shepless commented 1 year ago

I thought that at first too, but under the hood it is an m3u8, the .ts is doing a 301 redirect to the m3u8 (annoyingly!) I added this console log in to debug it inside the request module:

const redirect = resolveRedirectLocation(error.location, original_url, redirects)
console.log('REDIRECT DETECTED FROM', original_url, 'TO', redirect);

Example Output:

REDIRECT DETECTED
FROM
https://tvnow.best/api/stream/xxx/xxxx/livetv.epg/5.star.max.eastern.us.m3u8
TO
https://195-181-172-33.servers.tips:5052/live11/Otr8WWXE98TJUfn97yfsIQ/632263/1679019690/1022584.m3u8

curl -i https://195-181-172-33.servers.tips:5052/live11/Otr8WWXE98TJUfn97yfsIQ/632263/1679019690/1022584.m3u8 HTTP/1.1 200 OK Server: nginx Date: Thu, 16 Mar 2023 22:24:36 GMT Content-Type: application/vnd.apple.mpegurl Content-Length: 252 Connection: keep-alive Last-Modified: Thu, 16 Mar 2023 22:24:25 GMT ETag: "64139719-fc" Expires: Thu, 16 Mar 2023 22:24:41 GMT Cache-Control: max-age=5 Strict-Transport-Security: max-age=31536000; includeSubdomains; Access-Control-Allow-Origin: * Access-Control-Allow-Methods: GET, POST, OPTIONS Access-Control-Allow-Headers: DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range Access-Control-Expose-Headers: Content-Length,Content-Range

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:13
#EXT-X-MEDIA-SEQUENCE:2640
#EXTINF:11.878533,
1022584_2640.ts
#EXTINF:11.177833,
1022584_2641.ts
#EXTINF:12.212200,
1022584_2642.ts
#EXTINF:12.679333,
1022584_2643.ts
#EXTINF:12.645967,
1022584_2644.ts
warren-bank commented 1 year ago

i'm lost.. you said URLs with a .ts extension are redirecting to URLs with a .m3u8 extension.. but your example is .m3u8 to .m3u8

where did this come from: https://tvnow.best/api/stream/xxx/xxxx/livetv.epg/5.star.max.eastern.us.m3u8

warren-bank commented 1 year ago

and.. presumably.. having cherry picked the URL for an HLS manifest for only one channel.. will it work with the proxy (if used directly)?

Shepless commented 1 year ago

and.. presumably.. having cherry picked the URL for an HLS manifest for only one channel.. will it work with the proxy (if used directly)?

If I cherry pick one of this streams to proxy directly, it still fails because it isn't rewriting the m3u file contents, its just using the raw output for the chunks (/12345.ts)

warren-bank commented 1 year ago

ok.. I see the problem

you're feeding an EPG into the proxy in place of a manifest. because the manifest doesn't have the tags that the parser uses to identify it as being a master manifest, it is seen as being a video manifest.. and all URLs in a video manifest are for video segments (ie: .ts files)

try this.. forget about the EPG.. cherry pick one channel (ex: 5 Star Max (East)).. and try to proxy the URL for its HLS manifest (ex: https://tvnow.best/api/stream/<user>/<password>/livetv.epg/5.star.max.eastern.us.m3u8)

what happens?

Shepless commented 1 year ago

Sorry, I had just tried that before you replied. See the above response. Apologies (and thank for all your help with this!)

Screenshot from 2023-03-16 22-51-04

If i play the non-proxy url this is what happens: Screenshot from 2023-03-16 22-52-44

Screenshot from 2023-03-16 22-52-52

Screenshot from 2023-03-16 22-54-02

So it looks like the 301 redirect is affecting it somehow.

warren-bank commented 1 year ago

I think you misunderstood.. because your example that failed is still using a .ts extension.. which it shouldn't be

what I'm saying is.. use that URL (for the particular channel).. as your starting point to proxy.. and it should work

Shepless commented 1 year ago

I did just that. (used the base64 encoded string for https://tvnow.best/api/stream/<user>/<password>/livetv.epg/5.star.max.eastern.us.m3u8)

That first screen shot is when it tried to request chunks from the proxy. Here it is with more context.

Screenshot from 2023-03-16 22-58-32

the .ts chunks are 404ing because the proxy isn't modifying the m3u8 content to prepend the base64 hash. My guess is because https://tvnow.best/api/stream/<user>/<password>/livetv.epg/5.star.max.eastern.us.m3u8 does a 301 redirect to https://195-181-172-33.servers.tips:5052/live3/KJM_52qoazU4umeRv8Fhag/632263/1679022183/24005968.m3u8?

warren-bank commented 1 year ago

humor me..

  1. go here: http://webcast-reloaded.surge.sh/proxy.html
  2. enter: https://tvnow.best/api/stream///livetv.epg/5.star.max.eastern.us.m3u8
  3. configure for: 127.0.0.1:9002
  4. choose the Clappr video player, and watch the stream in your browser
    • you can inspect both the Network tab in Chrome devtools, and the command-line logs from your proxy
    • but I'm guessing it'll just work
warren-bank commented 1 year ago

sorry, "Clappr" is under "Chromecast sender"

warren-bank commented 1 year ago

choosing "HLS-Proxy configuration" is the page that I sent you to directly.. choosing it from the dropdown just loops you back in a circle

Shepless commented 1 year ago

Aye sorry, got there in the end. I get CORS errors, guessing thats my proxy setup?

warren-bank commented 1 year ago

that should never happen with the proxy.. it always adds CORS headers

Shepless commented 1 year ago

Screenshot from 2023-03-16 23-10-26

Shepless commented 1 year ago

If this is any help: Screenshot from 2023-03-16 23-12-08

warren-bank commented 1 year ago

that's specific to your browser and its security policy.. and that changes with every release..

you need a browser that can load an HTTP (not secure) page.. run a script from an HTTPS (secure) URL.. and XHR content from HTTP at localhost.

warren-bank commented 1 year ago

if you want.. you can run the proxy with https.. and load the same proxy page from here with https

warren-bank commented 1 year ago

otherwise, you'll need to figure out what options to change.. or flags to pass to the browser when you start it (from the command line)

Shepless commented 1 year ago

ok got it working. but its getting lots of 500 errors Screenshot from 2023-03-16 23-17-45

Im guessing you got logs on that?

warren-bank commented 1 year ago

don't look at me..

view-source:http://webcast-reloaded.surge.sh/4-clappr/index.html

shows that Clappr is coming from:

  <script type="text/javascript" src="https://cdn.jsdelivr.net/npm/clappr@0.3.13/dist/clappr.min.js"></script>
  <script type="text/javascript" src="https://cdn.jsdelivr.net/npm/level-selector@0.2.0/dist/level-selector.min.js"></script>
  <script type="text/javascript" src="https://cdn.jsdelivr.net/npm/clappr-chromecast-plugin@0.1.1/dist/clappr-chromecast-plugin.min.js"></script>
  <script type="text/javascript" src="https://cdn.jsdelivr.net/npm/dash-shaka-playback@3.0.3/dist/dash-shaka-playback.min.js"></script>

..that's a pretty standard CDN

Shepless commented 1 year ago

But those 500s are coming from the local proxy?

Screenshot from 2023-03-16 23-23-06

warren-bank commented 1 year ago

oh yeah, sorry.. you're right.. the XHR request was made by Clappr (which did load).. and that's what's hitting browser security blocks

anway.. this is getting way off track

warren-bank commented 1 year ago

sorry: "AirPlay sender"

Shepless commented 1 year ago

That works fine in VLC.

warren-bank commented 1 year ago

yeah, no browser security to mess things up

Shepless commented 1 year ago

So the conclusion is - it works :rofl: ? Was the issue with me trying it a web based player? (Aside from the M3U beng a TVG format - which I can sort my end)

warren-bank commented 1 year ago

that was supposed to be the fast way to test what I was saying.. but boy, did that go off track.. anyway.. it proves that:

  1. if you cherry pick the URL for one channel from your EPG guide.. you can use it with HLS Proxy
  2. if you want to play a stream from HLS-Proxy in your browser.. you need to configure your browser to allow it

as I said earlier.. your screenshot that you said failed shows you trying to proxy a manifest with a .ts extension.. which made no sense. this little test proved that you can proxy an .m3u8 extension URL.. taken from within your EPG.

warren-bank commented 1 year ago

..but NOT the .m3u8 URL for the EPG itself!!

..because it isn't an HLS manifest

Shepless commented 1 year ago

Oh man, I'm so sorry! I really appreciate your effort and patience with me on this! I've learnt a lot. Thanks again. I'll close this behemoth off.

warren-bank commented 1 year ago

no worries.. glad you find the proxy useful :)