Modify served content

[istvanmate]

Seeking help once again.

When proxying .m3u8 files, there are sometimes embedded URL-s that would have the client contact the server directly via the embedded URL. I would like to change that URL and serve the modified content. What I managed to do, is to check if the URL points to an .m3u8 file

ctx.body = req
if (dest.endsWith(".m3u8")) {
    ctx.body = req // here I would need to modify the content, but .replace() does not work
}

Could anyone point me to the right direction?

[rejetto]

since your files are not huge, your best solution is probably to load the file, to change it, and send the content. Other methods are possible, tho.

Your files are text files, and your case is generic enough to be an interesting example, so i added this in the documentation: https://github.com/rejetto/hfs/wiki/Middlewares#manipulate-text-file-content in my example i'm not considering reverse-proxy plugin, but i expect it to work with it.

[istvanmate]

Yes, it does work. Starting the stream gets considerably slower, but it is expected.

I do get a lot of requests with PROPFIND method, to which hfs replies with 405. Not sure, if this is a hfs or reverse-proxy issue, also happy to open another issue for it, if needed.

[istvanmate]

Will have to correct my previous post. Interestingly this solution only works with some clients, like mpc-hc on Windows, ottplayer.tv on iOS, but will mess up for ottplayer.tv on Windows and Samsung TV. Without the fix, these work too (of course the m3u8 files are unchanged).

[rejetto]

Yes, it does work. Starting the stream gets considerably slower, but it is expected.

i don't see why it should. isn't your m3u8 file small?

PROPFIND is webdav, and hfs doesn't support it yet, so 405 is correct. It means some client is hoping to find a webdav server on the other part.

The solution i suggested enables you to serve the files modified in real-time, in a "virtual" way. The client has no way to tell what the original file was, or if any change has been made. If some clients don't work it's more likely to be related to the type of change you decided to make on the files.

[rejetto]

You can consider making changes based on the client. You can see how it identifies in the user-agent string.

[istvanmate]

Can I enable somewhere to see the served contents as well? I see size discrepancy between served m3u8 files to the different clients...

[istvanmate]

You can consider making changes based on the client. You can see how it identifies in the user-agent string.

I'm not sure, I can do that. The reason (embedded link) is the encryption key's location, that I cannot configure on the client to look for somewhere else: Some (not all) m3u8 files contain a line like this

#EXT-X-KEY:METHOD=AES-128,URI="link to the encryption key"

I will change it to

#EXT-X-KEY:METHOD=AES-128,URI="/key"

and make a proxy entry for it.

[rejetto]

Can I enable somewhere to see the served contents as well? I see size discrepancy between served m3u8 files to the different clients...

open the URL with a browser. i expect you'll see the content

[istvanmate]

Can I enable somewhere to see the served contents as well? I see size discrepancy between served m3u8 files to the different clients...

open the URL with a browser. i expect you'll see the content

Problem with this is, that in the browser file is ok. I would like to see the file served to the problematic client. I may try Wireshark...

[rejetto]

You can consider making changes based on the client. You can see how it identifies in the user-agent string.

I'm not sure, I can do that. The reason (embedded link) is the encryption key's location, that I cannot configure on the client to look for somewhere else: Some (not all) m3u8 files contain a line like this

i don't understand what you are talking about. you can see the user-agent by looking at the logs. You'll know then if there's a difference that you can use (it's likely).

[rejetto]

Problem with this is, that in the browser file is ok. I would like to see the file served to the problematic client. I may try Wireshark...

how could the file be different? hfs is surely making no difference, when serving the file (only for the UI). so, you are supposing the other server (you are proxing to) is making a difference...

[rejetto]

no need for wireshark, just add a console.log when you assign ctx.body. then you'll get the content in the hfs' console

[istvanmate]

Interesting. User agent: NSPlayer/12.00.22621.4317 WMFSDK/12.00.22621.4317 Accessed index.m3u8 8 times: bodies (also there is a first error message?): correct body:

#EXTM3U #EXT-X-VERSION:5 #EXT-X-MEDIA:TYPE=AUDIO,URI="104-M4_HUGEO_MAIN-mp4a_140800_hun=1.m3u8",GROUP-ID="audio-AACL-141",LANGUAGE="hu",NAME="magyar",DEFAULT=YES,AUTOSELECT=YES,CHANNELS="2" #EXT-X-MEDIA:TYPE=AUDIO,URI="104-M4_HUGEO_MAIN-mp4a_140800_mul=3.m3u8",GROUP-ID="audio-AACL-141",LANGUAGE="mul",NAME="eredeti",AUTOSELECT=YES,CHANNELS="2" #EXT-X-STREAM-INF:BANDWIDTH=622827,AVERAGE-BANDWIDTH=566207,CODECS="avc1.42e015,mp4a.40.2",RESOLUTION=426x240,FRAME-RATE=25.000,AUDIO="audio-AACL-141" 104-M4_HUGEO_MAIN-avc1_400000=2.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=972589,AVERAGE-BANDWIDTH=884174,CODECS="avc1.42e01e,mp4a.40.2",RESOLUTION=640x360,FRAME-RATE=25.000,AUDIO="audio-AACL-141" 104-M4_HUGEO_MAIN-avc1_699968=4.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=1438989,AVERAGE-BANDWIDTH=1308174,CODECS="avc1.4d401e,mp4a.40.2",RESOLUTION=854x480,FRAME-RATE=25.000,AUDIO="audio-AACL-141" 104-M4_HUGEO_MAIN-avc1_1099968=5.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=1905389,AVERAGE-BANDWIDTH=1732174,CODECS="avc1.4d401f,mp4a.40.2",RESOLUTION=960x540,FRAME-RATE=25.000,AUDIO="audio-AACL-141" 104-M4_HUGEO_MAIN-avc1_1499968=6.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=2255228,AVERAGE-BANDWIDTH=2050207,CODECS="avc1.64001f,mp4a.40.2",RESOLUTION=1280x720,FRAME-RATE=25.000,AUDIO="audio-AACL-141" 104-M4_HUGEO_MAIN-avc1_1800000=7.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=4470590,AVERAGE-BANDWIDTH=4064173,CODECS="avc1.640028,mp4a.40.2",RESOLUTION=1920x1080,FRAME-RATE=25.000,AUDIO="audio-AACL-141" 104-M4_HUGEO_MAIN-avc1_3699968=8.m3u8

the others:

���[k�0��~�U6G�,���Jb����O�,�ZX�Q��}�)��vHG��;����ѭ�o򠽹uG��{o�:-��8��m��U�pg�i���m�j���m��e�����߱���ǯ���`T�n�����D�~�=}�3�!#43Ŧ5���Hharw��?�� M�ڴY�lMM۔��l|����(lVk��5���E��t��~<���ꦲ&��b�zg��C�4[-9W<�ƥ���>Rri\&6v�??2�L�C�>�� �l]fm��I#�G�����U����ͭ�,����~ҡ�@�@N�.ڣ�-ؕB�}8c�S ���(���x��0P�"����;�W'T������+`!{|���L���I�A�0<��e>�B��_�s0�{����W0�� �w֡���~�D�&k6���~�o�����e������� ���:�H_j8�

encoding issue?

Also, it did not get to the m3u8 containing the encryption key link...

[rejetto]

it's likely to be gzip compressed, a feature of http. i suggest you to disable gzip compression on the other server.

[rejetto]

if you can't disable gzip compression on the server, there's another way: you can remove the header that the client is sending, saying that it supports gzip. But to do that, you have to manipulate the headers of the request. I just published an update of this plugin that allows you to do that. https://github.com/rejetto/reverse-proxy?tab=readme-ov-file#messing-with-forwarded-requests

[istvanmate]

it's likely to be gzip compressed, a feature of http. i suggest you to disable gzip compression on the other server.

This is only happening with the rewrite fix in place. If I remove it, then all is OK, so I suspect that something along the replace procedure does this.

[rejetto]

that's weird, but also the "slower" thing was. Maybe you are doing something wrong

[istvanmate]

that's weird, but also the "slower" thing was. Maybe you are doing something wrong

I hope it too, that way I can fix it 😄

[istvanmate]

if you can't disable gzip compression on the server, there's another way: you can remove the header that the client is sending, saying that it supports gzip. But to do that, you have to manipulate the headers of the request. I just published an update of this plugin that allows you to do that. https://github.com/rejetto/reverse-proxy?tab=readme-ov-file#messing-with-forwarded-requests

I am trying to get the forward.headers and forward.body properties to show content in the console.log. Currently they are [object Object]

[rejetto]

if for some reason you are not getting a decent visualization of the objects, you can force it by console.log(JSON.stringify(forward.headers))

body is probably a stream, and won't work