use node-fetch instead of Request()

[amamadman]

Hello! I need to pass additional headers to access a playlist, and I recon its not possible with you request module. Can you please make a version of proxy.js that uses fetch. You project is very helpful, it would be great if the request headers are preserved while proxying the requests.

Thanks in advance.

[warren-bank]

there are (at least) 3x ways to add (multiple) headers to requests:

1. --req-headers /path/to/headers.json
   • {"name1": "value1", "name2": "value2"}
2. --header "name1: value1" --header "name2: value2"
3. --hooks /path/to/hooks.js
   • module.exports = {add_request_headers: (url, is_m3u8) => is_m3u8 ? {"name1": "value1", "name2": "value2"} : null}

Where the first 2x apply to all requests. Whereas, the last can be applied dynamically, conditionally depending on the value of its inputs.

[warren-bank]

note: You can test whether you've correctly configured the proxy by requesting a URL through the proxy that displays the request headers that the server received.

for example:

1. start proxy
   • hlsd --port 8080 --header "name1: value1" --header "name2: value2"
2. display request headers received by server
   • direct URL
   • proxied URL

[amamadman]

Thanks for the reply, Is it possible to send the headers from the request itself, so if the proxied url was requested with X-Requested-With, than send them to while making a request to proxy?

[amamadman]

Or like be able to provide other header in base64 encoded path similar to how the referer header is set

[warren-bank]

to be clear, you're asking to copy request headers sent to the proxy → to the requests made by the proxy to the origin server?

for example:

curl -H "name3: value3" "http://localhost:8080/aHR0cHM6Ly9odHRwYmluLm9yZy9nZXQ=.json"

..and you would expect "name3" to be echoed back by the httpbin server?

---

that's not currently supported.. it wouldn't be difficult to add.. and it could be conditionally enabled (for example, by setting a boolean flag: --copy-req-headers)

personally, I don't see any use-case for this.. but I suppose there's no harm in making it an available option.

I'm a little busy at the moment, but I'll circle back around to this pretty soon.

PS: hard "no" on adding any additional headers to the base64 encoded value.. Referer is a special case.. since almost every video server uses it to determine access to streaming content.

[amamadman]

Um, till you implement it, how do I temporarily implement this, so it always sends the headers. I think the request object in proxy.js already has the request headers, so they just have to sent to the origin server. I was wondering how do you send headers in your node-request module?

[amamadman]

One use case for this would be proxying ip-locked content, for example scraping m3u8 from a video host, since every request is just piped through the proxy, you can use it to fetch the video page and then scrape the m3u8 from there so it plays, if the page is requested normally and then the hls proxy is attached to the stream it wont work because it's ip-locked. But sometimes the embed video pages don't work without the 'Sec-Fetch-Dest': 'iframe' header.

[warren-bank]

I didn't quite follow that.. but in any case, v3.5.3 was just pushed to npm.. and it adds the command-line option: --copy-req-headers, which behaves as described, but takes the lowest precendence.. which is to say that all other ways to specify headers will override these inbound header values. (ex: --useragent 007 will override the User-Agent header of the request received by the proxy)

[warren-bank]

v3.5.4: tiny fix.. to blacklist copying the Host header

[warren-bank]

quick test:

1. start proxy (with static headers)

    hlsd --port 8080 --copy-req-headers --useragent "007" --header "name1: value1" --header "name2: value2"

2. send request through proxy (with dynamic headers) to display all request headers received by server

    curl -H "name3: value3" -H "name4: value4" "http://localhost:8080/aHR0cHM6Ly9odHRwYmluLm9yZy9nZXQ=.json"

[amamadman]

Thank You so much. I really appreciate your quick replies and helpfulness.