[bug]: Unexpected request header modification when using Proxy (Proxyscotch)

rizdaprasetya commented 2 years ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current behavior

On https://hoppscotch.io/ , using this feature Use the proxy middleware to send requests, when I create the following request:

CURL equivalent:

curl --request POST \
  --url 'https://hookb.in/YVkaPWyrBbCQjy0QmeZp?param=test' \
  --header 'content-type: application/json' \
  --header 'origin: https://mycustomorigin.com' \
  --header 'referer: https://mycustomreferrer.com/' \
  --header 'user-agent: mycustom-UA' \
  --header 'x-source: mycustom-source' \
  --data '{}'

Then I inspect the request on the destination, and get the following result instead:

## HTTP HEADERS 
accept-encoding: gzip
content-length: 2
content-type: application/json
host: hookb.in
origin: https://mycustomorigin.com
referer: https://mycustomreferrer.com/
user-agent: Proxyscotch/1.1
x-source: mycustom-source

## QUERY STRING
param: test

## BODY
{}

unexpectedly some of the headers are modified:

from user-agent: mycustom-UA to user-agent: Proxyscotch/1.1. Looks like the Proxyscotch is overwriting it.

Why is it concerning?

In some valid use case, you really want to send a custom HTTP headers of user-agent. e.g.
- in case like to debug API issue, or
- to reproduce exact same request created by other client.
- You want to emulate browser request, with device specific user-agent).

So having this is important.

Maybe instead of Proxy scotch overwriting the actual user-agent header, it should add its own headers e.g. x-requested-via-proxy-agent: Proxyscotch/1.1

Steps to reproduce

Prepare endpoint that can inspect HTTP request. I used https://hookbin.com/. e.g. I will create endpoint & get URL https://hookb.in/YVkaPWyrBbCQjy0QmeZp
On https://hoppscotch.io/ , using this feature Use the proxy middleware to send requests
import CURL provided above, modify the URL with URL you get from step 1.
Send request
Inspect the HTTP request result on the destination (in my case I refreshed the Hookbin page I created on step 1.)
You will get same result as mine above.

Environment

Production

Version

Cloud

AndrewBastin commented 2 years ago

Moving this issue to the Proxyscotch repo.

SamJakob commented 2 years ago

Ah yes! We've not had this yet and that's a good point, we overlooked the use of user-agent. We added the header to help prevent abuse of the proxy and as a curtesy to other services (also CloudFlare blocks requests in some cases if you don't specify a User Agent at all).

I do like your solution though, so will go ahead and implement that.

rizdaprasetya commented 2 years ago

Cool, thanks for confirming! 👍

We added the header to help prevent abuse of the proxy and as a curtesy to other services (also CloudFlare blocks requests in some cases if you don't specify a User Agent at all).

Ah I see, that make sense too. I think need to find the middle ground to satisfy both use-case then.

Maybe

if user-agent specified by user, then do not overwrite, instead add x-requested-.... headers to identify proxy's user-agent.
if user-agent not specified, valid to add default user-agent (for cases like Cloudflare & proxy abuse)

Also x-requested-via-proxy-agent header is totally made up by me 😅 , I have not research far enough, maybe there's a standardized HTTP header for proxy case like this to properly use instead.

Thanks, appreciate the effort!

rizdaprasetya commented 2 years ago

Did some quick search, I think the correct HTTP header to use is Via. According to these resource:

Also if decided to follow that kind of standard, there's a few other to add like x-forwarded-for

SamJakob commented 2 years ago

Okay yeah I'll evaluate adding some or all of those headers then.

Just to confirm, are you in agreement with still using our Proxyscotch/1.0 User-Agent if no user agent header was specified (and Via always be set regardless)?

(This is in order to help prevent CloudFlare from blocking requests automatically which it often does when there's no user agent) – or do we leave it up to the user to figure it out in case there is a use case for not having the header set?

rizdaprasetya commented 2 years ago

Just to confirm, are you in agreement with still using our Proxyscotch/1.0 User-Agent if no user agent header was specified (and Via always be set regardless)?

✅ Yep, I personally agree. It maybe also helpful to compare the behavior with other tools like: Postman & CURL. They set their own user-agent as default (when user does not specify). Then they will not overwrite user-agent when user specify it. I think same approach can be used here.

Btw scope wise, IMO ideally:

This behavior "set their own user-agent as default (when user does not specify)" should be implemented on the REST client side (hoopscotch.io) instead of the Proxy side. (Just like Postman, this behavior come from Postman, not the proxy).
- Although it is ok if you want to double implement it on Proxy side as well.
Proxy should only be responsible in adding via (and other proxy related) headers.

Just adding insight, I agree whatever approach you choose, as you guys are the main owners of the project & it's specs. ✌️

SamJakob commented 2 years ago

Okay yeah that all sounds good. I might look at adding something into the protocol then to allow Hoppscotch to set it from the frontend.

Edit: on second thoughts, have just submitted a ticket to get the user-agent added as a feature in hoppscotch directly, so will just push up a fix with the X-Forwarded-For and Via headers (the others seem unnecessary).

SamJakob commented 2 years ago

Pushed up a commit that should resolve this, please let me know if it does not or if there are changes needed by replying to this thread.

rizdaprasetya commented 2 years ago

Lookin good for now, thanks man! 👍

hoppscotch / proxyscotch