hoppscotch / proxyscotch

šŸ“” A simple proxy server created for https://hoppscotch.io
https://hoppscotch.io
MIT License
360 stars 93 forks source link

[bug]: Unexpected request header modification when using Proxy (Proxyscotch) #46

Closed rizdaprasetya closed 2 years ago

rizdaprasetya commented 2 years ago

Is there an existing issue for this?

Current behavior

On https://hoppscotch.io/ , using this feature Use the proxy middleware to send requests, when I create the following request:

Screen Shot 2022-09-05 at 15 46 39

CURL equivalent:

curl --request POST \
  --url 'https://hookb.in/YVkaPWyrBbCQjy0QmeZp?param=test' \
  --header 'content-type: application/json' \
  --header 'origin: https://mycustomorigin.com' \
  --header 'referer: https://mycustomreferrer.com/' \
  --header 'user-agent: mycustom-UA' \
  --header 'x-source: mycustom-source' \
  --data '{}'

Then I inspect the request on the destination, and get the following result instead:

## HTTP HEADERS 
accept-encoding: gzip
content-length: 2
content-type: application/json
host: hookb.in
origin: https://mycustomorigin.com
referer: https://mycustomreferrer.com/
user-agent: Proxyscotch/1.1
x-source: mycustom-source

## QUERY STRING
param: test

## BODY
{}

unexpectedly some of the headers are modified:

Why is it concerning?

So having this is important.

Maybe instead of Proxy scotch overwriting the actual user-agent header, it should add its own headers e.g. x-requested-via-proxy-agent: Proxyscotch/1.1

Steps to reproduce

  1. Prepare endpoint that can inspect HTTP request. I used https://hookbin.com/. e.g. I will create endpoint & get URL https://hookb.in/YVkaPWyrBbCQjy0QmeZp
  2. On https://hoppscotch.io/ , using this feature Use the proxy middleware to send requests
  3. import CURL provided above, modify the URL with URL you get from step 1.
  4. Send request
  5. Inspect the HTTP request result on the destination (in my case I refreshed the Hookbin page I created on step 1.)
  6. You will get same result as mine above.

Environment

Production

Version

Cloud

AndrewBastin commented 2 years ago

Moving this issue to the Proxyscotch repo.

SamJakob commented 2 years ago

Ah yes! We've not had this yet and that's a good point, we overlooked the use of user-agent. We added the header to help prevent abuse of the proxy and as a curtesy to other services (also CloudFlare blocks requests in some cases if you don't specify a User Agent at all).

I do like your solution though, so will go ahead and implement that.

rizdaprasetya commented 2 years ago

Cool, thanks for confirming! šŸ‘

We added the header to help prevent abuse of the proxy and as a curtesy to other services (also CloudFlare blocks requests in some cases if you don't specify a User Agent at all).

Ah I see, that make sense too. I think need to find the middle ground to satisfy both use-case then.

Maybe

Also x-requested-via-proxy-agent header is totally made up by me šŸ˜… , I have not research far enough, maybe there's a standardized HTTP header for proxy case like this to properly use instead.

Thanks, appreciate the effort!

rizdaprasetya commented 2 years ago

Did some quick search, I think the correct HTTP header to use is Via. According to these resource:

Also if decided to follow that kind of standard, there's a few other to add like x-forwarded-for

SamJakob commented 2 years ago

Okay yeah I'll evaluate adding some or all of those headers then.

Just to confirm, are you in agreement with still using our Proxyscotch/1.0 User-Agent if no user agent header was specified (and Via always be set regardless)?

(This is in order to help prevent CloudFlare from blocking requests automatically which it often does when there's no user agent) ā€“ or do we leave it up to the user to figure it out in case there is a use case for not having the header set?

rizdaprasetya commented 2 years ago

Just to confirm, are you in agreement with still using our Proxyscotch/1.0 User-Agent if no user agent header was specified (and Via always be set regardless)?

āœ… Yep, I personally agree. It maybe also helpful to compare the behavior with other tools like: Postman & CURL. They set their own user-agent as default (when user does not specify). Then they will not overwrite user-agent when user specify it. I think same approach can be used here.

Btw scope wise, IMO ideally:

Just adding insight, I agree whatever approach you choose, as you guys are the main owners of the project & it's specs. āœŒļø

SamJakob commented 2 years ago

Okay yeah that all sounds good. I might look at adding something into the protocol then to allow Hoppscotch to set it from the frontend.

Edit: on second thoughts, have just submitted a ticket to get the user-agent added as a feature in hoppscotch directly, so will just push up a fix with the X-Forwarded-For and Via headers (the others seem unnecessary).

SamJakob commented 2 years ago

Pushed up a commit that should resolve this, please let me know if it does not or if there are changes needed by replying to this thread.

rizdaprasetya commented 2 years ago

Lookin good for now, thanks man! šŸ‘