Scripting changes `+` to `%2B` in query string

[gabrielmaldi]

Description

When using a script like the following to rewrite the URL:

async function onRequest(context, url, request) {
  request.scheme = "http";
  request.host = "myserver";
  request.port = 1234;
  request.path = request.path.replace("/some-path/api/", "/another-path/api/");

  return request;
}

If the URL contains + chars in the query string (note the script doesn't alter request.queries), they get encoded to %2B, which may cause the request to fail on the server.

The Map Remote tool does not have this problem: I can successfully rewrite a request without other unwanted changes.

Steps to Reproduce

Create a script like the above and intercept a request which contains a + char.

The following first image shows the request being rewritten by Map Remote, and se second image shows the same request rewritten by Scripting, which causes the server to fail (the arrows point to the modified chars in the query string).

Current Behavior

+ chars are changed to %2B.

Expected Behavior

+ chars are preserved.

Environment

• App version: Proxyman 4.0.0 (40000)
• macOS version: macOS 13.1 (22C65)

[gabrielmaldi]

Just discovered request.isURLEncoding = false. That somewhat helps: it solves the problem I described above for that specific request, but causes other requests to fail (since as stated will prevent encoding of the query string).

Maybe the solution is to apply these encoding rules? In my case, + should be a space char, which the server would handle correctly (and the problem is that currently Proxyman is encoding it as %2B which is the plus sign char).

Thanks!

[NghiaTranUIT]

@gabrielmaldi by default, the + will always be percented-encoded, it's the standard of the Restful APIs.

If we force to not encode the +, it might cause bug on other apps.

I guess, if you don't modify the query, I will keep reserving the original data without performing the encoding. So it fixes your problem and all people.

[gabrielmaldi]

if you don't modify the query, I will keep reserving the original data without performing the encoding

I agree with that and would solve my issue, thanks!

Additionally, and this is just a suggestion without doing too much investigation (so I may be wrong), + in the query string apparently means a space:

https://www.w3.org/TR/html401/interact/forms.html#h-17.13.4.1

This is what encodeURI does, it preserves + chars.

Anyway, as I said before I'm not sure about this and don't want to invest mucho more time in this issue, so I totally understand (and maybe it's the right thing to do) if you decide to keep the current behavior and just leave the query string alone if it's not modified, which works for me 😊.

Thanks!

[NghiaTranUIT]

@gabrielmaldi if you don't mind, let's try this beta build: https://proxyman.s3.us-east-2.amazonaws.com/beta/Proxyman_4.1.0_Fix_Scripting_Query_Encoding.dmg

If you don't modify the Query, it will be intact, so the + in your query doesn't encode to %2B. If you modify the query, it will encode as it's how Apple framework does.

Please let me know if it works for you 👍

[gabrielmaldi]

@NghiaTranUIT, can confirm the beta works perfectly. Tested both cases (modifying the query string and leaving it alone) and they work as expected (as you described). So this solves the issue for me, thanks!

Perhaps a feature which would help with these scenarios (where you want to change the protocol, host, port, and path of a request) is for Map Remote to support capturing regular expressions. So then in the Path input you could use something like /another-path/new-api/$1 and $1 would get replaced by whatever you captured in Rule. This is not a feature request as my requirement is already fulfilled by the more general Scripting feature, just a suggestion to maybe add to the backlog if you deem it worthwhile 😊.

I think Proxyman is awesome, love every aspect of it, thanks for creating it, working hard and listening to users! If there is a beta update channel I will happily switch to it and report issues I find. Thanks!

[NghiaTranUIT]

Thanks for your kind word @gabrielmaldi 🙇

Capture Regular Expression is a huge change. I will consider to support it 👍

[gabrielmaldi]

If it's huge then I'd say forget about it, we can just use Scripting 😊.

[hatalho]

This exact same issue still occurs in the current version 2.14 for Windows, is there any way to fix it? Thanks

[NghiaTranUIT]

@hatalho I will open a Windows ticket and fix it in the next release 👍

[hatalho]

@hatalho I will open a Windows ticket and fix it in the next release 👍

hi @NghiaTranUIT

2 months have passed since your last update, any news? Thanks!

[NghiaTranUIT]

@kics223w1 can you look on it ?

[hatalho]

It seems to be working now, I'll let you know if I run into any difficulties, thanks!