[BUG] Scripting Tool cannot handle URL querystring parameters containing square brackets

michaelhogg commented 3 years ago

Proxyman version? (Ex. Proxyman 1.4.3)

2.19.0 (21900)

macOS Version? (Ex. mac 10.14)

10.14.3

Steps to reproduce

Enable SSL proxying for domain deliver.kenticocloud.com:
1. Tools menu > SSL Proxying List.
2. Check the Enabled SSL Proxying Tool checkbox.
3. Add the domain deliver.kenticocloud.com to the Include List.
Create a script for deliver.kenticocloud.com:
1. Scripting menu > Script List
2. Create a new rule, and configure it with:
  - URL = *deliver.kenticocloud.com*
  - Method = GET
  - Pattern matching = Use Wildcard
  - Leave the Include all subpaths of this URL checkbox unchecked
  - Run Script on = Response only
  - Script:
```
function onResponse(context, url, request, response) {
console.log('response.statusCode = ' + response.statusCode);
return response;
}
```
3. Click the Save & Activate button.
Click the Console button to open the Scripting Console.

In your browser (eg: Chrome), load this URL:

https://deliver.kenticocloud.com/5ac93d1e-567d-01e6-e3b7-ac435f77b907/items?system.id[eq]=a5f68d0f-6c6c-4845-b4d4-6bda702f0c1c

In the Scripting Console, the following error is displayed:

----------------------------------
01:23:48.444: [onResponse] with Request ID=15
❌Error: Couldn not build JSObject with URL=https://deliver.kenticocloud.com/5ac93d1e-567d-01e6-e3b7-ac435f77b907/items?system.id[eq]=a5f68d0f-6c6c-4845-b4d4-6bda702f0c1c

Expected behavior

The Scripting Console should display:

----------------------------------
01:23:48.444: [onResponse] with Request ID=15
response.statusCode = 200
01:23:48.446 onResponse() is executed!

Screenshot

Further details

If I remove [eq] from the URL:

- https://deliver.kenticocloud.com/5ac93d1e-567d-01e6-e3b7-ac435f77b907/items?system.id[eq]=a5f68d0f-6c6c-4845-b4d4-6bda702f0c1c
+ https://deliver.kenticocloud.com/5ac93d1e-567d-01e6-e3b7-ac435f77b907/items?system.id=a5f68d0f-6c6c-4845-b4d4-6bda702f0c1c

and then load this modified URL in Chrome, the Scripting Console displays the expected output:

----------------------------------
01:26:07.738: [onResponse] with Request ID=20
response.statusCode = 200
01:26:07.740 onResponse() is executed!

So it seems to be the [eq] querystring parameter in square brackets which is causing the Couldn not build JSObject with URL error :x:

There is some discussion online about whether square brackets should be percent-encoded ([] encoded as %5B%5D) to make a valid URL:

[Stack Overflow] Is array syntax using square brackets in URL query strings valid?
- "For many years, every browser has left square brackets in query strings unencoded when submitting the request to the server. As such, I imagine a huge portion of the web has come to rely on this behavior, which makes it extremely unlikely to change."

As an example, the Kentico Kontent Delivery API expects unencoded square brackets:

https://docs.kontent.ai/reference/delivery-api#tag/Filtering-content
- system.type[eq]=article
- elements.price[lt]=10
- elements.price[range]=10.5,50

So I think Proxyman's Scripting Tool needs to be able to support unencoded square brackets in querystring parameters, without crashing with the Couldn not build JSObject with URL error.

NghiaTranUIT commented 3 years ago

Hey @michaelhogg, thanks for your detailed bug report. I love it 😄

I will check it out to make sure the Scripting is able to parse the unencoded square brackets and send you a BETA build this week 👍

NghiaTranUIT commented 3 years ago

@michaelhogg Please check out this beta build: https://proxyman.s3.us-east-2.amazonaws.com/beta/Proxyman_2.19.0_Fix_URL_Scripting.dmg

Please let me know if it works for you 😄

michaelhogg commented 3 years ago

Hey @NghiaTranUIT! :wave: Wow that was a quick response! :smiley:

Thank you so much for implementing a fix and sending me that beta build :star2: I can confirm that my test URL (with the unencoded square brackets) is now working! :white_check_mark:

https://deliver.kenticocloud.com/5ac93d1e-567d-01e6-e3b7-ac435f77b907/items?system.id[eq]=a5f68d0f-6c6c-4845-b4d4-6bda702f0c1c

function onResponse(context, url, request, response) {
  console.log('response.statusCode = ' + response.statusCode);
  console.log('url = ' + url);
  return response;
}

----------------------------------
19:08:21.029: [onResponse] with Request ID=41
response.statusCode = 200
url = https://deliver.kenticocloud.com/5ac93d1e-567d-01e6-e3b7-ac435f77b907/items?system.id[eq]=a5f68d0f-6c6c-4845-b4d4-6bda702f0c1c
19:08:21.033 onResponse() is executed!

I really appreciate you fixing this bug so quickly – thank you! :1st_place_medal: :tada:

Proxyman is really awesome and I've just purchased a license today :slightly_smiling_face:

Proxyman registration

NghiaTranUIT commented 3 years ago

Wow. Thank you @michaelhogg so much for supporting Proxyman Team ❤️ 🌮

Feel free to ping us if you find any problems 😄

ProxymanApp / Proxyman