Support External Proxy

[NghiaTranUIT]

Description

Some users report that they are using their corporated Proxy Server to access the Internet. They need the External Proxy Tools from macOS to work on Windows.

See how it works on macOS:

• See on Slack

Requirement for the External Proxy

UI

• [x] Menu Item in the Tool menu: Use External Proxy and External Proxy Settings…
• [x] External Proxy Settings UI: Including the Web Proxy, Secure Web Proxy, Enable External Proxy Checkbox, Proxy Details (Host, Port, Require Password, Username, Password)
• [x] Inclusive List for External Proxy
• [x] Bypass List for External Proxy
• [x] Always bypass External Proxy for localhost checkbox
• [x] Cancel and Done

How External Proxy Works

• It's different the the Proxy Setting in the Setting -> Proxy Tab. In this case, it just simply passes the Proxy config to the axios and it's done.
• However, in the case of External Proxy, it's different.
• Here is how it works behind the scenes.
  1. Set up Charles Proxy at port 8080 -> Install the certificate on Mac -> Verify we can decrypt HTTPS response content with Charles Proxy.
  2. Open Proxyman at port 9090 -> Verify Proxyman is overridden the system proxy at port 9090
  3. Make a HTTPS Request to port 9090 via cURL: In this normal case:

     cURL -> Proxyman -> To the Internet

4. On Proxyman -> Tools -> Proxy Setting -> External Proxy Settings

   • Enable External proxy checkbox
   • Tick on HTTP/HTTPS
   • Set 127.0.0.1 at port 8080 (Charles Proxy is a External Proxy Server)

5. Make a cURL to port 9090 (Proxyman). Here is how it works:

5.1. cURL sent a HTTP Request to Proxyman 5.2. Proxyman detects that External Proxy Service is ON 5.3. Proxyman sent CONNECT Request

CONNECT 127.0.0.1:8080 HTTP/1.1
Host: 127.0.0.1:8080
Connection: keep-alive
Proxy-Authorization: basic <username_password_base64> (If the Required Authentication is checked): Read at https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/CONNECT

5.4. Charles receives a new CONNECT and then response:

HTTP/1.1 200 Connection Established

5.5 Proxyman reads this Response from Charles Proxy and starts a tunnel to Charles Proxy 5.6 When the Tunnel is ready, here is how it looks

• cURL -> Proxyman ---(External Proxy)---> Charles PRoxy -> The Internet

• ⚠️ it should work with HTTP, HTTPS (intercept), and proxy HTTPS Mode.

Logic: External Proxy Setting

• [x] Ping @NghiaTranUIT to get ExternalProxyService, ExternalProxySetting, BypassProxyEntry and ExternalProxySettingService and try to port to Windows/Linux

[NghiaTranUIT]

Will add more details requirement for the External Proxy, Bypass Proxy, Include List and the Unit Test soon 👍

[NghiaTranUIT]

External Proxy Service Requirement

• [x] Porting two files: ExternalProxyService, ExternalProxySetting
• [x] HTTP and HTTPS should be 2 separate instances (It doesn't like the Proxy Setting in Setting, they are the same).
• [x] When I select the HTTP or HTTPS row -> It will show the ProxySetting in the detail view properly (See the video)

https://github.com/ProxymanApp/proxyman-windows-linux/assets/5878421/5f0cddb3-dba9-41e8-a0d4-a2d920be7df9

• [x] Hit Done will save or Cancel will cancel my action. The decision is: External Proxy is critical feature, so it needs a user to click on the Done button to confirm it

• [x] Username and Password should be securely stored with a secret password. Ping @NghiaTranUIT to get the file EncryptionService.swift
• [x] As soon as we have the Connection from a given host and port from the ProxyServer, we should call proxySetting(isHTTPS: Bool, host: String, port: Int) -> ExternalProxySetting to return to the correct Proxy for this Host/Port => Verify we carefully port it to TypeScript

---

How to test External Proxy manually

1. Start Charles Proxy at port 8080
2. Start Proxyman at 9090
3. Prepare External Proxy -> HTTP and HTTPS with host = 127.0.0.1, and port = 8080
4. Clear the Bypass Proxy and Include List
5. Set bypass localhost is unchecked

• [x] curl 'https://httpbin.proxyman.app/get?id=123' --proxy http://localhost:9090 -> Verify: httpbin.proxyman.app shows in both Proxyman + Charles Proxy
• [x] curl 'https://www.google.com/' --proxy http://localhost:9090 -> Verify: www.google.com shows in both Proxyman + Charles Proxy
• [x] curl "http://httpbin.org/get?id=123" --proxy localhost:9090 -> Verify: httpbin.org shows in both Proxyman + Charles Proxy
• [x] curl "http://www.producthunt.com" --proxy localhost:9090 -> Verify: www.producthunt.com shows in both Proxyman + Charles Proxy

⚠️ Important Logic

• [x] External Proxy should work with HTTP
• [x] External Proxy should work with HTTPS (SSL Mode or Just CONNECT Mode)
• [x] External Proxy should work with WS and WSS
• [x] All debugging tools should work with External Proxy (Breakpoint, Scripting ...)
• [x] Test with bad scenario: Don't open Charles proxy, but has External Proxy enabled -> Verify we can see error request/response on the main table.

[NghiaTranUIT]

Bypass Proxy Requirement

• [x] Port ExternalProxySettingService.swift
• [x] Verify the ported code from paresHostList(hostList:) works correctly to parse the Host from a given List.
• [x] Verify it supports the wildcard: *.api.com

For example: 127.0.0.1, localhost,data.com,proxyman,io, api.proxyman.com will parse to

127.0.0.1
localhost
data.com
proxyman.io
api.proxyman.io

-> Seperated by the , and trim the whitespace + newline of each entry (Leading & Trailing position)

• [x] Bypass proxy + Bypass Localhost are working with manually steps

How to test Bypass Proxy manually

1. Start Charles Proxy at port 8080
2. Start Proxyman at 9090
3. Prepare External Proxy -> HTTP and HTTPS with host = 127.0.0.1, and port = 8080
4. Add bypass proxy = httpbin.proxyman.app:443, www.google.com
5. Set bypass localhost is unchecked

• [x] curl "https://httpbin.proxyman.app/get?id=123" --proxy localhost:9090 -> Verify: httpbin.proxyman.app only shows in Proxyman, not Charles Proxy (because it matched the Bypass Proxy)
• [x] curl "https://www.google.com" --proxy localhost:9090 -> Verify: www.google.com only shows in Proxyman, not Charles Proxy (because it matched the Bypass Proxy)
• [x] curl "http://httpbin.org/get?id=123" --proxy localhost:9090 -> Verify: httpbin.org only shows in Proxyman + Charles Proxy -> It matches the external Proxy, but doesn't match the bypass proxy
• [x] curl "http://www.producthunt.com" --proxy localhost:9090 -> Verify: www.producthunt.com only shows in Proxyman + Charles Proxy -> It matches the external Proxy, but doesn't match the bypass proxy

---

• [x] Start a local server at port 3000 -> curl "http://localhost:3000" --proxy localhost:9090 -> Verify: localhost:3000 only shows in Proxyman + Charles Proxy -> It matches the external Proxy, but doesn't match the bypass proxy

How to test "Bypass Localhost" manually

1. Start Charles Proxy at port 8080
2. Start Proxyman at 9090
3. Prepare External Proxy -> HTTP and HTTPS with host = 127.0.0.1, and port = 8080
4. Add bypass proxy = httpbin.proxyman.app:443, www.google.com
5. Set bypass localhost is checked ✅

• [x] curl "http://www.producthunt.com" --proxy localhost:9090 -> Verify: www.producthunt.com only shows in Proxyman + Charles Proxy -> It matches the external Proxy, but doesn't match the bypass proxy
• [x] Start a local server at port 3000 -> curl "http://localhost:3000" --proxy localhost:9090 -> Verify: localhost:3000 only shows in Proxyman, not Charles Proxy -> It matches the external Proxy, and matched the Always bypass localhost.

[NghiaTranUIT]

Include List Proxy Requirement

• Include List means: If a given request (host + port) matches with the Include List -> It will go through the External Proxy. Otherwise, just skip it
• [x] This logic is subclassed from ExternalProxySettingService, so it should inherit all logic from the Bypass Proxy

How to test Include List manually

1. Start Charles Proxy at port 8080
2. Start Proxyman at 9090
3. Prepare External Proxy -> HTTP and HTTPS with host = 127.0.0.1, and port = 8080
4. Add Include List = *.app
5. Remove the bypass proxy list
6. Set bypass localhost is unchecked

• [x] curl "https://httpbin.proxyman.app/get?id=123" --proxy localhost:9090 -> Verify: httpbin.proxyman.app shows in Proxyman and Charles Proxy (because it matched the Include List)
• [x] curl "https://www.google.com" --proxy localhost:9090 -> Verify: www.google.com only shows in Proxyman, not Charles Proxy (because it doesn't match the Include List)
• [x] curl "http://httpbin.org/get?id=123" --proxy localhost:9090 -> Verify: httpbin.org only shows in Proxyman, not Charles Proxy (because it doesn't match the Include List)
• [x] curl "http://www.producthunt.com" --proxy localhost:9090 -> Verify: www.producthunt.com only shows in Proxyman, not Charles Proxy (because it doesn't match the Include List)

---

• [x] Start a local server at port 3000 -> curl "http://localhost:3000" --proxy localhost:9090 -> Verify: localhost:3000 only shows in Proxyman, not Charles Proxy -> It matches the external Proxy, but doesn't match the Include List

[NghiaTranUIT]

Unit Tests

• [x] Port ExternalProxyServiceTests.swift, ByPassProxyServiceTests.swift, and IncludeListProxyServiceTests.swift. Ping @NghiaTranUIT to get these files.
• [x] ✅ Important: Write Unit Test to test 3 manually steps (describe above comments) and verify it works

Example:

1. Start Proxyman Server by code
2. Start a local proxy server (can be found on NPM)
3. Set External Proxy to this local proxy server
4. Send a Request by code
5. Verify the `flowPool` of the ProxyCore and the List of passed requests on the Local Server -> Confirm it's passed or not.
6. Repeat with all `cURL`

[NghiaTranUIT]

How to test with External Proxy v2

1. Start Charles Proxy at port 8080 -> Install the certificate -> Verify we can see HTTPS Response with this app
2. Open Proxyman at port 9090 -> Set External Proxy for both HTTP and HTTPS to 127.0.0.1 at 8080
3. Open Google Chrome -> Visit any website, youtube, ...
4. ✅ Verify we can access these websites as usual, not errors
5. ✅ Verify we see the request/response (header, body) by enabling the SSL Proxying on some domains -> Can see it on Proxyman and Charles -> There are no errors in 2 apps.
6. Try using Map Local, Breakpoint on Proxyman -> Verify it's working.