Node.js requests don't display in Proxyman

[fishcharlie]

Proxyman version? (Ex. Proxyman 1.4.3)

Proxyman 1.4.7

macOS Version? (Ex. mac 10.14)

macOS 10.15 Beta (19A526h)

Steps to reproduce

1. Open Proxyman
2. Create request in Node.js (ex. https://github.com/axios/axios)
3. Run Node.js script
4. Observe that request doesn't display in Proxyman

Expected behavior

Request from Node.js to display in Proxyman

Screenshots (optional)

NA

[NghiaTranUIT]

Hey, thank for the report. I will check it out today 👍

[NghiaTranUIT]

Hey @fishcharlie, I've checked and the answer is that we have to explicitly point the axios to Proxyman.

const axios = require('axios');

// Proxy to Proxyman
axios.defaults.proxy = {
    host: '127.0.0.1',
    port: 9090,
}

// Make a request for a user with a given ID
axios.get('https://httpbin.org/get?data=123&value=nghia')
  .then(function (response) {
    // handle success
    console.log(response.data);
  })
  .catch(function (error) {
    // handle error
    console.log(error);
  })
  .finally(function () {
    // always executed
  });

then, the request will appear in Proxyman

It's relevant to https://github.com/ProxymanApp/Proxyman/issues/175#issuecomment-515282923, since some CLI doesn't use System Proxy until we explicitly override it.

Additionally, we have to do same for Charles Proxy too.

Hope that could help you 😄 🌮

[NghiaTranUIT]

If you don't want to explicitly point the axios to Proxyman, we can use

$ export http_proxy=http://127.0.01:9090
$ export https_proxy=http://127.0.01:9090

Then all CLIs (curl, nodejs, ...) will automatically proxy to Proxyman, but make sure we remove it when we don't use Proxyman. Or the app will be failed 😄

[shirshak55]

actually its working on mine. If we use terminal it wont work directly we need to explicitly say axios to use proxy and that is how it work in other cli apps too :) Every cli supports proxy like curl etc.. And as u have already given implicit way using environment variable its fine.

But please add it in request so other don't get same problem.

[alnorris]

I'm getting this when ever I try both the environmental vars and the axios proxy. Any ideas?

Error: socket hang up
    at connResetException (internal/errors.js:610:14)
    at Socket.socketOnEnd (_http_client.js:453:23)
    at Socket.emit (events.js:327:22)
    at Socket.EventEmitter.emit (domain.js:483:12)
    at endReadableNT (_stream_readable.js:1220:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  code: 'ECONNRESET',

[shirshak55]

@alnorris it can be your server.

[NghiaTranUIT]

@alnorris it might be your port is different than the listening port of Proxyman ( default is 9090)

You can go to proxyman Preference and see the port, then setting it correctly

[NghiaTranUIT]

If you don’t mind, pls do the same config with other proxy tools to see where the bug is 😄 because there are many hidden factors could affect

[dvcrn]

Is there a workaround to enable the proxy for everything? I sometimes have no influence on what the node app is doing, so having an option to transparently proxy everything would be nice.

Right now I am falling back to mitmproxy which has a workaround using pfctl and transparent mode to route traffic system-wide to the proxy. Works pretty good, but if Proxyman could do that out of the box, it would be reaaaally nice. Setup like this: https://docs.mitmproxy.org/stable/howto-transparent/#macos

[shirshak55]

@dvcrn it still wont work if they enable certs pinning (Like zoom app enable cert pinning). And transparent proxy is different thing than http/https proxy. And regarding node app you will have source code right so you can change proxy easily from source code? But yes transparent proxy can help in many situations like some app that don't obey proxy but it still isn't easy today due to certs pinning, android certs being hard to place, and we can forget ios app as most of them use certs pinning anyway. I know we can use frida but thats not easy.

[NghiaTranUIT]

I haven't tested pfctl, but wondering what happens if we use 9090 in the following script

rdr pass on en0 inet proto tcp to any port {80, 443} -> 127.0.0.1 port 9090

9090 is where Proxyman serves

[dvcrn]

I don't always have access to the source. For example I was pentesting one mac app but it just wasn't showing up in proxyman/charles and I had no idea why. It took me a while to figure out that the app was spawning a node worker in the background that does the connection and that's why nothing was showing up, which got me wondering what else is happening that I'm missing with Proxyman.

Using mitmproxy + pfctl worked fine even for SSL after trusting the certificate globally

[shirshak55]

@dvcrn did u use pfctl to forward to proxyman? BSD uses pf firewall which is a bit different than iptables unfortunately.

[NghiaTranUIT]

He forwarded it to mitmproxy @shirshak55. Btw, it's a cool idea to use pfctl to do transparent Proxy.

From what I see, if pfctl can work with Proxyman, we can fix the VPN issue 😄

[shirshak55]

@NghiaTranUIT pfctl is just a firewall :) And you should have transparent proxy support on proxyman (I think at the moment proxyman supports Http/Https proxy only). pfctl isn't any magic its like iptables for osx. Buy the way if you land transparant proxy soon let me know I can help you with docs regarding how to connect andriod with that transparent proxy.

[NghiaTranUIT]

There is a workaround by using proxychains to swamp your app under proxychain umbrella , and all traffic will appear on Proxyman. Read more at https://github.com/ProxymanApp/Proxyman/issues/544

It's quite difficult to set up and requires to disable SIP on your mac, so I would not recommend this approach.

[shirshak55]

@dvcrn I don't know why you are saying charles proxy doesn't work. Charles proxy has transparent proxy so it should have covered your usecase. Its hidden in ssl settings second tab. I haven't used it for year probably but charles proxy does support transparent proxy.

[Cykelero]

Seems like Node.js doesn't respect the http_proxy/https_proxy variables; instead, you need to use something like global-agent. This is what ended up working in my case.

Step by step:

1. Install global agent: npm install global-agent
2. Import it in your code: require("global-agent/bootstrap") (or see their docs for how to inject it from the CLI)
3. Set its env variable to Proxyman's server: export GLOBAL_AGENT_HTTP_PROXY=http://127.0.01:9090

This should be it!

For those interested, there are lengthy issues on the subject of adding support for the standard proxy vars to Node: https://github.com/nodejs/node/issues/8381, https://github.com/nodejs/node/issues/15620

[NghiaTranUIT]

Thank you so much for the contribution 🎉 I will add your comment to official Proxyman doc

[CyberMew]

Has this been fixed yet such that even javascript requests in browsers or some apps which I don't have code access to will work, and have the network requests show up?

[NghiaTranUIT]

@CyberMew it's not a bug, so I can't fix it. It's how NodeJS works.

If you could not access the source code to change the HTTP Proxy, please try to override on your terminal.

export http_proxy=127.0.0.1:9090
export HTTP_PROXY=127.0.0.1:9090
export https_proxy=127.0.0.1:9090
export HTTPS_PROXY=127.0.0.1:9090

It is not 100% work if your app doesn't respect the env http_proxy

Ref: https://www.serverlab.ca/tutorials/osx/administration-osx/configuring-a-network-proxy-for-osx/

[djpowers]

I wanted to check if people have since found a workaround for this. I had tried exporting HTTP(S) proxy variables, and using the global-agent approach, but didn't have any luck. Have people been successful with any other techniques?

Also it looks like the issues referenced above about adding proxy support to Node have been closed, so I'm wondering if people are aware of any other discussions about making that happen.

[NghiaTranUIT]

The solution is depended on what network library you're using on NodeJS. Can you share with me what the lib is? I might google to find a solution 👍

[djpowers]

Thanks @NghiaTranUIT! I'm specifically working on a Raycast extension using their useFetch hook. It's not currently open source, but the team says the API follows the node-fetch API.

[NghiaTranUIT]

Google shows that we can pass the HTTPAgent to tell node-fetch to use the Proxy, like this code:

const fetch = require('node-fetch');
const HttpsProxyAgent = require('https-proxy-agent');

(async () => {
    const proxyAgent = new HttpsProxyAgent('http://localhost:9090');
    const response = await fetch('https://httpbin.org/ip?json', { agent: proxyAgent});
    const body = await response.text();
    console.log(body);
})();

However, useFetch wraps the node-fetch lib, so I'm not sure how to pass the HttpsProxyAgent. Maybe you should ask Raycast devs for hints.

[shirshak55]

@NghiaTranUIT if its react, shouldn't it be running in Browser? So, setting global proxies from windows etc. should work?

[NghiaTranUIT]

@shirshak55 If it's a React, which access from the web browser or using NodeJS to serve a static file (Build from ReactJS). Proxyman can capture it since the web browser would use the system HTTP Proxy. (Localhost would pass the proxy by default, but there is a solution to fix it).

@djpowers case is different, he is making a request from NodeJS. Thus, it doesn't respect the global/system HTTP proxy.

[sonic1981]

I also had to run

process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = '0';

[gkatsanos]

set the shell env vars,

master   pnpm start

> request-frontend@0.0.1 start /Users/gkatsanos/repos/visable/requests-frontend
> NODE_OPTIONS='-r dd-trace/init' HOST=0.0.0.0 PORT=3000 node ./dist/server/entry.mjs

HTTP_PROXY 192.168.1.107:9090
HTTPS_PROXY 192.168.1.107:9090
Server listening on http://0.0.0.0:3000

• [x] local bypass in OSX configuration as shown here https://docs.proxyman.io/troubleshooting/.local-doesnt-appear-in-proxyman#2-solution
• [x] added the /etc/hosts entries:

Host Database

#

localhost is used to configure the loopback interface

when the system is booting. Do not change this entry.

127.0.0.1 localhost 255.255.255.255 broadcasthost ::1 localhost

Added by Docker Desktop

127.0.0.1 www.wlw-local.de

START wlw-compose-setup

127.0.0.1 local.wlw.de local.wlw.at local.wlw.ch api.local.visable.com 127.0.0.1 redirect.local.wlw.de redirect.local.wlw.at redirect.local.wlw.ch 127.0.0.1 api.local.visable.com 127.0.0.1 aws.local.visable.com 127.0.0.1 www.local.europages.co.uk

END wlw-compose-setup

- [x] tried: http://0.0.0.0:3000/status/health , no logs
- [x] http://proxyman.local:3000/ - same, no logs
- [x] http://localhost.proxyman.io:3000/requests-frontend/status/health - no logs

Could someone explain why would someone buy this product which is supposed to work out of the box but doesnt work even after 3-4 workarounds/hacks?
I'm super disappointed about the effort this requires to get to work. Who is this product for? If its for developers then you're clearly doing something wrong.

[NghiaTranUIT]

Could someone explain why would someone buy this product which is supposed to work out of the box but doesnt work even after 3-4 workarounds/hacks?

Sorry @gkatsanos, the problem is from the NodeJS, it doesn't respect the system HTTP Proxy, so it doesn't work out of the box. If you develop iOS/Android app, it works out of the box.

Your /etc/host looks like doesn't have the following line:

127.0.0.1 proxyman.local
::1 proxyman.local

Thus, http://proxyman.local:3000/ doesn't work.

Ref: https://docs.proxyman.io/troubleshooting/couldnt-see-any-request-from-localhost-server#solution-2-map-localhost-to-the-domain-name-in-etc-hosts

[gkatsanos]

But then why recommend to people to set the HTTP_PROXY environment variables in their shell? I feel everytime I try to use Proxyman I'm ping ponged from hack to hack..

[NghiaTranUIT]

It's because NodeJS development is really messy. Some libraries require the HTTP_PROXY env, some don't respect the System Proxy -> Needs to manually config in their source code, and some libs do respect the proxy -> Proxyman can work out of the box.

There is no silver bullet to fix all NodeJS problems automatically.

Proxyman helps you a lot behind the scene that you don't notice, such as auto install the certificate on your Mac, auto overriding/reverting the system proxy, or auto-scripts for iOS, and Android, ...

You can try other tools, Charles Proxy, Fiddler, Wireshark, and you have to google a lot to make it works with NodeJS too.

[gkatsanos]

I understand. Concluding, could we have maybe a dedicated NodeJS-specific docs/FAQ page on how to fix it since it's probably a big chunk of the audience?

[NghiaTranUIT]

@gkatsanos it depends on your setup. If you don't mind, what is the NodeJS web framework you're using?

Here is a simple sample:

1. Use ExpressJS to create a localhost server

   
   const express = require('express')
   const app = express()
   const port = 3000

app.get('/', (req, res) => { res.send('Hello World!') })

app.listen(port, () => { console.log(Example app listening on port ${port}) })

2. Open /etc/hosts and add the following text:

127.0.0.1 proxyman.local ::1 proxyman.local

3. Done. Use Google Chrome, visit `http://proxyman.local:3000` -> Proxyman will capture the traffic as usual.

![CleanShot 2023-04-15 at 10 18 40@2x](https://user-images.githubusercontent.com/5878421/232180487-985733a0-516f-4e85-9bf6-b3e8b1903ecb.jpg)

----------------

If you;re using fetch or axios to call to 3rd-server, and you'd like to capture these traffic. Please check this doc: https://docs.proxyman.io/debug-devices/nodejs

[NghiaTranUIT]

@gkatsanos I've updated the NodeJS Documentation to make it clears what the Proxyman & solution: https://docs.proxyman.io/debug-devices/nodejs

[gkatsanos]

thank you , that worked.

[NghiaTranUIT]

@gkatsanos thanks for your suggestion. I've found the solution to Automatically capture all traffic that is called from the Terminal app.

It can work with NodeJS, cURL, Ruby, and Python without any manual config 💯

For example: Just open Proxyman -> Click on "Terminal" -> New Terminal/iTerm shows up -> Run your NodeJS server here -> Traffic is automatically captured and send to Proxyman 👍

I'm going to play around and support this feature now. It will resolve all pain we have 😄

[gkatsanos]

Sounds awesome! is it possible to start a terminal already now from the current Proxyman version or is this a future feature? :)

[NghiaTranUIT]

@gkatsanos here is a sneak peek of what I'm doing 😄

• Proxyman automatically captures all traffic from this Terminal. Support cURL (no need --proxy flag), and NodeJS (including fetch, axios, request, and superagent libraries).

Video Demo

• Setup: Use ExpressJS for a local web server
• Use fetch to call to an external API

=> ✅ Proxyman captures it automatically => ✅ User doesn't need to modify any source code.

https://user-images.githubusercontent.com/5878421/232966680-20327230-2818-49a2-b671-5b8f2b6c3f38.mp4

[NghiaTranUIT]

@gkatsanos just wondering: What's the NodeJS network library you're using? Currently, in the Beta build, It supports axios, got, superagent, fetch (node v18), node-fetch, and a built-in NodeJS https package.

It works out of the box ✅ (No Proxy config, no certificate)

[gkatsanos]

@NghiaTranUIT https://docs.astro.build/en/guides/data-fetching/ I think it uses some form of native fetch

[NghiaTranUIT]

Good news: @gkatsanos @sonic1981 @fishcharlie @dvcrn @shirshak55 @Cykelero

Proxyman can:

• Capture HTTP/HTTPS traffic from NodeJS (axios, fetch, node-fetch, got, superagent), Ruby, and Python out of the box ✅
• Don't need to set the Proxy or trust the certificate manually.
• 1-click and done 🎉

Download

Beta: https://download.proxyman.io/beta/Proxyman_4.6.1_Feature:_Automatic_Setup.dmg

Automatic Setup

1. Proxyman -> Setup menu -> Automatic Setup -> Start the Terminal App
2. Grant AppleScript permission if needed
3. You can start your NodeJS server or run a script on this pre-configured Terminal
4. Done. Enjoy new traffic on Proxyman ✅

Manual Setup

• Just follow the instruction

Documentation:

https://docs.proxyman.io/automatic-setup/automatic-setup

Demo video:

• Use Automatic Setup from Proxyman, to capture traffic from Python (aiohttp library). NodeJS is similar.
• Old approach: The user has to add the Proxy and disable the SSL Verification on their source code.
• New approach: 1-click and done ✅
• Video: https://drive.google.com/file/d/1UrWOJtY-tAFVv_OcgulTBB9qRlojgOnp/view?usp=sharing

[NghiaTranUIT]

@gkatsanos if you don't mind, please give it a try. I've tested with NodeJS (fetch). Proxyman can see traffic from fetch requests 👍

[Geczy]

trying this now, i think its working to capture requests, but it's not actually forwarding the request to the original server from what i can tell. im getting a HTTP/1.1 301 Moved Permanently on all outgoing requests from inside the electron app. i started the app from the terminal via open ....xdyz/Title.app

i don't have this issue in httptoolkit

it looks like maybe proxyman is not following the location (L parameter in curl)

so the original request is http and it's not following to https

[NghiaTranUIT]

@Geczy can you share with me how your setup your Electron app 🤔

You have the source code, which allows you to start the Electron app from the Terminal, right? Or you'd like to capture traffic from other Electron app?

---

From what I know, I need to support ElectronJS. The above Beta build is only for NodeJS, Ruby, and Python.

[Geczy]

well previously in the production release, the electron app did not even display in proxyman as an "app" or any of the domains or anything. it's only captured if i use the terminal that proxyman opens to open the app.

i start this app by doing open /System/Volumes/Data/Users/matt/Applications/xyz.app in the spawned terminal

in this build, it does display. the electron app uses axios under the hood, if that matters. i don't have the source code for it.

[NghiaTranUIT]

I see, if you don't mind, can you share with me the name of your Electron app via nghia@proxyman.io ? I'd like to download production build, test it and fix it 👍

[Geczy]

sure, i sent you the email. thanks for taking a look. super sexy app

[NghiaTranUIT]

New Beta build 🎉

https://github.com/ProxymanApp/Proxyman/issues/1611#issuecomment-1521044841

[NghiaTranUIT]

Just a friendly reminder that we've released the automatic / manual setup 🎉

cc @gkatsanos @sonic1981 @fishcharlie @dvcrn @shirshak55 @Cykelero

You can checkout the latest build or from this comment: https://github.com/ProxymanApp/Proxyman/issues/236#issuecomment-1518572201