Open fishcharlie opened 5 years ago
Hey, thank for the report. I will check it out today ๐
Hey @fishcharlie, I've checked and the answer is that we have to explicitly point the axios
to Proxyman.
const axios = require('axios');
// Proxy to Proxyman
axios.defaults.proxy = {
host: '127.0.0.1',
port: 9090,
}
// Make a request for a user with a given ID
axios.get('https://httpbin.org/get?data=123&value=nghia')
.then(function (response) {
// handle success
console.log(response.data);
})
.catch(function (error) {
// handle error
console.log(error);
})
.finally(function () {
// always executed
});
then, the request will appear in Proxyman
It's relevant to https://github.com/ProxymanApp/Proxyman/issues/175#issuecomment-515282923, since some CLI doesn't use System Proxy until we explicitly override it.
Additionally, we have to do same for Charles Proxy too.
Hope that could help you ๐ ๐ฎ
If you don't want to explicitly point the axios to Proxyman, we can use
$ export http_proxy=http://127.0.01:9090
$ export https_proxy=http://127.0.01:9090
Then all CLIs (curl, nodejs, ...) will automatically proxy to Proxyman, but make sure we remove it when we don't use Proxyman. Or the app will be failed ๐
actually its working on mine. If we use terminal it wont work directly we need to explicitly say axios to use proxy and that is how it work in other cli apps too :) Every cli supports proxy like curl etc.. And as u have already given implicit way using environment variable its fine.
But please add it in request so other don't get same problem.
I'm getting this when ever I try both the environmental vars and the axios proxy. Any ideas?
Error: socket hang up
at connResetException (internal/errors.js:610:14)
at Socket.socketOnEnd (_http_client.js:453:23)
at Socket.emit (events.js:327:22)
at Socket.EventEmitter.emit (domain.js:483:12)
at endReadableNT (_stream_readable.js:1220:12)
at processTicksAndRejections (internal/process/task_queues.js:84:21) {
code: 'ECONNRESET',
@alnorris it can be your server.
@alnorris it might be your port is different than the listening port of Proxyman ( default is 9090)
You can go to proxyman Preference and see the port, then setting it correctly
If you donโt mind, pls do the same config with other proxy tools to see where the bug is ๐ because there are many hidden factors could affect
Is there a workaround to enable the proxy for everything? I sometimes have no influence on what the node app is doing, so having an option to transparently proxy everything would be nice.
Right now I am falling back to mitmproxy which has a workaround using pfctl
and transparent mode to route traffic system-wide to the proxy. Works pretty good, but if Proxyman could do that out of the box, it would be reaaaally nice. Setup like this: https://docs.mitmproxy.org/stable/howto-transparent/#macos
@dvcrn it still wont work if they enable certs pinning (Like zoom app enable cert pinning). And transparent proxy is different thing than http/https proxy. And regarding node app you will have source code right so you can change proxy easily from source code? But yes transparent proxy can help in many situations like some app that don't obey proxy but it still isn't easy today due to certs pinning, android certs being hard to place, and we can forget ios app as most of them use certs pinning anyway. I know we can use frida but thats not easy.
I haven't tested pfctl
, but wondering what happens if we use 9090 in the following script
rdr pass on en0 inet proto tcp to any port {80, 443} -> 127.0.0.1 port 9090
9090 is where Proxyman serves
I don't always have access to the source. For example I was pentesting one mac app but it just wasn't showing up in proxyman/charles and I had no idea why. It took me a while to figure out that the app was spawning a node worker in the background that does the connection and that's why nothing was showing up, which got me wondering what else is happening that I'm missing with Proxyman.
Using mitmproxy + pfctl worked fine even for SSL after trusting the certificate globally
@dvcrn did u use pfctl to forward to proxyman? BSD uses pf firewall which is a bit different than iptables unfortunately.
He forwarded it to mitmproxy @shirshak55. Btw, it's a cool idea to use pfctl to do transparent Proxy.
From what I see, if pfctl can work with Proxyman, we can fix the VPN issue ๐
@NghiaTranUIT pfctl is just a firewall :) And you should have transparent proxy support on proxyman (I think at the moment proxyman supports Http/Https proxy only). pfctl isn't any magic its like iptables for osx. Buy the way if you land transparant proxy soon let me know I can help you with docs regarding how to connect andriod with that transparent proxy.
There is a workaround by using proxychains to swamp your app under proxychain umbrella , and all traffic will appear on Proxyman. Read more at https://github.com/ProxymanApp/Proxyman/issues/544
It's quite difficult to set up and requires to disable SIP on your mac, so I would not recommend this approach.
@dvcrn I don't know why you are saying charles proxy doesn't work. Charles proxy has transparent proxy so it should have covered your usecase. Its hidden in ssl settings second tab. I haven't used it for year probably but charles proxy does support transparent proxy.
Seems like Node.js doesn't respect the http_proxy
/https_proxy
variables; instead, you need to use something like global-agent. This is what ended up working in my case.
Step by step:
npm install global-agent
require("global-agent/bootstrap")
(or see their docs for how to inject it from the CLI)export GLOBAL_AGENT_HTTP_PROXY=http://127.0.01:9090
This should be it!
For those interested, there are lengthy issues on the subject of adding support for the standard proxy vars to Node: https://github.com/nodejs/node/issues/8381, https://github.com/nodejs/node/issues/15620
Thank you so much for the contribution ๐ I will add your comment to official Proxyman doc
Has this been fixed yet such that even javascript requests in browsers or some apps which I don't have code access to will work, and have the network requests show up?
@CyberMew it's not a bug, so I can't fix it. It's how NodeJS works.
If you could not access the source code to change the HTTP Proxy, please try to override on your terminal.
export http_proxy=127.0.0.1:9090
export HTTP_PROXY=127.0.0.1:9090
export https_proxy=127.0.0.1:9090
export HTTPS_PROXY=127.0.0.1:9090
It is not 100% work if your app doesn't respect the env http_proxy
Ref: https://www.serverlab.ca/tutorials/osx/administration-osx/configuring-a-network-proxy-for-osx/
I wanted to check if people have since found a workaround for this. I had tried exporting HTTP(S) proxy variables, and using the global-agent approach, but didn't have any luck. Have people been successful with any other techniques?
Also it looks like the issues referenced above about adding proxy support to Node have been closed, so I'm wondering if people are aware of any other discussions about making that happen.
The solution is depended on what network library you're using on NodeJS. Can you share with me what the lib is? I might google to find a solution ๐
Thanks @NghiaTranUIT! I'm specifically working on a Raycast extension using their useFetch
hook. It's not currently open source, but the team says the API follows the node-fetch API.
Google shows that we can pass the HTTPAgent to tell node-fetch
to use the Proxy, like this code:
const fetch = require('node-fetch');
const HttpsProxyAgent = require('https-proxy-agent');
(async () => {
const proxyAgent = new HttpsProxyAgent('http://localhost:9090');
const response = await fetch('https://httpbin.org/ip?json', { agent: proxyAgent});
const body = await response.text();
console.log(body);
})();
However, useFetch wraps the node-fetch
lib, so I'm not sure how to pass the HttpsProxyAgent
. Maybe you should ask Raycast devs for hints.
@NghiaTranUIT if its react, shouldn't it be running in Browser? So, setting global proxies from windows etc. should work?
@shirshak55 If it's a React, which access from the web browser or using NodeJS to serve a static file (Build from ReactJS). Proxyman can capture it since the web browser would use the system HTTP Proxy. (Localhost would pass the proxy by default, but there is a solution to fix it).
@djpowers case is different, he is making a request from NodeJS. Thus, it doesn't respect the global/system HTTP proxy.
I also had to run
process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = '0';
set the shell env vars,
master ๎ฐ pnpm start
> request-frontend@0.0.1 start /Users/gkatsanos/repos/visable/requests-frontend
> NODE_OPTIONS='-r dd-trace/init' HOST=0.0.0.0 PORT=3000 node ./dist/server/entry.mjs
HTTP_PROXY 192.168.1.107:9090
HTTPS_PROXY 192.168.1.107:9090
Server listening on http://0.0.0.0:3000
#
127.0.0.1 localhost 255.255.255.255 broadcasthost ::1 localhost
127.0.0.1 www.wlw-local.de
127.0.0.1 local.wlw.de local.wlw.at local.wlw.ch api.local.visable.com 127.0.0.1 redirect.local.wlw.de redirect.local.wlw.at redirect.local.wlw.ch 127.0.0.1 api.local.visable.com 127.0.0.1 aws.local.visable.com 127.0.0.1 www.local.europages.co.uk
- [x] tried: http://0.0.0.0:3000/status/health , no logs
- [x] http://proxyman.local:3000/ - same, no logs
- [x] http://localhost.proxyman.io:3000/requests-frontend/status/health - no logs
Could someone explain why would someone buy this product which is supposed to work out of the box but doesnt work even after 3-4 workarounds/hacks?
I'm super disappointed about the effort this requires to get to work. Who is this product for? If its for developers then you're clearly doing something wrong.
Could someone explain why would someone buy this product which is supposed to work out of the box but doesnt work even after 3-4 workarounds/hacks?
Sorry @gkatsanos, the problem is from the NodeJS, it doesn't respect the system HTTP Proxy, so it doesn't work out of the box. If you develop iOS/Android app, it works out of the box.
Your /etc/host looks like doesn't have the following line:
127.0.0.1 proxyman.local
::1 proxyman.local
Thus, http://proxyman.local:3000/ doesn't work.
But then why recommend to people to set the HTTP_PROXY environment variables in their shell? I feel everytime I try to use Proxyman I'm ping ponged from hack to hack..
It's because NodeJS development is really messy. Some libraries require the HTTP_PROXY env, some don't respect the System Proxy -> Needs to manually config in their source code, and some libs do respect the proxy -> Proxyman can work out of the box.
There is no silver bullet to fix all NodeJS problems automatically.
Proxyman helps you a lot behind the scene that you don't notice, such as auto install the certificate on your Mac, auto overriding/reverting the system proxy, or auto-scripts for iOS, and Android, ...
You can try other tools, Charles Proxy, Fiddler, Wireshark, and you have to google a lot to make it works with NodeJS too.
I understand. Concluding, could we have maybe a dedicated NodeJS-specific docs/FAQ page on how to fix it since it's probably a big chunk of the audience?
@gkatsanos it depends on your setup. If you don't mind, what is the NodeJS web framework you're using?
Here is a simple sample:
const express = require('express')
const app = express()
const port = 3000
app.get('/', (req, res) => { res.send('Hello World!') })
app.listen(port, () => {
console.log(Example app listening on port ${port}
)
})
2. Open /etc/hosts and add the following text:
127.0.0.1 proxyman.local ::1 proxyman.local
3. Done. Use Google Chrome, visit `http://proxyman.local:3000` -> Proxyman will capture the traffic as usual.
![CleanShot 2023-04-15 at 10 18 40@2x](https://user-images.githubusercontent.com/5878421/232180487-985733a0-516f-4e85-9bf6-b3e8b1903ecb.jpg)
----------------
If you;re using fetch or axios to call to 3rd-server, and you'd like to capture these traffic. Please check this doc: https://docs.proxyman.io/debug-devices/nodejs
@gkatsanos I've updated the NodeJS Documentation to make it clears what the Proxyman & solution: https://docs.proxyman.io/debug-devices/nodejs
thank you , that worked.
@gkatsanos thanks for your suggestion. I've found the solution to Automatically capture all traffic that is called from the Terminal app.
It can work with NodeJS, cURL, Ruby, and Python without any manual config ๐ฏ
For example: Just open Proxyman -> Click on "Terminal" -> New Terminal/iTerm shows up -> Run your NodeJS server here -> Traffic is automatically captured and send to Proxyman ๐
I'm going to play around and support this feature now. It will resolve all pain we have ๐
Sounds awesome! is it possible to start a terminal already now from the current Proxyman version or is this a future feature? :)
@gkatsanos here is a sneak peek of what I'm doing ๐
--proxy
flag), and NodeJS (including fetch
, axios
, request
, and superagent
libraries).fetch
to call to an external API=> โ Proxyman captures it automatically => โ User doesn't need to modify any source code.
https://user-images.githubusercontent.com/5878421/232966680-20327230-2818-49a2-b671-5b8f2b6c3f38.mp4
@gkatsanos just wondering: What's the NodeJS network library you're using? Currently, in the Beta build, It supports axios, got, superagent, fetch (node v18), node-fetch, and a built-in NodeJS https
package.
It works out of the box โ (No Proxy config, no certificate)
@NghiaTranUIT https://docs.astro.build/en/guides/data-fetching/ I think it uses some form of native fetch
Good news: @gkatsanos @sonic1981 @fishcharlie @dvcrn @shirshak55 @Cykelero
Proxyman can:
out of the box
โ
Beta: https://download.proxyman.io/beta/Proxyman_4.6.1_Feature:_Automatic_Setup.dmg
https://docs.proxyman.io/automatic-setup/automatic-setup
aiohttp
library). NodeJS is similar.@gkatsanos if you don't mind, please give it a try. I've tested with NodeJS (fetch). Proxyman can see traffic from fetch
requests ๐
trying this now, i think its working to capture requests, but it's not actually forwarding the request to the original server from what i can tell. im getting a HTTP/1.1 301 Moved Permanently
on all outgoing requests from inside the electron app. i started the app from the terminal via open ....xdyz/Title.app
i don't have this issue in httptoolkit
it looks like maybe proxyman is not following the location (L
parameter in curl)
so the original request is http
and it's not following to https
@Geczy can you share with me how your setup your Electron app ๐ค
You have the source code, which allows you to start the Electron app from the Terminal, right? Or you'd like to capture traffic from other Electron app?
From what I know, I need to support ElectronJS. The above Beta build is only for NodeJS, Ruby, and Python.
well previously in the production release, the electron app did not even display in proxyman as an "app" or any of the domains or anything. it's only captured if i use the terminal that proxyman opens to open the app.
i start this app by doing open /System/Volumes/Data/Users/matt/Applications/xyz.app
in the spawned terminal
in this build, it does display. the electron app uses axios under the hood, if that matters. i don't have the source code for it.
I see, if you don't mind, can you share with me the name of your Electron app via nghia@proxyman.io ? I'd like to download production build, test it and fix it ๐
sure, i sent you the email. thanks for taking a look. super sexy app
https://github.com/ProxymanApp/Proxyman/issues/1611#issuecomment-1521044841
Just a friendly reminder that we've released the automatic / manual setup ๐
cc @gkatsanos @sonic1981 @fishcharlie @dvcrn @shirshak55 @Cykelero
You can checkout the latest build or from this comment: https://github.com/ProxymanApp/Proxyman/issues/236#issuecomment-1518572201
Proxyman version? (Ex. Proxyman 1.4.3)
Proxyman 1.4.7
macOS Version? (Ex. mac 10.14)
macOS 10.15 Beta (19A526h)
Steps to reproduce
Expected behavior
Request from Node.js to display in Proxyman
Screenshots (optional)
NA