Open thom-nic opened 4 years ago
I wrote a quick test to ensure the DNS server works when it can listen properly, but it seems like 9253 isn't reachable at all on your machine.
Would you mind running sw_vers; nc -zv localhost 9253
and providing the result?
$ sw_vers; nc -zv localhost 9253
ProductName: Mac OS X
ProductVersion: 10.15.3
BuildVersion: 19D76
nc: connectx to localhost port 9253 (tcp) failed: Connection refused
nc: connectx to localhost port 9253 (tcp) failed: Connection refused
ETA: Shouldn't 9253 be a UDP socket?
Edit 2: sudo lsof -i :9253
also returns nothing
ETA: Shouldn't 9253 be a UDP socket?
puma-dev
's DNS is intended to serve both TCP and UDP. https://github.com/puma/puma-dev/blob/master/dev/dns.go#L75-L85
Running two copies of puma-dev
in the foreground, puma-dev
bails on the http/s port, but it doesn't bail on the DNS port.
So, I expect puma-dev
is failing to bind to 9253 and failing silently.
# keep this one running
$ puma-dev
2020/03/26 15:41:02 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/norton/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: 9280
* HTTPS Server port: 9283
! Puma dev listening on http and https
# attempt no. 1
$ puma-dev
2020/03/26 19:42:16 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/norton/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: 9280
* HTTPS Server port: 9283
! Puma dev listening on http and https
2020/03/26 19:42:16 Error listening: listen tcp 127.0.0.1:9280: bind: address already in use
# attempt no. 2
$ puma-dev -http-port 9000 -https-port 90001
2020/03/26 19:42:48 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/norton/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: 9000
* HTTPS Server port: 90001
! Puma dev listening on http and https
Also, this "works," which is disconcerting.
$ puma-dev -dns-port -1
2020/03/26 20:09:00 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/norton/.puma-dev
* Domains: test
* DNS Server port: -1
* HTTP Server port: 9280
* HTTPS Server port: 9283
! Puma dev listening on http and https
@thom-nic will you try running the following?
puma-dev -dns-port 32000
sudo lsof -i :32000
edit: I realized I'm using GNU Your nc
. Not sure what the equivalent BSD nc
flags are. One sec.lsof
approach is better for checking port binding.
Confirmed that DNS port bind will fail silently. I have a WIP branch that will expose the error messages and bail.
For some reason when I try to run on OSX, puma-dev can't seem to bind to any HTTP/S port...?
puma-dev -dns-port 32000 -http-port 34999 -https-port 34998
2020/03/27 09:57:38 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/thom/.puma-dev
* Domains: test
* DNS Server port: 32000
* HTTP Server port: 34999
* HTTPS Server port: 34998
! Puma dev listening on http and https
2020/03/27 09:57:38 Error listening: listen tcp 127.0.0.1:34999: bind: operation not permitted
This may be part of the root cause of the issue I'm facing. Although the HTTP bind doesn't fail when running as a launch daemon, but then again in that case HTTP/S are being forwarded via launchd socket.
Can you bind to those ports with other user processes e.g. rackup
?
I tried:
$ python3 -m http.server 34999
Serving HTTP on 0.0.0.0 port 34999 (http://0.0.0.0:34999/) ...
This prompt popped up:
(I clicked yes) EDIT Firewall panel:
Oddly enough, nodejs isn't listed here and I do most of my development on node. Unsure why I'm not prompted for that. Or why I'm not prompted when I attempt to run puma-dev in the foreground.
EDIT 2: I manually added puma-dev
in the firewall list and it still fails with the same error.
I have the same issue, the firewall is stopping puma-dev
from binding to the DNS port, however thought it might be worth mentioning that I am still using Mojave (10.14.6).
I suspect it's because puma-dev
is not signed
I can't replicate the blocked behavior on my 10.14 machine. Haven't tried on 10.15 yet.
Can you run the following and report what you see?
$ /usr/libexec/ApplicationFirewall/socketfilterfw --getappblocked $(which puma-dev)
The application is not part of the firewall
And maybe try
$ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add $(which puma-dev)
Application at path ( /path/to/puma/dev/puma-dev ) added to firewall
$ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblockapp $(which puma-dev)
Incoming connection to the application is permitted
Just for the record: it is also broken on my machine running macOS 10.14.6 (Mojave) after installing the following update from Apple:
XProtectPlistConfigData:
Version: 2118
Source: Apple
Install Date: 11.04.20, 18:23
the launchd
still spawns, but routing traffic from .test
domains does not resolve to localhost, but does a request to my regular DNS provider.
Didn't find much about that update from Apple, but it seems to be about blocking possibly malicious processes: https://support.apple.com/en-gb/HT207005
@swiknaba I'm upgrading my mojave machine w/ latest security patches now and will attempt to reproduce.
For anyone who can reproduce the issue and is comfortable at the command-line, I'd ask that you try to disable gatekeeper temporarily and see if that fixes the port binding issue. That, at least, will allow us to test the unsigned hypothesis.
sudo spctl --master-disable
will disable gatekeeper (and bring back the previous "allow from anywhere" option in system preferences).
sudo spctl --master-enable
will then re-enable gatekeeper, once you're done testing.
Finished latest security patches, still not having any issues with port bindings with firewall and gatekeeper enabled. ☹️
+1 Same issue for me.
$ sw_vers; puma-dev
ProductName: Mac OS X
ProductVersion: 10.15.4
BuildVersion: 19E287
2020/04/21 09:35:25 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/noah/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: 9280
* HTTPS Server port: 9283
! Puma dev listening on http and https
2020/04/21 09:35:25 Error listening: listen tcp 127.0.0.1:9280: bind: operation not permitted
I'm going to rename this issue to correctly reflect the port-binding issue.
@NoahFisher note that you had a problem binding to HTTPS, not DNS. And, in puma-dev v0.13, issues binding to dns/https fail silently, so my guess it you weren't able to bind to any of the 3 required ports. if you can install / build from master, you'll get better error messages.
Given I can't recreate this issue, having trouble debugging. Any additional diagnostic information is greatly appreciated.
@nonrational - thanks for the reply. I just built from source and included the error output below.
Is there any other information that would be helpful to provide?
~/go/bin $ ./puma-dev
2020/04/21 17:38:01 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/noah/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: 9280
* HTTPS Server port: 9283
! Puma dev running...
2020/04/21 17:38:01 ! HTTP Server failed: listen tcp 127.0.0.1:9280: bind: operation not permitted
! DNS Server failed: listen tcp 127.0.0.1:9253: bind: operation not permitted
~/go/bin $ ./puma-dev -V
Version: devel (go1.14.2)
I had a working theory that SIP is involved, but if you're getting the same issue running a binary from your home directory, that rules that out.
@NoahFisher grasping at straws here, but
spctl
approach discussed above? what were the results? puma-dev -http-port=13080 -https-port=13443 -dns-port=13053
I'm not sure what the difference between "connection refused" and "operation not permitted" error messages are, but I suspect that every participant in this thread is having a slightly different issue. 🙃
I suspect that every participant in this thread is having a slightly different issue.
@nonrational - fair point! I tried on another machine today and was not able to reproduce.
I tried restarting my machine today and the issue is resolved. 🤷 Thanks for you help!
Maddeningly, I just started getting this error in tests after upgrading to Catalina. 😡 Bright side, might be able to discover root cause, at least for the "not permitted" error. Still not sure about the "connection refused."
Edit: And of course it doesn't fail when being run in a debugger. Edit2: Disabling gatekeeper didn't change anything, but turning off the macOS firewall fixed the error. Edit3: https://github.com/golang/go/blob/83d25c61e45f2d38e1015d6eef50cccc0dc0d9b3/src/net/fd_unix.go#L59 this is the line that generates the error.
@nonrational I use zsh so at first I couldn't get your sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblockapp $(which puma-dev)
to work because I think it was getting confused by the symlink action (installed via brew, so lots of symlinks). I replaced the $(which puma-dev)
with the actual location of the binary and suddently (after a few puma-dev restarts) everything is working again.
For those using homebrew, the binary is somewhere like this: /usr/local/Cellar/puma-dev/0.13/bin/puma-dev
but YMMV based on homebrew install and version of puma-dev, etc.
ProductName: Mac OS X
ProductVersion: 10.15.4
BuildVersion: 19E287
I just hit this as well, and wanted to report that the suggested firewall whitelisting worked, but only with the physical path to puma-dev (homebrew sets a symlink from /usr/local/bin/puma-dev
to /usr/local/Cellar/puma-dev/0.14/bin/puma-dev
). Just going to leave this here in case anyone else hits the same issue:
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/local/Cellar/puma-dev/0.14/bin/puma-dev
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblockapp /usr/local/Cellar/puma-dev/0.14/bin/puma-dev
Ah. Excellent find @stevenkaras! Adding a bit of portability for new versions, since 0.15
is right around the corner...
PUMA_DEV_BIN_PATH="$(brew --prefix puma/puma/puma-dev)/bin/puma-dev" # /usr/local/Cellar/puma-dev/0.14/bin/puma-dev
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add "$PUMA_DEV_BIN_PATH"
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblockapp "$PUMA_DEV_BIN_PATH"
I've had puma-dev
working without issue for a long time. Today it stopped working :( (possibly related to me doing brew update
and brew upgrade
today)
I've tried uninstalling (/etc/resolver/
becomes empty) then reinstalling:
$ puma-dev -V
Version: 0.16.0 (go1.16.2)
$ sw_vers
ProductName: macOS
ProductVersion: 11.3.1
BuildVersion: 20E241
$ sudo puma-dev -setup
Password:
* Configuring /etc/resolver to be owned by me
* Changing '/etc/resolver/test' to be owned by me
$ puma-dev -install
2021/05/07 17:09:43 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Use '/usr/local/bin/puma-dev' as the location of puma-dev
* Installed puma-dev on ports: http 80, https 443
But then in ~/Library/Logs/puma-dev.log
:
2021/05/07 17:09:43 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/me/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: inherited from launchd
* HTTPS Server port: inherited from launchd
! Puma dev running...
! DNS Server failed: listen tcp 127.0.0.1:9253: bind: operation not permitted
Note that here it shows the DNS port not binding.
However a bit different if I run in the foreground:
$ puma-dev
2021/05/07 17:13:35 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/me/.puma-dev
* Domains: test
* DNS Server port: 9253
* HTTP Server port: 9280
* HTTPS Server port: 9283
! Puma dev running...
2021/05/07 17:13:35 ! HTTP Server failed: listen tcp 127.0.0.1:9280: bind: operation not permitted
Here the HTTP port isn't binding.
I have no issue binding an HTTP port e.g. with python3 -m http.server 34999
. Only puma-dev has an issue it seems.
I have tried repeating all this after running this:
PUMA_DEV_BIN_PATH="$(brew --prefix puma/puma/puma-dev)/bin/puma-dev"
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add "$PUMA_DEV_BIN_PATH"
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblockapp "$PUMA_DEV_BIN_PATH"
Didn't help.
Same issue with custom ports:
puma-dev -dns-port 32000 -http-port 34999 -https-port 34998
2021/05/07 17:18:25 Existing valid puma-dev CA keypair found. Assuming previously trusted.
* Directory for apps: /Users/me/.puma-dev
* Domains: test
* DNS Server port: 32000
* HTTP Server port: 34999
* HTTPS Server port: 34998
! Puma dev running...
! HTTPS Server failed: listen tcp 127.0.0.1:34998: bind: operation not permitted
2021/05/07 17:18:25 ! HTTP Server failed: listen tcp 127.0.0.1:34999: bind: operation not permitted
Any advice or suggestions?
After a while I also see a few lines like this in puma-dev.log
:
2021/05/07 20:46:05 http: TLS handshake error from 127.0.0.1:57366: EOF
2021/05/07 20:46:06 http: TLS handshake error from 127.0.0.1:57367: EOF
2021/05/07 20:46:06 http: TLS handshake error from 127.0.0.1:57368: tls: client using inappropriate protocol fallback
2021/05/07 20:47:38 http: TLS handshake error from 127.0.0.1:57418: EOF
2021/05/07 20:48:04 http: TLS handshake error from 127.0.0.1:57434: EOF
2021/05/07 20:48:04 http: TLS handshake error from 127.0.0.1:57435: EOF
And....... of course, after restarting my computer a SECOND time, it all works again. (I restarted once before posting everything above). Sorry for the noise.
This is so much wtf. macOS just gives up after a while? (This is on macOS Ventura)
Yeah, I was playing around with puma-dev, installing and uninstalling it, building from source etc. (e.g. taking https://github.com/puma/puma-dev/pull/322 for a spin).
I can run the net/http
example program just fine: https://pkg.go.dev/net/http#hdr-Servers, but if I replace the whole Serve
method https://github.com/puma/puma-dev/blob/db9ec15c986ec978c2534d628e8694a8e5efc250/dev/http_darwin.go#L51-L75
with http.ListenAndServe(":8080", nil)
, it still fails 🤯 2023/09/26 20:14:56 listen tcp :8080: bind: operation not permitted
Meanwhile, on my other macOS laptop, I just checked out puma-dev from source, built it and ran puma-dev -debug
and it is working just fine. This is macOS Monterey though.
I guess I will be soon rebooting my Ventura laptop...
Puma dev's DNS resolution doesn't appear to work at all anymore on MacOS Catalina.
I've done
puma-dev -cleanup
and even uninstalled/reinstalled from homebrew, thensudo puma-dev -setup; puma-dev -install
again.However everything looks good otherwise. Puma is running, resolver file in place, responds to HTTP but not DNS on port
Summary: HTTP proxy works, but DNS does not.