Open peterhoneder opened 2 years ago
Hello, Homebrew worked for me on macOS 12.6
zeronsd 63175 root 14u IPv4 0x7c52d7942caa8867 0t0 TCP 10.147.19.23:53 (LISTEN)
Are you on Arm? I'm on Intel.
I will try to comb over the issues in detail next week. Been away from the computer for a while.
A knee-jerk on this ticket however suggests that there is a firewall involved. I would look there first.
So, reviewing this again, can you run this command for me and paste the result? It should produce no secrets:
sudo lsof -p `pidof zeronsd`
It would help me diagnose your issue.
One way to reproduce this is to not use be root/sudo when your start zeronsd. For whatever reason macos doesn't complain or crash the program, but the program doesn't work. sudo lsof -P -iUDP | grep zero
does not contain any zeronsd unless you start it with sudo.
It should require root to run. Port 53 is < 1024, and thus a "privileged" port that requires root
apple got rid of privileged ports at some point, but -get this- it's buggy.
https://news.ycombinator.com/item?id=18302380 https://developer.apple.com/forums/thread/674179 (there aren't any actual docs on the topic)
But yeah, zeronsd should exit when it's not root, or not able to use port 53 I guess. I tried on linux and the same thing happens. It starts but doesn't work.
I have a Raspberry Pi 1 and Pi Zero for 2 different zerotier networks running exactly the same zeronsd binary (built over 12 hours with 512M swap) and listens port 53 on one but not on the other. Both are running pi-hole besides zeronsd. Here is the requested output for one that does not listen:
colos@jozsi:~ $ sudo lsof -p pidof zeronsd
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
zeronsd 2343 root cwd DIR 179,2 4096 2 /
zeronsd 2343 root rtd DIR 179,2 4096 2 /
zeronsd 2343 root txt REG 179,2 15558984 391959 /home/colos/zeronsd
zeronsd 2343 root mem REG 179,2 17972 137627 /usr/lib/arm-linux-gnueabihf/libnss_dns-2.31.so
zeronsd 2343 root mem REG 179,2 75568 137744 /usr/lib/arm-linux-gnueabihf/libresolv-2.31.so
zeronsd 2343 root mem REG 179,2 13760 137632 /usr/lib/arm-linux-gnueabihf/libnss_mdns4_minimal.so.2
zeronsd 2343 root mem REG 179,2 42628 137628 /usr/lib/arm-linux-gnueabihf/libnss_files-2.31.so
zeronsd 2343 root mem REG 179,2 1319784 137204 /usr/lib/arm-linux-gnueabihf/libc-2.31.so
zeronsd 2343 root mem REG 179,2 13864 137274 /usr/lib/arm-linux-gnueabihf/libdl-2.31.so
zeronsd 2343 root mem REG 179,2 386572 137539 /usr/lib/arm-linux-gnueabihf/libm-2.31.so
zeronsd 2343 root mem REG 179,2 137364 137721 /usr/lib/arm-linux-gnueabihf/libpthread-2.31.so
zeronsd 2343 root mem REG 179,2 116324 137362 /usr/lib/arm-linux-gnueabihf/libgcc_s.so.1
zeronsd 2343 root mem REG 179,2 2150824 137243 /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.1
zeronsd 2343 root mem REG 179,2 459032 137813 /usr/lib/arm-linux-gnueabihf/libssl.so.1.1
zeronsd 2343 root mem REG 179,2 9512 137135 /usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so
zeronsd 2343 root mem REG 179,2 146888 136992 /usr/lib/arm-linux-gnueabihf/ld-2.31.so
zeronsd 2343 root 0r CHR 1,3 0t0 4 /dev/null
zeronsd 2343 root 1u unix 0xcf31dc27 0t0 43155 type=STREAM
zeronsd 2343 root 2u unix 0xcf31dc27 0t0 43155 type=STREAM
zeronsd 2343 root 3u a_inode 0,13 0 6967 [eventpoll]
zeronsd 2343 root 4u a_inode 0,13 0 6967 [eventfd]
zeronsd 2343 root 5u a_inode 0,13 0 6967 [eventpoll]
zeronsd 2343 root 6u unix 0x2cdb490e 0t0 43171 type=STREAM
zeronsd 2343 root 7u unix 0x2d73d79d 0t0 43172 type=STREAM
zeronsd 2343 root 8u unix 0x2cdb490e 0t0 43171 type=STREAM
zeronsd 2343 root 9u IPv4 43193 0t0 TCP pi.hole:59370->151.101.1.91:https (ESTABLISHED)
colos@jozsi:~ $
And here is the requested output for the one that does listen:
jordana@pi1b:~ $ sudo lsof -p `pidof zeronsd`
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
zeronsd 15574 root cwd DIR 179,2 4096 2 /
zeronsd 15574 root rtd DIR 179,2 4096 2 /
zeronsd 15574 root txt REG 179,2 15558984 3184 /home/jordana/zeronsd
zeronsd 15574 root mem REG 179,2 17972 8494 /usr/lib/arm-linux-gnueabihf/libnss_dns-2.31.so
zeronsd 15574 root mem REG 179,2 75568 8519 /usr/lib/arm-linux-gnueabihf/libresolv-2.31.so
zeronsd 15574 root mem REG 179,2 13760 7546 /usr/lib/arm-linux-gnueabihf/libnss_mdns4_minimal.so.2
zeronsd 15574 root mem REG 179,2 42628 8495 /usr/lib/arm-linux-gnueabihf/libnss_files-2.31.so
zeronsd 15574 root mem REG 179,2 1319784 8471 /usr/lib/arm-linux-gnueabihf/libc-2.31.so
zeronsd 15574 root mem REG 179,2 13864 8477 /usr/lib/arm-linux-gnueabihf/libdl-2.31.so
zeronsd 15574 root mem REG 179,2 386572 8480 /usr/lib/arm-linux-gnueabihf/libm-2.31.so
zeronsd 15574 root mem REG 179,2 137364 8512 /usr/lib/arm-linux-gnueabihf/libpthread-2.31.so
zeronsd 15574 root mem REG 179,2 116324 7299 /usr/lib/arm-linux-gnueabihf/libgcc_s.so.1
zeronsd 15574 root mem REG 179,2 2150824 11580 /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.1
zeronsd 15574 root mem REG 179,2 459032 11584 /usr/lib/arm-linux-gnueabihf/libssl.so.1.1
zeronsd 15574 root mem REG 179,2 9512 7093 /usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so
zeronsd 15574 root mem REG 179,2 146888 7656 /usr/lib/arm-linux-gnueabihf/ld-2.31.so
zeronsd 15574 root 0r CHR 1,3 0t0 4 /dev/null
zeronsd 15574 root 1u unix 0x61a3094a 0t0 283766 type=STREAM
zeronsd 15574 root 2u unix 0x61a3094a 0t0 283766 type=STREAM
zeronsd 15574 root 3u a_inode 0,13 0 6967 [eventpoll]
zeronsd 15574 root 4u a_inode 0,13 0 6967 [eventfd]
zeronsd 15574 root 5u a_inode 0,13 0 6967 [eventpoll]
zeronsd 15574 root 6u unix 0xb7630605 0t0 283774 type=STREAM
zeronsd 15574 root 7u unix 0x0adc4997 0t0 283775 type=STREAM
zeronsd 15574 root 8u unix 0xb7630605 0t0 283774 type=STREAM
zeronsd 15574 root 9u IPv4 283794 0t0 TCP pi.hole:41042->151.101.1.91:https (ESTABLISHED)
zeronsd 15574 root 10u IPv4 283801 0t0 TCP 192.168.193.1:domain (LISTEN)
zeronsd 15574 root 11u IPv4 283802 0t0 UDP 192.168.193.1:domain
jordana@pi1b:~ $
There is slightly more logs in journal for the one that listens and here is the diff:
May 08 11:24:02 pi1b sudo[15582]: jordana : TTY=pts/0 ; PWD=/home/jordana ; USER=root ; COMMAND=/usr/bin/lsof -p 15574
May 08 11:24:02 pi1b sudo[15582]: pam_unix(sudo:session): session opened for user root(uid=0) by jordana(uid=1000)
May 08 11:24:03 pi1b sudo[15582]: pam_unix(sudo:session): session closed for user root
Do you have the journalctl for the zeronsd logs?
------- Original Message ------- On Monday, May 8th, 2023 at 3:48 AM, jordanarpad @.***> wrote:
I have a Raspberry Pi 1 and Pi Zero for 2 different zerotier networks running exactly the same zeronsd binary (built over 12 hours with 512M swap) and listens port 53 on one but not on the other. Both are running pi-hole besides zeronsd. Here is the requested output for one that does not listen:
@.:~ $ sudo lsof -p pidof zeronsd lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs Output information may be incomplete. COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME zeronsd 2343 root cwd DIR 179,2 4096 2 / zeronsd 2343 root rtd DIR 179,2 4096 2 / zeronsd 2343 root txt REG 179,2 15558984 391959 /home/colos/zeronsd zeronsd 2343 root mem REG 179,2 17972 137627 /usr/lib/arm-linux-gnueabihf/libnss_dns-2.31.so zeronsd 2343 root mem REG 179,2 75568 137744 /usr/lib/arm-linux-gnueabihf/libresolv-2.31.so zeronsd 2343 root mem REG 179,2 13760 137632 /usr/lib/arm-linux-gnueabihf/libnss_mdns4_minimal.so.2 zeronsd 2343 root mem REG 179,2 42628 137628 /usr/lib/arm-linux-gnueabihf/libnss_files-2.31.so zeronsd 2343 root mem REG 179,2 1319784 137204 /usr/lib/arm-linux-gnueabihf/libc-2.31.so zeronsd 2343 root mem REG 179,2 13864 137274 /usr/lib/arm-linux-gnueabihf/libdl-2.31.so zeronsd 2343 root mem REG 179,2 386572 137539 /usr/lib/arm-linux-gnueabihf/libm-2.31.so zeronsd 2343 root mem REG 179,2 137364 137721 /usr/lib/arm-linux-gnueabihf/libpthread-2.31.so zeronsd 2343 root mem REG 179,2 116324 137362 /usr/lib/arm-linux-gnueabihf/libgcc_s.so.1 zeronsd 2343 root mem REG 179,2 2150824 137243 /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.1 zeronsd 2343 root mem REG 179,2 459032 137813 /usr/lib/arm-linux-gnueabihf/libssl.so.1.1 zeronsd 2343 root mem REG 179,2 9512 137135 /usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so zeronsd 2343 root mem REG 179,2 146888 136992 /usr/lib/arm-linux-gnueabihf/ld-2.31.so zeronsd 2343 root 0r CHR 1,3 0t0 4 /dev/null zeronsd 2343 root 1u unix 0xcf31dc27 0t0 43155 type=STREAM zeronsd 2343 root 2u unix 0xcf31dc27 0t0 43155 type=STREAM zeronsd 2343 root 3u a_inode 0,13 0 6967 [eventpoll] zeronsd 2343 root 4u a_inode 0,13 0 6967 [eventfd] zeronsd 2343 root 5u a_inode 0,13 0 6967 [eventpoll] zeronsd 2343 root 6u unix 0x2cdb490e 0t0 43171 type=STREAM zeronsd 2343 root 7u unix 0x2d73d79d 0t0 43172 type=STREAM zeronsd 2343 root 8u unix 0x2cdb490e 0t0 43171 type=STREAM zeronsd 2343 root 9u IPv4 43193 0t0 TCP pi.hole:59370->151.101.1.91:https (ESTABLISHED) @.:~ $
And here is the requested output for the one that does listen:
@.:~ $ sudo lsof -p
pidof zeronsd
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs Output information may be incomplete. COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME zeronsd 15574 root cwd DIR 179,2 4096 2 / zeronsd 15574 root rtd DIR 179,2 4096 2 / zeronsd 15574 root txt REG 179,2 15558984 3184 /home/jordana/zeronsd zeronsd 15574 root mem REG 179,2 17972 8494 /usr/lib/arm-linux-gnueabihf/libnss_dns-2.31.so zeronsd 15574 root mem REG 179,2 75568 8519 /usr/lib/arm-linux-gnueabihf/libresolv-2.31.so zeronsd 15574 root mem REG 179,2 13760 7546 /usr/lib/arm-linux-gnueabihf/libnss_mdns4_minimal.so.2 zeronsd 15574 root mem REG 179,2 42628 8495 /usr/lib/arm-linux-gnueabihf/libnss_files-2.31.so zeronsd 15574 root mem REG 179,2 1319784 8471 /usr/lib/arm-linux-gnueabihf/libc-2.31.so zeronsd 15574 root mem REG 179,2 13864 8477 /usr/lib/arm-linux-gnueabihf/libdl-2.31.so zeronsd 15574 root mem REG 179,2 386572 8480 /usr/lib/arm-linux-gnueabihf/libm-2.31.so zeronsd 15574 root mem REG 179,2 137364 8512 /usr/lib/arm-linux-gnueabihf/libpthread-2.31.so zeronsd 15574 root mem REG 179,2 116324 7299 /usr/lib/arm-linux-gnueabihf/libgcc_s.so.1 zeronsd 15574 root mem REG 179,2 2150824 11580 /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.1 zeronsd 15574 root mem REG 179,2 459032 11584 /usr/lib/arm-linux-gnueabihf/libssl.so.1.1 zeronsd 15574 root mem REG 179,2 9512 7093 /usr/lib/arm-linux-gnueabihf/libarmmem-v6l.so zeronsd 15574 root mem REG 179,2 146888 7656 /usr/lib/arm-linux-gnueabihf/ld-2.31.so zeronsd 15574 root 0r CHR 1,3 0t0 4 /dev/null zeronsd 15574 root 1u unix 0x61a3094a 0t0 283766 type=STREAM zeronsd 15574 root 2u unix 0x61a3094a 0t0 283766 type=STREAM zeronsd 15574 root 3u a_inode 0,13 0 6967 [eventpoll] zeronsd 15574 root 4u a_inode 0,13 0 6967 [eventfd] zeronsd 15574 root 5u a_inode 0,13 0 6967 [eventpoll] zeronsd 15574 root 6u unix 0xb7630605 0t0 283774 type=STREAM zeronsd 15574 root 7u unix 0x0adc4997 0t0 283775 type=STREAM zeronsd 15574 root 8u unix 0xb7630605 0t0 283774 type=STREAM zeronsd 15574 root 9u IPv4 283794 0t0 TCP pi.hole:41042->151.101.1.91:https (ESTABLISHED) zeronsd 15574 root 10u IPv4 283801 0t0 TCP 192.168.193.1:domain (LISTEN) zeronsd 15574 root 11u IPv4 283802 0t0 UDP 192.168.193.1:domain @.:~ $There is slightly more logs in journal for the one that listens and here is the diff:
May 08 11:24:02 pi1b sudo[15582]: jordana : TTY=pts/0 ; PWD=/home/jordana ; USER=root ; COMMAND=/usr/bin/lsof -p 15574 May 08 11:24:02 pi1b sudo[15582]: pam_unix(sudo:session): session opened for user root(uid=0) by jordana(uid=1000) May 08 11:24:03 pi1b sudo[15582]: pam_unix(sudo:session): session closed for user root
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>
Not exactly sure what you mean but I realised the diff is more, again in favour of the one that listens as the following lines are also there and they are not there for the one that does not listen. This other diff snippet with a few lines before and after:
May 09 00:14:37 pi1b zeronsd[20722]: May 09 00:14:37.333 INFO zeronsd::init: Welcome to ZeroNS!
May 09 00:14:38 pi1b zeronsd[20722]: May 09 00:14:38.694 INFO zeronsd::init: Your IP for this network: 192.168.x.x
May 09 00:14:38 pi1b zeronsd[20722]: May 09 00:14:38.709 INFO trust_dns_server::store::forwarder::authority: loading forwarder config: .
May 09 00:14:38 pi1b zeronsd[20722]: May 09 00:14:38.716 INFO trust_dns_server::store::forwarder::authority: forward resolver configured: .:
May 09 00:14:39 pi1b zeronsd[20722]: May 09 00:14:39.003 INFO zeronsd::authority: Adding new record zt-xxxxxx.zeronsd.: (192.168.x.x)
It can be seen from this snippet that the initial post of this issue also misses these lines that refer to the forwarder. A bit difficult to share more publicly from journalctl without exposing my network.
what I'm specifically looking for are any errors about bind() calls or listen() calls. If there's something like that in the log, please include it and mask your IP.
------- Original Message ------- On Tuesday, May 9th, 2023 at 1:34 AM, jordanarpad @.***> wrote:
Not exactly sure what you mean but I realised the diff is more, again in favour of the one that listens as the following lines are also there and they are not there for the one that does not listen. This other diff snippet with a few lines before and after:
May 09 00:14:37 pi1b zeronsd[20722]: May 09 00:14:37.333 INFO zeronsd::init: Welcome to ZeroNS! May 09 00:14:38 pi1b zeronsd[20722]: May 09 00:14:38.694 INFO zeronsd::init: Your IP for this network: 192.168.x.x May 09 00:14:38 pi1b zeronsd[20722]: May 09 00:14:38.709 INFO trust_dns_server::store::forwarder::authority: loading forwarder config: . May 09 00:14:38 pi1b zeronsd[20722]: May 09 00:14:38.716 INFO trust_dns_server::store::forwarder::authority: forward resolver configured: .: May 09 00:14:39 pi1b zeronsd[20722]: May 09 00:14:39.003 INFO zeronsd::authority: Adding new record zt-xxxxxx.zeronsd.: (192.168.x.x)
It can be seen from this snippet that the initial post of this issue also misses these lines that refer to the forwarder. A bit difficult to share more publicly from journalctl without exposing my network.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>
there are no zeronsd logs with bind or listen in neither journal unfortunately even after setting ZERONSD_LOG and RUST_LOG to debug. do I need to set there environmental variables during the build process for a debug binary?
Not that I can recall. I'm not sure what is happening, unfortunately, and I don't have a testbed that can accurately replicate this.
If you could, for the benefit of others, clearly detail the OS you're using and hardware platform it runs on, that may help down the road.
I'm very sorry there is not a better solution here.
------- Original Message ------- On Tuesday, May 9th, 2023 at 4:09 AM, jordanarpad @.***> wrote:
there are no zeronsd logs with bind or listen in neither journal unfortunately even after setting ZERONSD_LOG and RUST_LOG to debug. do I need to set there environmental variables during the build process for a debug binary?
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>
I figured out the difference. On the one where zeronsd listens I had configured pi-hole to bind only on eth0. On the other one zeronsd could not bind because pi-hole bind was not limited to an interface so bind them all.
just for the record I used these commands to build zeronsd on Raspberry Pi 1 running latest Raspberry Pi OS:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
sudo apt install librust-openssl-dev
cargo install --git https://github.com/zerotier/zeronsd --branch main --jobs 1
and I had to increase swap memory to 512M for which I used this article as guidance: increase swap memory the build process took a painful 12 hours but I can happily use the binary accross all my Raspberry Pi Zero W and Raspberry Pi 1
This makes sense, so zeronsd could not bind to the port because it was already registered as bound.
Travis, can you make a note of this? There's probably a check in the bind() calls in the startup routines that isn't being handled as a failure.
------- Original Message ------- On Tuesday, May 9th, 2023 at 3:53 PM, jordanarpad @.***> wrote:
I figured out the difference. On the one where zeronsd listens I had configured pi-hole to bind only on eth0. On the other one where zeronsd could not bind because pi-hole bind was not limited to any interface.
just for the record I used these commands to build zeronsd on Raspberry Pi 1 running latest Raspberry Pi OS:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh sudo apt install librust-openssl-dev cargo install --git https://github.com/zerotier/zeronsd --branch main --jobs 1
and I had to increase swap memory to 512M for which I used this article as guidance: increase swap memory the build process took a painful 12 hours but I can happily use the binary accross all my Raspberry Pi Zero W and Raspberry Pi 1
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>
Thanks for diagnosing, btw!
------- Original Message ------- On Tuesday, May 9th, 2023 at 3:53 PM, jordanarpad @.***> wrote:
I figured out the difference. On the one where zeronsd listens I had configured pi-hole to bind only on eth0. On the other one where zeronsd could not bind because pi-hole bind was not limited to any interface.
just for the record I used these commands to build zeronsd on Raspberry Pi 1 running latest Raspberry Pi OS:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh sudo apt install librust-openssl-dev cargo install --git https://github.com/zerotier/zeronsd --branch main --jobs 1
and I had to increase swap memory to 512M for which I used this article as guidance: increase swap memory the build process took a painful 12 hours but I can happily use the binary accross all my Raspberry Pi Zero W and Raspberry Pi 1
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>
ah sorry guys. I think #218 would have helped (but I haven't double checked). I haven't been able to focus on this project recently.
side question: you can't cross compile from a big computer?
yeah, probably would have. I forgot about it completely.
If you need help cutting releases, shoot me an email and we can do a video chat or something. It's an effort. :)
-Erik
------- Original Message ------- On Tuesday, May 9th, 2023 at 8:07 PM, Travis LaDuke @.***> wrote:
ah sorry guys. I think #218 would have helped (but I haven't double checked). I haven't been able to focus on this project recently.
side question: you can't cross compile from a big computer?
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>
I am conscious that I hijacked this issue with something ended up unrelated so this will be my last comment under this issue. @laduke great to see possible fixes are on the way - reading it definitely sounds related even with the original issue in this post. I will test if it fixes my issue once merged. @erikh I am very grateful for your swift and continuous support in this issue and your questions did guide me through to find my solution. Will gladly take your offer and will drop you an email about cross-compilation.
Hi!
I can see typical startup output:
The records that are added looked correct to me.
But then I tried to resolve using the IP (172.25.221.93) and nothing happened. Looking at the list of open ports (sudo lsof -i -P -n | grep LISTEN|grep 53) shows nothing listening on port 53.
I then tried to launch with debug and trace log levels, but did not see any other information that would point me to the issue. Do you have any other ideas on how to debug this?
OS: MacOS 12.4 installed: via brew (github repo instructions)