Closed Paraphraser closed 2 years ago
zyclonite
commented
2 years ago Hi Phill and thank you for your contribution!
Does the routing mode break the bridge functionality? Or are there any reasons why you replace it? Would it make sense to support iptables as well as nftables or switch natively to nftables?
Paraphraser
commented
2 years ago Does the routing mode break the bridge functionality? Or are there any reasons why you replace it?
I think there must be a fundamental misunderstanding somewhere.
I'd like to try walking through my reasoning and see if you wind up agreeing with me, or are able to spot my mistakes. My face will be really red if I turn out to be wrong (but I won't really mind because I will have learned a lot).
I'll start with the existing entrypoint-bridge.sh script and, in particular, these three lines:
iptables -t nat -A POSTROUTING -o $PHY_IFACE -j MASQUERADE
iptables -A FORWARD -i $PHY_IFACE -o $ZT_IFACE -m state --state RELATED,ESTABLISHED -j ACCEPT
iptables -A FORWARD -i $ZT_IFACE -o $PHY_IFACE -j ACCEPTSomething to note in passing is the key word "POSTROUTING" in the first line. I think of that as a bit of a scene-setter.
Next, let's look at the existing README file. Under the "Bridge mode" heading, the "paper" link points to:
If you read through that paper from top to bottom, you won't find the three lines from the existing entrypoint-bridge.sh script anywhere in that paper.
In the ZeroTier knowledge base, the document immediately above that one is:
If you read through that second document, you'll find this:
Those lines from the Knowledge-base paper mirror what is implemented in the existing entrypoint-bridge.sh script.
What I think has happened is that the "router" paper from the ZeroTier knowledge base was implemented but it somehow wound up being called the "bridge", and the README compounded that by pointing to the wrong paper.
Does that sound even halfway plausible?
It's true that this discrepancy could be explained by the ZeroTier knowledge base having things back-to-front. But I don't think that's the case because the actual behaviour of what is currently on DockerHub as zyclonite/zerotier:bridge is that of a router, not a bridge.
It gets back to definitions. A bridge operates at Layer 2. It segments a broadcast domain into separate collision domains. Frames don't leave the broadcast domain because the source and destination IP addresses are in the same subnet.
A router operates at Layer 3. It forwards packets between broadcast domains. The source and destination IP addresses are in different subnets.
Aside from observing whether the source and destination IP addresses of a datagram are in the same or different subnets, you can use a tool like traceroute to see what's going on.
Here's some output from my laptop:
$ traceroute 192.168.132.60
traceroute to 192.168.132.60 (192.168.132.60), 64 hops max, 52 byte packets
1 10.244.0.1 (10.244.0.1) 2.434 ms 6.373 ms 1.684 ms
2 192.168.132.60 (192.168.132.60) 2.260 ms 14.485 ms 2.014 msMy laptop is connected to the Internet via a cellular link. The only possible way my laptop can reach 192.168.132.60 is via ZeroTier.
To lay it out step by step:
10.244.131.22 is the ZeroTier-assigned IP address of the ZeroTier client running on my laptop.
The IP stack in my laptop knows where to forward the packets because of a managed route set up in ZeroTier Central:
10.244.0.1 is the ZeroTier-assigned IP address of the ZeroTier daemon, running in a Docker container on my Raspberry Pi.
The Raspberry Pi's IP address on my home LAN is 192.168.132.102.
192.168.132.60 is another host on my home LAN.
From the perspective of the traceroute packets, only two subnets are involved:
I think of the need for the packets to traverse broadcast domains (subnets) as proof that Layer 3 routing is involved, because Layer 2 bridges simply can't do that.
If you agree with this analysis, then I also hope you'll see that renaming the "bridge" to "router" achieves two things:
I really hope that all makes sense. I try to be as careful as I can but sometimes my Aussie turn-of-phrase can be misinterpreted.
Would it make sense to support iptables as well as nftables or switch natively to nftables?
This is well beyond my knowledge. It took me a while to figure out that iptables didn't work on my Raspberry Pi, and even longer to figure out that simply changing the command to iptables-nft solved all the problems.
The only device I have to test this on is a Raspberry Pi running full 64-bit Bullseye. I have no idea whether iptables would work if I reverted to a Buster system, or if the dependency on iptables-nft is peculiar to all flavours of Raspberry Pi OS.
To put this another way, I didn't just want to propose "hey, let's all switch to iptables-nft" because of the risk that that might break something on other platforms.
With that as background, and to answer your question directly, I think I read somewhere that nftables (as distinct from iptables-nft) uses different syntax. If that's true, it couldn't be a direct substitution like I've proposed in the PR. Something like a case statement would be needed. But, in principle, I can't think of any reason why multiple variations on the theme could not be supported - so long as the "add" and "remove" functionality is implemented for each table type.
I'm definitely up for testing anything anyone wants to propose.
Paraphraser
commented
2 years ago Overnight it occurred to me that I could also answer your first question in a slightly different way.
Let's start with this existing command as our baseline:
$ docker run --name zerotier-one --device=/dev/net/tun \
--cap-add=NET_ADMIN --cap-add=NET_RAW --cap-add=SYS_ADMIN \
-v /var/lib/zerotier-one:/var/lib/zerotier-one zyclonite/zerotier:bridgeAs I explained in my earlier reply, despite the word "bridge" being in the tag, the container actually implements a "router".
Let's suppose that you accept the PR and push an image to DockerHub with the "router" tag:
Unless you remove the "bridge" image at the same time (which I'm not suggesting), anyone who continues to use the "bridge" tag will see no change in behaviour.
Anyone who does nothing more than replace "bridge" with "router" will also see no change in behaviour.
Next, let's suppose that ZeroTier updates from 1.10.1 to 1.10.2. You rebuild and push out both zyclonite/zerotier and zyclonite/zerotier:router to DockerHub.
At that point, anyone using the "bridge" tag will still be getting 1.10.1. If they want to get 1.10.2 they'll have to switch to the "router" tag. So that's really the first point where users would have to change their commands, scripts or compose files.
All the other changes in the PR are governed by environment variables. The defaults when the variables are omitted are "no change" so they're backwards compatible. Users will need to opt-in to each variable, case by case, to suit their individual needs.
bfg100k
commented
2 years ago Hi Phil, nice work. I agree with you that calling the existing implementation a bridge is technically incorrect. A year ago I forked this repo to add "gateway" functionality which essentially is your router mode but with added flexibility to define direction of traffic access between the ZT and local networks. i.e. for scenarios where you may only want devices in local network to access devices in ZT network (my use case for backing up to remote servers but not allowing ingress for security) or vice versa or both. https://github.com/bfg100k/zerotier-gateway
Given the similarity in features with your proposed enhancements, I'm wondering if you are keen to include the above functionality to avoid duplicating efforts? The only change I believe is in the entrypoint script where different set of iptables rules are configured (and removed in the cleanup function) according to the environment variable GATEWAY_MODE. ( https://github.com/bfg100k/zerotier-gateway/blob/master/main.sh)
On Mon, 25 Jul 2022 at 15:21, Phill @.***> wrote:
Enhancements to router (formerly bridge) functionality
1.
Rename "bridge" to "router"
Rename files with "bridge" in the name to "router". - Edit file contents referring to "bridge" so they refer to "router". - point to the correct link in the ZeroTier documentation (existing link points to "bridge" but the actual implementation with iptables rules is that of the "router").The problem I'm foreseeing is what happens if there's a need to implement an actual bridge variant? 2.
timezone support
Adding tzdata to Dockerfile.router activates TZ environment variable. Avoids need for mental gymnastics when examining logs. 3.
support environment variables:
PUID + PGID. Default to 999 and 994, respectively, mimicking what happens on a "native" install of ZeroTier-One on a Raspberry Pi. Only used to perform unconditional reset of ownership (PUID:PGID) throughout persistent store on each launch. Self-repair mitigates permission problems that can sometimes occur in docker environments, which can send containers into restart loops.
ZEROTIER_ONE_LOCAL_PHYS. Allows for overriding to wlan0 (eg Raspberry Pi Zero 2W), or both eth0 wlan0 to support multiple subnets or failover modes, or similar situations. Defaults to eth0 to maintain backwards compatibility.
ZEROTIER_ONE_USE_IPTABLES_NFT. If true, substitutes iptables-nft. This definitely seems to be necessary on Raspberry Pi running Bullseye. Default is false which uses iptables and maintains backwards compatibility.
ZEROTIER_ONE_NETWORK_IDS. Supports passing one or more default network IDs. Intended as an alternative to waiting for a container to initialise and then issuing one or more join commands. Also means container can fail safe if its persistent storage is erased (will look like a new identity but can be authorised in ZeroTier Central and will then be reachable for additional configuration). 4.
improve router entry-point script:
basic problem is iptables rules do not get cleaned-up when container quits. This can lead to multiple duplicate rules.
launches ZeroTier daemon as a detached (nohup) process, with an asynchronous tail as an intermediary to bring any stdout/stderr messaging into PID=1.
traps INT, TERM and HUP to run termination handler to clean up iptables rules (TERM is the common case).
suspends on ZeroTier daemon then runs termination handler on normal exit.
tested correct handling of:
docker commands to restart, terminate, recreate, stack down, etc. - reboots while the container is running (assuming "unless stopped" in force). - external action (eg user killall) clobbering the ZeroTier daemon. 5.add example docker-compose service definition for router variant. 6.
refactor documentation to separate zerotier-router into its own readme.
You can view, comment on, or merge this pull request online at:
https://github.com/zyclonite/zerotier-docker/pull/12 Commit Summary
- ddb2636 https://github.com/zyclonite/zerotier-docker/pull/12/commits/ddb263614170b87dd811e5022591963902607563 rename "bridge" to "router"
- bf15adc https://github.com/zyclonite/zerotier-docker/pull/12/commits/bf15adc4cedc74833432331dcd46ee6302ac582e timezone support
- f28b665 https://github.com/zyclonite/zerotier-docker/pull/12/commits/f28b665afda6c68693af88dc8f556f34d07ebe70 Extend router launch script:
- 226cf83 https://github.com/zyclonite/zerotier-docker/pull/12/commits/226cf83004cf26b6a8eac77e07f0427e0d50433e add example docker-compose service definition for router
- 7f6239f https://github.com/zyclonite/zerotier-docker/pull/12/commits/7f6239fae47232dcd474f87e48d72d5daed46bed update documentation to describe router-mode environment variables
- 9b98b47 https://github.com/zyclonite/zerotier-docker/pull/12/commits/9b98b470f8967f2728c477c0d33b9ce881527329 Change
ZEROTIER_ONE_NETWORK_IDtoZEROTIER_ONE_NETWORK_IDSso more- 41d4501 https://github.com/zyclonite/zerotier-docker/pull/12/commits/41d4501593d1e7dbeb620444466729b78af2187c Improve
entrypoint-router.sh:- 1ae3680 https://github.com/zyclonite/zerotier-docker/pull/12/commits/1ae368044b4a39a968b3188db604669de28a4f61 refactor documentation to separate zerotier-router into separate readme
File Changes
(9 files https://github.com/zyclonite/zerotier-docker/pull/12/files)
- M .github/workflows/multiarch.yml https://github.com/zyclonite/zerotier-docker/pull/12/files#diff-0d479f1d504a929896510d94024407a1796ab3d2d8ee570c5a5f3b85030248b0 (4)
- R .github/workflows/router.yml https://github.com/zyclonite/zerotier-docker/pull/12/files#diff-9c16e717461d5114894ab9faf8fb262b957ea3aafc7f09e074e5b206183d6c20 (24)
- R Dockerfile.router https://github.com/zyclonite/zerotier-docker/pull/12/files#diff-a704fd570e2ed9cedc382d7dc01deb312cbe7e8c4cb751ccf44faed16975bfbf (10)
- A README-router.md https://github.com/zyclonite/zerotier-docker/pull/12/files#diff-338d4bf8ccae84ccd3a3e1bd9a2c0269fdd592257583513834e98df2a7b0b8fb (172)
- M README.md https://github.com/zyclonite/zerotier-docker/pull/12/files#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5 (9)
- A docker-compose-router.yml https://github.com/zyclonite/zerotier-docker/pull/12/files#diff-8f67717cf94a658e45fbf5843136db74a56449f8924d6eb78f4030a2a158041b (22)
- M k8s/deployment.yaml https://github.com/zyclonite/zerotier-docker/pull/12/files#diff-4bbade0e44f036b32fc63c2e5838f57268ec49ff70dfce1afb9e12849515dd38 (2)
- D scripts/entrypoint-bridge.sh https://github.com/zyclonite/zerotier-docker/pull/12/files#diff-8db9ef661c6b580d4e235e64b7c9ad8c97323084737ad110b7f9cd90259e7c77 (14)
- A scripts/entrypoint-router.sh https://github.com/zyclonite/zerotier-docker/pull/12/files#diff-05d09c7d2f7b7937188a82f5928a93d68d7dece26ed898070e491514cadea19d (114)
Patch Links:
- https://github.com/zyclonite/zerotier-docker/pull/12.patch
- https://github.com/zyclonite/zerotier-docker/pull/12.diff
— Reply to this email directly, view it on GitHub https://github.com/zyclonite/zerotier-docker/pull/12, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQWGN5BMEMCV5LCBIARDVDVVYP7JANCNFSM54Q6OTHQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>
Paraphraser
commented
2 years ago @bfg100k - I'm definitely not opposed to doing this but some (or even a lot) of it might be beyond my ken, as it were.
For example, I take the following to mean "add this rule if it is not already there":
iptables -t nat -C POSTROUTING -o $LO_DEV -j MASQUERADE 2>/dev/null || {
iptables -t nat -A POSTROUTING -o $LO_DEV -j MASQUERADE
}Knowing what (little) I know right now, I would assume that the only way a rule could be there already would be if an earlier run of the container had not removed its rules. Part of my focus in this PR was on ensuring the container always cleaned-up after itself, so my "first blush" reaction is that the check is at best unnecessary, at worst potentially masking a deeper problem such as two processes both adding rules - which would imply that subsequent removal of a rule by one process could wrong-foot the other.
But you've got the practical experience so, rather than me make untested and unwarranted assumptions, I'd rather be guided by you. If there's a good reason for check-don't-duplicate syntax then maybe I should adopt that too.
Also, I notice that you added bash to your container. Do you happen to recall which bash features you needed? I'm more of a "try really hard to work with what I have" kind of person. I like to avoid adding stuff to containers unless there's no reasonable alternative.
bfg100k
commented
2 years ago Aye, the check may be unnecessary given this is running in a container that only serve one purpose and we can assume we have absolute control. I must have copied the code from some other projects I've worked on.
In regards to bash, I guess I'm the opposite in the sense that "why use a manual screwdriver when I can use an electric one (for a small cost)". I won't be able to tell you which bash features are required here given I always start with bash.
BTW, I run this on a synology NAS so happy to help you test it.
On Tue, 26 Jul 2022 at 15:44, Phill @.***> wrote:
@bfg100k - I'm definitely not opposed to doing this but some (or even a lot) of it might be beyond my ken, as it were.
For example, I take the following to mean "add this rule if it is not already there":
iptables -t nat -C POSTROUTING -o $LO_DEV -j MASQUERADE 2>/dev/null || { iptables -t nat -A POSTROUTING -o $LO_DEV -j MASQUERADE }
Knowing what (little) I know right now, I would assume that the only way a rule could be there already would be if an earlier run of the container had not removed its rules. Part of my focus in this PR was on ensuring the container always cleaned-up after itself, so my "first blush" reaction is that the check is at best unnecessary, at worst potentially masking a deeper problem such as two processes both adding rules - which would imply that subsequent removal of a rule by one process could wrong-foot the other.
But you've got the practical experience so, rather than me make untested and unwarranted assumptions, I'd rather be guided by you. If there's a good reason for check-don't-duplicate syntax then maybe I should adopt that too.
Also, I notice that you added bash to your container. Do you happen to recall which bash features you needed? I'm more of a "try really hard to work with what I have" kind of person. I like to avoid adding stuff to containers unless there's no reasonable alternative.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>
zyclonite
commented
2 years ago @Paraphraser makes all total sense to me, i personally do not use the bridge/router tag because i do the routing on the host directly
i just wanted to make sure we do not break things for the users of the previous contribution
another thought
would it make sense to somehow test if iptables or nftables are supported by the host and decide in the entrypoint script if iptables or iptables-nft is required to be used? (so no env var is needed)
Paraphraser
commented
2 years ago I thought about that but I couldn't see how to do it. It's not that iptables doesn't exist on Raspberry Pi Bullseye - it does. It's not that it returns an error when you try the "nat" command - the command appears to succeed and doesn't return an error. It just doesn't have any effect. It's weird. Any ideas?
Paraphraser
commented
2 years ago @bfg100k the only thing I've spotted so far is the force variable to lower case in the case statement. I'm in compare/contrast/consider mode at the moment.
I've also been wondering about the test for whether ip routing is enabled. I've yet to encounter a Pi that doesn't have it enabled by default. What's your experience with other systems? If that's a common problem the test would be a good idea. However, I wouldn't abort on that condition. I'd just fall back to letting the daemon run "as is" with no iptables rules and bung out a message about it. I work in the compose space for everything else and containers that go into restart loops rather than failing sensibly drive me nuts.
Paraphraser
commented
2 years ago @bfg100k I have the changes coded and I will test it tomorrow (I'm at UTC+10).
But, put a pin in that for a moment.
Right now, my RPi is running as per the current state of the pull request. That corresponds with "inbound" for your extensions, right? That suits my needs at the moment. I just want a remote client to be able to access anything on my home LAN going via ZT.
So far, so good.
If you head over to one of the comments on issue 10, you'll see @red-avtovo seems to be saying that bi-directional routing can be had in a two-site setup just by adding static routes to the respective site routers.
In considering your rules for "inbound" vs "outbound" and "both", it seems to me that you're implicitly saying that static routes in local routers would not get the job done all by themselves. The additional rules would be needed.
The last two paragraphs would seem to represent a contradiction.
Any ideas?
Paraphraser
commented
2 years ago Further on the topic of why iptables vs iptables-nft.
This might turn out to be an Alpine problem.
Run iptables inside the container:
$ docker exec zerotier iptables -S POSTROUTING -t nat
-P POSTROUTING ACCEPTRun iptables-nft inside the container:
$ docker exec zerotier iptables-nft -S POSTROUTING -t nat
-P POSTROUTING ACCEPT
# Warning: iptables-legacy tables present, use iptables-legacy to see them
-A POSTROUTING -s 172.28.0.0/16 ! -o br-b7d898c052fd -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.28.0.2/32 -d 172.28.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.28.0.2/32 -d 172.28.0.2/32 -p udp -m udp --dport 67 -j MASQUERADE
-A POSTROUTING -s 172.28.0.2/32 -d 172.28.0.2/32 -p tcp -m tcp --dport 53 -j MASQUERADE
-A POSTROUTING -s 172.28.0.3/32 -d 172.28.0.3/32 -p tcp -m tcp --dport 1880 -j MASQUERADE
-A POSTROUTING -s 172.28.0.4/32 -d 172.28.0.4/32 -p tcp -m tcp --dport 9000 -j MASQUERADE
-A POSTROUTING -s 172.28.0.5/32 -d 172.28.0.5/32 -p tcp -m tcp --dport 3000 -j MASQUERADE
-A POSTROUTING -s 172.28.0.2/32 -d 172.28.0.2/32 -p udp -m udp --dport 53 -j MASQUERADE
-A POSTROUTING -s 172.28.0.6/32 -d 172.28.0.6/32 -p tcp -m tcp --dport 8086 -j MASQUERADE
-A POSTROUTING -s 172.28.0.7/32 -d 172.28.0.7/32 -p tcp -m tcp --dport 1883 -j MASQUERADE
-A POSTROUTING -s 172.28.0.4/32 -d 172.28.0.4/32 -p tcp -m tcp --dport 8000 -j MASQUERADE
-A POSTROUTING -s 172.28.0.8/32 -d 172.28.0.8/32 -p udp -m udp --dport 51820 -j MASQUERADE
-A POSTROUTING -o eth0 -j MASQUERADE
-A POSTROUTING -o wlan0 -j MASQUERADE
-A POSTROUTING -s 172.28.0.9/32 -d 172.28.0.9/32 -p tcp -m tcp --dport 8080 -j MASQUERADERun iptables outside the container:
$ sudo iptables -S POSTROUTING -t nat
# Warning: iptables-legacy tables present, use iptables-legacy to see them
-P POSTROUTING ACCEPT
-A POSTROUTING -s 172.28.0.0/16 ! -o br-b7d898c052fd -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.28.0.2/32 -d 172.28.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.28.0.2/32 -d 172.28.0.2/32 -p udp -m udp --dport 67 -j MASQUERADE
-A POSTROUTING -s 172.28.0.2/32 -d 172.28.0.2/32 -p tcp -m tcp --dport 53 -j MASQUERADE
-A POSTROUTING -s 172.28.0.3/32 -d 172.28.0.3/32 -p tcp -m tcp --dport 1880 -j MASQUERADE
-A POSTROUTING -s 172.28.0.4/32 -d 172.28.0.4/32 -p tcp -m tcp --dport 9000 -j MASQUERADE
-A POSTROUTING -s 172.28.0.5/32 -d 172.28.0.5/32 -p tcp -m tcp --dport 3000 -j MASQUERADE
-A POSTROUTING -s 172.28.0.2/32 -d 172.28.0.2/32 -p udp -m udp --dport 53 -j MASQUERADE
-A POSTROUTING -s 172.28.0.6/32 -d 172.28.0.6/32 -p tcp -m tcp --dport 8086 -j MASQUERADE
-A POSTROUTING -s 172.28.0.7/32 -d 172.28.0.7/32 -p tcp -m tcp --dport 1883 -j MASQUERADE
-A POSTROUTING -s 172.28.0.4/32 -d 172.28.0.4/32 -p tcp -m tcp --dport 8000 -j MASQUERADE
-A POSTROUTING -s 172.28.0.8/32 -d 172.28.0.8/32 -p udp -m udp --dport 51820 -j MASQUERADE
-A POSTROUTING -o eth0 -j MASQUERADE
-A POSTROUTING -o wlan0 -j MASQUERADE
-A POSTROUTING -s 172.28.0.9/32 -d 172.28.0.9/32 -p tcp -m tcp --dport 8080 -j MASQUERADERun iptables-nft outside the container:
$ sudo iptables-nft -S POSTROUTING -t nat
# Warning: iptables-legacy tables present, use iptables-legacy to see them
-P POSTROUTING ACCEPT
-A POSTROUTING -s 172.28.0.0/16 ! -o br-b7d898c052fd -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.28.0.2/32 -d 172.28.0.2/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.28.0.2/32 -d 172.28.0.2/32 -p udp -m udp --dport 67 -j MASQUERADE
-A POSTROUTING -s 172.28.0.2/32 -d 172.28.0.2/32 -p tcp -m tcp --dport 53 -j MASQUERADE
-A POSTROUTING -s 172.28.0.3/32 -d 172.28.0.3/32 -p tcp -m tcp --dport 1880 -j MASQUERADE
-A POSTROUTING -s 172.28.0.4/32 -d 172.28.0.4/32 -p tcp -m tcp --dport 9000 -j MASQUERADE
-A POSTROUTING -s 172.28.0.5/32 -d 172.28.0.5/32 -p tcp -m tcp --dport 3000 -j MASQUERADE
-A POSTROUTING -s 172.28.0.2/32 -d 172.28.0.2/32 -p udp -m udp --dport 53 -j MASQUERADE
-A POSTROUTING -s 172.28.0.6/32 -d 172.28.0.6/32 -p tcp -m tcp --dport 8086 -j MASQUERADE
-A POSTROUTING -s 172.28.0.7/32 -d 172.28.0.7/32 -p tcp -m tcp --dport 1883 -j MASQUERADE
-A POSTROUTING -s 172.28.0.4/32 -d 172.28.0.4/32 -p tcp -m tcp --dport 8000 -j MASQUERADE
-A POSTROUTING -s 172.28.0.8/32 -d 172.28.0.8/32 -p udp -m udp --dport 51820 -j MASQUERADE
-A POSTROUTING -o eth0 -j MASQUERADE
-A POSTROUTING -o wlan0 -j MASQUERADE
-A POSTROUTING -s 172.28.0.9/32 -d 172.28.0.9/32 -p tcp -m tcp --dport 8080 -j MASQUERADEBasically, iptables inside the container is the odd one out. All the others agree.
I can't immediately see how to make use of this to let the script conclude "iptables isn't working, therefore use iptables-nft".
zyclonite
commented
2 years ago @Paraphraser an idea to check for iptales vs nftables could be lsmod | grep ^nf_tables
Paraphraser
commented
2 years ago Hmmm. This is what I see:
Other than things being in a different order inside-container vs out, they are the same.
I could get a lot further if I either had a test vehicle which worked with iptables, or someone whose system worked with iptables could run some compare/contrast tests for various ideas until we can settle on something that will work.
bfg100k
commented
2 years ago I believe he said that because he mistakenly assumed that you had not configured the static routes correctly. I don't think the 2 paragraphs contradict each other. Rather, they serve different purposes. Static routes allow devices to find their way whereas iptable rules determine which way traffic is allowed (or blocked) between your local and ZT networks (i.e. incoming / outgoing / or both). (Essentially it is a firewall) You can easily test this out by setting the static route in your local router and pinging a ZT device from LAN. You will find that it fails as your current iptable rules do not allow new connections to pass from the LAN to ZT interfaces.
So in summary,
(A) if you are configuring the network for ingress only (i.e. your use case where remote devices need to access LAN resources), then you need the "inbound" rules setup in the ZT container and the static route defined in the ZT admin console.
(B) if you are configuring the network for egress only (i.e. my use case where my local devices are backing up to remote servers), then you need the "outbound" rules setup in the ZT container and the static route defined in the local router
(C) if you are configuring the network for bi-directional access, then you need the "both" rules setup in the ZT container as well as the static route in BOTH the ZT admin console and the local router.
Hope that makes sense?
On Tue, 26 Jul 2022 at 23:12, Phill @.***> wrote:
@bfg100k https://github.com/bfg100k I have the changes coded and I will test it tomorrow (I'm at UTC+10).
But, put a pin in that for a moment.
Right now, my RPi is running as per the current state of the pull request. That corresponds with "inbound" for your extensions, right? That suits my needs at the moment. I just want a remote client to be able to access anything on my home LAN going via ZT.
So far, so good.
If you head over to one of the comments on issue 10 https://github.com/zyclonite/zerotier-docker/issues/10#issuecomment-1186896180, you'll see @red-avtovo https://github.com/red-avtovo seems to be saying that bi-directional routing can be had in a two-site setup just by adding static routes to the respective site routers.
In considering your rules for "inbound" vs "outbound" and "both", it seems to me that you're implicitly saying that static routes in local routers would not get the job done all by themselves. The additional rules would be needed.
The last two paragraphs would seem to represent a contradiction.
Any ideas?
— Reply to this email directly, view it on GitHub https://github.com/zyclonite/zerotier-docker/pull/12#issuecomment-1195466704, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQWGN45ZLFM75B5U47RWHDVV7P47ANCNFSM54Q6OTHQ . You are receiving this because you were mentioned.Message ID: @.***>
red-avtovo
commented
2 years ago It totally does! I was assuming, that static route configuration on ZT admin console is a task, which will be performed regardless of the situation. Even for egress case (B) you can't just send the traffic to void
So if we take the ZT admin console configuration out of the equation, then our process descriptions matches.
Paraphraser
commented
2 years ago Err - I was still testing the changes to support three gateway modes plus fail safe if IP routing wasn't enabled. I was probably an hour away from pushing the commit.
This is where my GitHub knowledge runs into a wall. What do I do now? Do I push the commits on the PR branch or do I need to start a new PR?
Also, what's this "Multiarch build, Attempt #2 / Publish images Failed in 19 seconds" message? Is that something I need to do something about and, if so, where do I begin?
Paraphraser
commented
2 years ago I'm going to trust the GitHub UI where it tells me that I can still push more commits. I hope I don't break anything.
I've completed testing the extra changes:
Supports three routing modes as proposed by @bfg100k
a ZEROTIER_ONE_GATEWAY_MODE variable with supported values of inbound (forward traffic from ZeroTier cloud to local interfaces), outbound (forward traffic from local interfaces to ZeroTier cloud) and both (bi-directional).
Defaults to inbound (which means no change in behaviour so it's fully backwards compatible).
Also checks for net.ipv4.ip_forward=1. If not enabled, falls back to standard client mode. Previously, the iptables rules would be put in place but nothing would actually happen. Although this new check doesn't affect the observed behaviour, you do get a warning in the log.
zyclonite
commented
2 years ago i had to fix the build, can you rebase to the latest main first - if everything is ok from your side, i can merge after that
Paraphraser
commented
2 years ago I'm sorry to be a pain but you may as well be speaking Martian.
I have heard the term "rebase" but I have no idea what it means in practice. From the context, I assume "main" is a branch so "latest" is likely to imply a pull of some kind but how that all fits into a structure where I'm pushing commits into a fork of your repo…
Here's what I see in my working copy:
$ git branch -a
* 20220718-router-master
main
remotes/origin/20220718-router-master
remotes/origin/HEAD -> origin/main
remotes/origin/main
remotes/upstream/main
$ git remote -v
origin https://github.com/Paraphraser/zerotier-docker.git (fetch)
origin https://github.com/Paraphraser/zerotier-docker.git (push)
upstream https://github.com/zyclonite/zerotier-docker.git (fetch)
upstream https://github.com/zyclonite/zerotier-docker.git (push)
$ git status
On branch 20220718-router-master
nothing to commit, working tree clean
$ I'll happily execute any commands you tell me to.
The alternative is iterations of Google-and-guess, which always comes with the risk of guessing poorly and creating an even greater mess.
bfg100k
commented
2 years ago Hi Phill, I suggest you change the default mode to both (i.e. most permissive) rather than choosing a side. Furthermore, this mode is also backwards compatible for both existing inbound or outbound only scenarios.
On Wed, 27 Jul 2022 at 22:11, Phill @.***> wrote:
I'm going to trust the GitHub UI where it tells me that I can still push more commits. I hope I don't break anything.
I've completed testing the extra changes:
Supports three routing modes as proposed by @bfg100k
a ZEROTIER_ONE_GATEWAY_MODE variable with supported values of inbound (forward traffic from ZeroTier cloud to local interfaces), outbound (forward traffic from local interfaces to ZeroTier cloud) and both (bi-directional).
Defaults to inbound (which means no change in behaviour so it's fully backwards compatible).
Also checks for net.ipv4.ip_forward=1. If not enabled, falls back to standard client mode. Previously, the iptables rules would be put in place but nothing would actually happen. Although this new check doesn't affect the observed behaviour, you do get a warning in the log.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>
Paraphraser
commented
2 years ago I did give some thought to that but I really wanted to be able to swear, hand-on-heart, that everything was fully backwards-compatible. For all I know, there are installations that assume what we're now calling "inbound", simply because that's the existing behaviour.
In any event, you've probably seen earlier in this thread that I'm being asked to do something with Git/GitHub that I have no idea how to tackle. I'm always leery of compounding problems and switching to "both" sounds like I might compound whatever problem I may've created already.
How about we wait until the dust has settled, and then one of us can put in a second PR to make "both" the default?
zyclonite
commented
2 years ago I'll happily execute any commands you tell me to.
maybe the easiest way is to use the web for doing the rebase (see: https://github.blog/changelog/2022-02-03-more-ways-to-keep-your-pull-request-branch-up-to-date/)
Paraphraser
commented
2 years ago Just to make sure that I'm looking at the right page (the PR in your repo, not anywhere in my fork of your repo):
This is what I see:
The link you sent me before seems to show a "Merge Pull Request" button that, when clicked, reveals the "Update Branch" drop-down-menu button. I don't see the first button.
I've clicked on anything that seems clickable. Nada.
I'd really love to help get past this problem but I can't get to first base. Sorry for being a pest.
Paraphraser
commented
2 years ago Over on my fork, I see:
Clicking "Discard" seems the wrong thing to do. Clicking Open leads to:
but where to next ... ?
zyclonite
commented
2 years ago no worries, you can look for git rebase but i did revert my main branch and will merge your PR now
zyclonite
commented
2 years ago @Paraphraser you can now give the image tag router-main a test run, if everything is fine i can run a release build
Paraphraser
commented
2 years ago I've tested:
image: "zyclonite/zerotier:router-main"on two separate hosts and I'm pretty sure it's working "as advertised".
The only thing that made my palms itch was:
$ docker images
zyclonite/zerotier router-main 759e05814b5c 12 hours ago 12.6MB
iotstack_zerotier latest 4b3d74896be9 18 hours ago 13.3MBI was not expecting a significant change (5.6%) in size. I don't view it as evidence of a problem. Just something I noticed in passing.
The workflows suddenly started running on my fork - failed at the upload to DockerHub of course - but I figured out how to turn all that off.
Getting back to yesterday's question of the rebase, after Googling about without finding any example that exactly matched the situation, I pieced-together a couple of hints and came up with:
$ cd pull-request-folder
$ git switch main
$ git merge upstream/main
$ git switch pull-request-branch
$ git rebase origin/main
$ git push --forceDoes that look right to you? If yes, I'll squirrel that snippet away in case there's a "next time".
bfg100k
commented
2 years ago Tested the image on my synology NAS and works as expected. Only issue seems to be that somehow the default FORWARD policy is ACCEPT hence will need additional DROP rules to make sure the inbound / outbound scenarios work as expected. I've made the change and created a new pull request for this.
On Fri, 29 Jul 2022 at 11:50, Phill @.***> wrote:
I've tested:
image: "zyclonite/zerotier:router-main"
on two separate hosts and I'm pretty sure it's working "as advertised".
The only thing that made my palms itch was:
$ docker images zyclonite/zerotier router-main 759e05814b5c 12 hours ago 12.6MB iotstack_zerotier latest 4b3d74896be9 18 hours ago 13.3MB
I was not expecting a significant change (5.6%) in size. I don't view it as evidence of a problem. Just something I noticed in passing.
The workflows suddenly started running on my fork - failed at the upload to DockerHub of course - but I figured out how to turn all that off.
Getting back to yesterday's question of the rebase, after Googling about without finding any example that exactly matched the situation, I pieced-together a couple of hints and came up with:
$ cd pull-request-folder $ git switch main $ git merge upstream/main $ git switch pull-request-branch $ git rebase origin/main $ git push --force
Does that look right to you? If yes, I'll squirrel that snippet away in case there's a "next time".
— Reply to this email directly, view it on GitHub https://github.com/zyclonite/zerotier-docker/pull/12#issuecomment-1198799918, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQWGN6EHMISDKKAZT4UYBDVWM2FPANCNFSM54Q6OTHQ . You are receiving this because you were mentioned.Message ID: @.***>
zyclonite
commented
2 years ago The only thing that made my palms itch was:
$ docker images zyclonite/zerotier router-main 759e05814b5c 12 hours ago 12.6MB iotstack_zerotier latest 4b3d74896be9 18 hours ago 13.3MBI was not expecting a significant change (5.6%) in size. I don't view it as evidence of a problem. Just something I noticed in passing.
could be that the layers are compressed on one of the images
$ cd pull-request-folder $ git switch main $ git merge upstream/main $ git switch pull-request-branch $ git rebase origin/main $ git push --forceDoes that look right to you? If yes, I'll squirrel that snippet away in case there's a "next time".
looks right 👍
Identifier
commented
1 year ago Is bfg100k/zerotier-gateway obsolete now? It looks like everything there has been merged into zyclonite/zerotier-docker now.
I would have asked this question on bfg100k/zerotier-gateway but I don't see an Issues tab there.
P.S.
This thread in itself was very helpful in wrapping my head around all the networking going on here! Special thanks to @Paraphraser for being such a stickler about the terminology. Things get so confusing when people mix up the terms bridge/router/gateway, etc. And it was great to see @bfg100k and @zyclonite jumping in and the 3 of you working together so well.
Personally, I'm on a journey to try and get my remote device (iPad on LTE) to be able to connect my home LAN while I'm on the road, but my home ISP uses a CGNAT (i.e. my home router is behind a double nat). For all I know the iPad on LTE is sometimes behind a CGNAT too.
I tried using Tailscale first (the experience is soooo much easier than Zerotier), but I couldn't get Steam Link to work over the connection. Reddit suggests Steam Link doesn't work over Tailscale because Tailscale operates at Level 3 whereas the mechanism that Steam Link uses (multicasting? broadcasting?) only works over Level 2. Since I read that Zerotier operates at Level 2, I'm trying to set it up on my home network to act as a bridge so my remote devices (e.g. my iPad) will appear to be directly connected to my home LAN, just as if they were physically at home and connected to my home router's Wifi.
My router at home is running stock ASUS firmware, which doesn't have support for Zerotier (yet. But they just added support for Wireguard recently, so maybe Zerotier will be added in the future?) I might be able to install OpenWRT on it, but I've read reports that OpenWRT doesn't work well anymore on modern routers and I don't want to risk messing up my connectivity at home.
I do have a NAS at home though, a QNAP TS-253B, so I decided to give that a try to host Zerotier. It even has two NICs, so it seemed perfect for following Zerotier's guide Layer 2 Bridging of Ethernet and ZeroTier Networks on Linux. I was able to install Zerotier on the QNAP by uploading the QPKG via it's web GUI, and I'm also able to SSH into it, but unfortunately QNAP runs Busybox which is missing many of the commands instructed in the guides, such as ifup. Also, even if I was able to get the IP tables and /etc/network/interfaces files set up, they'd likely be obliterated on reboot, because that's just how QNAP works (and maybe other NAS devices too).
But not all hope is lost. QNAP is good at running Docker, and the docker containers are well-persisted between reboots, so that's what brought me to @zyclonite's zerotier-docker and @bfg100k's zerotier-gateway. I'm just unsure of which one to install, hence the reason for my initial question above.
Paraphraser
commented
1 year ago Perhaps read these in the IOTstack Wiki:
I have the ZeroTier-router running in IOTstack on a Raspberry Pi. I have the client software on macOS (laptop, desktop) and on iOS (iPhone, iPad). It all "just works". The only real issue is the iOS app hasn't been updated in a while and, judging from its behaviour, it doesn't support DNS propagation. The way I want to run my remote clients is for DNS to come from Pi-hole over the ZeroTier subnet. That works on macOS (if you enable the option in the connection menu) but not (yet) on iOS. I guess they'll fix it one day.
Paraphraser
commented
1 year ago And to answer your question directly: zerotier-router is what you need if you want to "phone home" to your home network, from a remote client, via a shared ZeroTier Cloud network. It's equivalent to what you get with WireGuard.
zerotier (non router) is what you would run if you want a client to attach to a ZeroTier Cloud network just "as itself". It's equivalent to what you get if you run the ZeroTier app on iOS or macOS.
Identifier
commented
1 year ago Thanks @Paraphraser for the response. By "zerotier-router" I'm assuming you mean zyclonite/zerotier:router.
I installed that image using the following docker compose yaml in Container Station on my QNAP:
version: '3'
services:
zerotier:
image: "zyclonite/zerotier:router"
container_name: zerotier-one
devices:
- /dev/net/tun
network_mode: host
volumes:
- /share/Containers/zerotier-one:/var/lib/zerotier-one
cap_add:
- NET_ADMIN
- SYS_ADMIN
- NET_RAW
restart: unless-stopped
environment:
- TZ=Etc/UTC
- PUID=999
- PGID=994
- ZEROTIER_ONE_LOCAL_PHYS=eth0
- ZEROTIER_ONE_USE_IPTABLES_NFT=true
- ZEROTIER_ONE_GATEWAY_MODE=both
- ZEROTIER_ONE_NETWORK_IDS=MyNeTwOrKiDThe first thing I noticed is the container errored out right away, with the output:
launching ZeroTier-One in routing mode
adding iptables-nft rules for bi-directional traffic (local interfaces eth0 to/from ZeroTier)
Warning: Extension MASQUERADE revision 0 not supported, missing kernel module?
iptables v1.8.8 (nf_tables): Could not fetch rule set generation id: Invalid argumentSo apparently it's a good thing you kept ZEROTIER_ONE_USE_IPTABLES_NFT false by default for backwards compatibility.
I changed ZEROTIER_ONE_USE_IPTABLES_NFT back to false and the container starts successfully:
version: '3'
services:
zerotier:
image: "zyclonite/zerotier:router"
container_name: zerotier-one
devices:
- /dev/net/tun
network_mode: host
volumes:
- /share/Containers/zerotier-one:/var/lib/zerotier-one
cap_add:
- NET_ADMIN
- SYS_ADMIN
- NET_RAW
restart: unless-stopped
environment:
- TZ=Etc/UTC
- PUID=999
- PGID=994
- ZEROTIER_ONE_LOCAL_PHYS=eth0
- ZEROTIER_ONE_USE_IPTABLES_NFT=false
- ZEROTIER_ONE_GATEWAY_MODE=both
- ZEROTIER_ONE_NETWORK_IDS=MyNeTwOrKiD
launching ZeroTier-One in routing mode
adding iptables rules for bi-directional traffic (local interfaces eth0 to/from ZeroTier)
ZeroTier daemon is running as process 16Then I went to https://my.zerotier.com/network/MyNeTwOrKiD and authorized the new node, and I can get to some of my home computers from my iPad (testing HTTP connections with Chrome at first). For example I can get to 192.168.50.1 (my home router), and I can get to 192.168.50.2:6789 (NZBGet running in another docker container on my QNAP (in NAT network mode)). But I can't get to 192.168.50.2:8080 (my QNAP's web UI), or 192.168.50.2:8581 (HomeBridge running in another docker container on my QNAP in Host network mode). I verified that both of those URLs work fine when the iPad is on my home Wifi.
Personally, I'd be fine without those specific URLs working, but unfortunately Steam Link also fails to work, which is why I'm trying so hard to get ZeroTier working in the first place. Otherwise, Tailscale would have been sufficient (and way easier). Here's some observations I'm seeing with ZeroTier in it's half-working state I'm in now:
In the ZeroTier Central (my.zerotier.com/network/MyNeTwOrKiD) Managed Routes, I have 172.22.0.0/16 (LAN) and 192.168.50.0/23 via 127.22.0.1:
But I was surprised not to see a route for 192.168.50.0/23 (i.e. a genmask of 255.255.254.0) in the QNAP or zerotier-one's routing tables:
All you see is the normal 192.168.50.0/24 (255.255.255.0) route that came from the router. Not sure if that has anything to do with anything.
Also, checking the Allow Ethernet Bridging box in ZeroTier Central had no noticeable effect, which brings me to my final observation in all this:
Even after all this effort, ZeroTier in this configuration is at most only going to act as a "router" (yet another NAT), when what I really want is true ethernet "bridged" mode. For example I'd imagine in bridge mode my iPad would have an IP in the 192.168.50.* range and appear to coexist on the same subnet as my home LAN. It could presumably even obtain its IP address from my router's DHCP server itself once it's connected to the ZeroTier bridge. (But if that's not possible, I'd be fine setting my router's DHCP address limit to 192.168.50.199 and let ZeroTier auto-assign IPs from 192.168.50.200-255).
Would that be possible for ZeroTier to do from within a docker container? It has access to /dev/net/tun and NET_ADMIN, SYS_ADMIN, and NET_RAW, so it sounds feasible.
Paraphraser
commented
1 year ago By "zerotier-router" I'm assuming you mean zyclonite/zerotier:router.
Yes. Sorry for the confusion. I tend to think about these in terms of IOTstack. There the menu offers "zerotier-client" and "zerotier-router" with two separate service definitions:
image: zyclonite/zerotierimage: zyclonite/zerotier:routerBut, apart from the change back to iptables, the service definition you're using is the same as the IOTstack "router".
Your managed routes look correct in ZeroTier Central. I assume you're implementing what I call Topology 2 in the IOTstack documentation, rather than Topology 3.
In other words, the behaviour you expect when your iPad is at a remote location:
If you were expecting a WireGuard-like full tunnel where everything goes home and stuff destined for the Internet comes back out of your home router then you need Topology 3.
But, putting a pin in that topic, and proceeding on the assumption that Topology 2 is your goal, the only thing you haven't confirmed is whether you added a static route in your home router pointing back to the ZeroTier Cloud, as in:
destination: 172.22.0.0/16 next-hop 192.168.50.xwhere X is the LAN-side IP address of the device where ZeroTier (router) is running. From the way you describe the "this works, that doesn't", this is probably the missing link. I just can't be sure because I don't understand the physical devices that are involved.
Also, if you did already add that route and just didn't mention it in your earlier discussion, please go and check that it is actually there. You would not be the first person to go to a router, add a static route, think it had "stuck", only to find later that it had vanished.
All you see is the normal 192.168.50.0/24 (255.255.255.0) route that came from the router. Not sure if that has anything to do with anything.
No. It doesn't have anything to do with anything.
To elaborate, that route isn't coming from your router. When you do an ifconfig eth0 you'll see the IP address for that interface and the subnet mask. The two are ANDed:
192.168.50.x AND 255.255.255.0 = 192.168.50.0The simple fact of an interface having an IP address (static, fixed-DHCP, variable-DHCP) results in an interface route being added to the routing table.
So the last line of the routing table is read as "to reach network 192.168.50.0/24, go via eth0". When a packet comes along where the destination address is in that network, IP (layer 3) sees it's a local interface, ARP's "who has this destination IP address?", gets back a MAC address of the destination host, then hands the packet to the MAC layer for transmission.
If a packet has a destination address that doesn't match anything in the routing table, it arrives at the default route where the next-hop is your router - which is in 192.168.50.0/24 so ARP-to-MAC.
There's nothing saying your router can't be advertising that route, or that the device where you ran the route command can't be running a routing protocol daemon to hear and respond to those advertisements but it would be fairly unusual and, without a lot of mucking about, wouldn't override a local interface route which would have a higher priority.
By the way, I reckon routing tables make more sense if you use the -n switch. That stops the thing from trying to figure out names. In the "gateway" column where you see RT-AX86U.local, using -n will turn that into the IP address. You can look at RT-AX86U.local and think you know that it means 192.168.50.1 (or whatever number you're using in that last octet) but I'd rather see the stone cold hard fact that the address is what I think it is.
If you want to go further with this, perhaps a diagram of the hosts involved would help - your home topology, IP addresses for the hosts we're talking about, what's running on each. It's just a bit hard to wrap my mind around what might be going on.
Here's my routing table for comparison:
$ docker exec zerotier route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.132.1 0.0.0.0 UG 202 0 0 eth0
0.0.0.0 192.168.132.1 0.0.0.0 UG 303 0 0 wlan0
10.244.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ztr2qsmswx
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.30.0.0 0.0.0.0 255.255.252.0 U 0 0 0 br-1b18f152b579
192.168.132.0 0.0.0.0 255.255.255.0 U 202 0 0 eth0
192.168.132.0 0.0.0.0 255.255.255.0 U 303 0 0 wlan0
$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.132.1 0.0.0.0 UG 202 0 0 eth0
0.0.0.0 192.168.132.1 0.0.0.0 UG 303 0 0 wlan0
10.244.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ztr2qsmswx
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.30.0.0 0.0.0.0 255.255.252.0 U 0 0 0 br-1b18f152b579
192.168.132.0 0.0.0.0 255.255.255.0 U 202 0 0 eth0
192.168.132.0 0.0.0.0 255.255.255.0 U 303 0 0 wlan0Both eth0 and wlan0 into the same subnet (bridged Ethernet and WiFi). 172.x are docker. 10.244 is ZeroTier Cloud.
No sign of 192.168.132/23.
You can think about this in two ways. Let's suppose I do actually have the two /24 subnets:
The assumption behind the IOTstack topology descriptions is that the device running "zerotier-router" knows how to reach both of those. A good example might be where Ethernet is 132 and WiFi is 133.
If the device running "zerotier-router" has an interface in each subnet, it will add the necessary interface routes (as above).
If the device running "zerotier-router" only has, say, Ethernet then when it gets a packet destined for .133, we don't want it to send the packet back to the ZeroTier Cloud because the ZeroTier Cloud will send it straight back because it has "192.168.132/23 goes to the device running zerotier-router". We'd have a routing loop.
If we assume the home router has interfaces for both Ethernet and WiFi then the device running zerotier-router will send something for .133 to the router - which knows how to deliver it.
It's only if we get to the point where the home router doesn't know how to reach .133 that we need to consider adding a static route to the device running zerotier-router to tell it the next hop.
Make sense?
Now, hold that thought and switch focus to a remote device. That doesn't have any local interfaces in .132 or .133 so it needs to learn "192.168.132/23 is via the ZeroTier Cloud". By definition that remote device has an interface route to the ZeroTier cloud. That remote device doesn't need to care whether both .132 and .133 exist. That's a problem for the device running zerotier-router to solve once the packet arrives.
It's also true that devices on your home network other than the device running zerotier-router and which have their own interfaces into the ZeroTier Cloud will learn 192.168.132/23 from ZeroTier.
Visualise the following:
The packet will match 192.168.132/23, be forwarded to Z via ZeroTier Cloud, will arrive at Z, and Z will forward the packet via its default route (your home router).
Your home router may have a filter which drops RFC1918 packets before they get sent over the WAN to your ISP but the usual situation is the packet gets sent to the ISP. The ISP promptly drops it with "destination network unreachable".
So, it can happen but the probability is low because it depends on making a mistake with .133.
Anyway, I'll shut up now.
Identifier
commented
1 year ago Your managed routes look correct in ZeroTier Central. I assume you're implementing what I call Topology 2 in the IOTstack documentation, rather than Topology 3.
Exactly.
If you were expecting a WireGuard-like full tunnel where everything goes home and stuff destined for the Internet comes back out of your home router then you need Topology 3.
Nope, no need for that in this scenario. I'm fine having general internet traffic use the LTE provider.
But, putting a pin in that topic, and proceeding on the assumption that Topology 2 is your goal, the only thing you haven't confirmed is whether you added a static route in your home router pointing back to the ZeroTier Cloud, as in:
destination: 172.22.0.0/16 next-hop 192.168.50.xwhere X is the LAN-side IP address of the device where ZeroTier (router) is running. From the way you describe the "this works, that doesn't", this is probably the missing link. I just can't be sure because I don't understand the physical devices that are involved.
You're correct that I forgot to do that. I've done it now, adding a route for 172.22.0.0 with netmask 255.255.0.0 to gateway 192.168.50.2 (my QNAP):
It didn't change the behavior of "this works, that doesn't" though. It's still like this:
Using Chrome on my iPad on LTE + ZeroTier (client) VPN enabled: ✅ Navigating to 192.168.50.1:80 (my ASUS RT-AX86U router) ❌ Navigating to 192.168.50.2:8080 (my QNAP web gui) ✅ Navigating to 192.168.50.2:6789 (NZBGet running in a docker container on my QNAP on NAT network lxcbr0) ❌ Navigating to 192.168.50.2:8581 (HomeBridge running in a docker container on my QNAP on Host network) ✅ Navigating to 192.168.60.2:7878 (Radarr running in a docker container on my QNAP on container network br-a6b89ff490cc) ✅ Navigating to 192.168.50.88:80 (Philips HUE bridge connected to my ASUS router) ✅ Navigating to 192.168.50.160:3000 (Node.js hello world running on my home PC) ❌ Steam Link streaming from 192.168.50.160. It can connect to the PC but fails to stream.
So I think the first two failure cases are just some problem with the QNAP routing to itself, but that's ok, I don't need access to those anyway.
By the way, I reckon routing tables make more sense if you use the
-nswitch. That stops the thing from trying to figure out names. In the "gateway" column where you see RT-AX86U.local, using-nwill turn that into the IP address. You can look atRT-AX86U.localand think you know that it means 192.168.50.1 (or whatever number you're using in that last octet) but I'd rather see the stone cold hard fact that the address is what I think it is.
Sure, no problem!
If you want to go further with this, perhaps a diagram of the hosts involved would help - your home topology, IP addresses for the hosts we're talking about, what's running on each. It's just a bit hard to wrap my mind around what might be going on.
Here's my routing table for comparison:
$ route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.132.1 0.0.0.0 UG 202 0 0 eth0 0.0.0.0 192.168.132.1 0.0.0.0 UG 303 0 0 wlan0 10.244.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ztr2qsmswx 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 172.30.0.0 0.0.0.0 255.255.252.0 U 0 0 0 br-1b18f152b579 192.168.132.0 0.0.0.0 255.255.255.0 U 202 0 0 eth0 192.168.132.0 0.0.0.0 255.255.255.0 U 303 0 0 wlan0Both
eth0andwlan0into the same subnet (bridged Ethernet and WiFi). 172.x are docker. 10.244 is ZeroTier Cloud.
Interesting, from what I read on IOTStack, when a route matches two routing rules, it's up to the manufacturer on which one gets used. How does your system know that it should send the routes via both eth0 and wlan0 instead of just picking "the first one"? The reason I'm asking is because I want to understand how I can eventually get ZeroTier to run in bridge mode on my QNAP instead of in router mode, to see if it will make Steam Link start working.
Paraphraser
commented
1 year ago If a host is reachable for one thing (eg port 6789) but not for another then it's reachable so routing works, which means the problem lies with the thing that doesn't work, not with TCP/IP or ZeroTier per se. I'm not familiar with most of the containers you run but my recollection is homebridge runs in host mode so it can participate in multicast. So maybe the works/not works is host mode/not host mode. I'm right in concluding .2 is where ZeroTier-router is running in a container - yes?
If it were me, I'd start using tcpdump to see what's going where.
On your last question, study the metric column. If this device is connected to both eth0 and wlan0 then eth0 will be chosen first because 202 is a lower metric. Disconnect eth0 and that route goes away so there's no problem - packets now go out wlan0. This metric column is specific to (at least) Debian and (possibly) Linux generally but not macOS. That's the implementation dependency and generally, however it's actually implemented for a given host, when it comes to physical interfaces, the tie-breaker will prefer a faster interface over a slower interface. It gets a bit trickier when a route is learned via a routing protocol or when the host can't actually disambiguate on speed, number of hops, and so on. Then you generally get load-levelling (nothing about IP says any two packets have to traverse the same path). Most of the time one route will wind up being preferred. The corollary is that if you actually want multipath routing, you usually have to force it to happen by manipulating the implementation-dependent mechanism. Well, in my experience anyway.
Also think about it from the perspective of two other devices, one only has Ethernet, one only has WiFi. If our target device has both interfaces active, it's reachable on both. If the WiFi device knows the IP address of the Wifi interface then WiFi is the medium. If the WiFi device only knows the IP of the Ethernet interface then a "bridge" somewhere (eg the WiFi base station) has to translate between WiFi and Ethernet.
Does any of that help?
Identifier
commented
1 year ago If a host is reachable for one thing (eg port 6789) but not for another then it's reachable so routing works, which means the problem lies with the thing that doesn't work, not with TCP/IP or ZeroTier per se. I'm not familiar with most of the containers you run but my recollection is homebridge runs in host mode so it can participate in multicast. So maybe the works/not works is host mode/not host mode. I'm right in concluding .2 is where ZeroTier-router is running in a container - yes?
Yup, that's my understanding too. And yes, 192.168.50.2 is where ZeroTier-router is running in a container.
If it were me, I'd start using tcpdump to see what's going where.
Unfortunately tcpdump isn't available, but like I said, I'm fine with those ports being inaccessible.
Does any of that help?
Yes, thank you, all of your advice is very helpful for understanding all of this! I think we've successfully set up ZeroTier as a "router". Unfortunately Steam Link still isn't working, so I'm basically at the same point I was with Tailscale.
I think the next step is to try and get ZeroTier working as a true "bridge", where my ZeroTier cloud addresses are literally in the same subnet as my home LAN. I think this will involve checking the "Allow Ethernet Bridging" box in ZeroTier central, removing the auto-assigned IPs, and figuring out how to bridge eth0 with zt5u4qe64t on my QNAP. Do you suggest I stick with image: zyclonite/zerotier:router? Or should I use zerotier/zerotier:latest or zyclonite/zerotier:latest (the non-router version) instead?
Identifier
commented
1 year ago FYI the official zerotier/zerotier:latest was a non-starter since it couldn't access /dev/net/tun on my QNAP, since the zerotier/zerotier container runs as user 999.
Manually ssh-ing into the QNAP and doing sudo chmod go+rw /dev/net/tun fixes that problem, but unfortunately it resets back on reboot. I tried changing the compose file to specify user: root and/or user: 0:0, but zerotier/zerotier:latest still insists on running as 999 no matter what I do.
zyclonite/zerotier:latest doesn't have that problem even though it also runs as 999 (good job, @zyclonite!), so I've been doing my best to get things working with zyclonite's version.
I've tried different permutations of "Allow Ethernet Bridging", assigning an IP, not assigning an IP, setting allowManaged=0, setting back to allowManaged=1, but I think it all boils down to there's something I need to manually do on the QNAP to bridge my zt5u4qe64t with eth0. I found I could use brctl to do it in Busybox, but whenever I try to brctl addif br0 eth0 zt5u4qe64t, the network goes down and I have to reboot. I can't tell you how many times I've rebooted my QNAP in the past 24 hours. And the thing takes 25 minutes to boot up since it has full-disk encryption 😭
At this point I'm just going to buy a Raspberry Pi and give up trying to get ZeroTier to work in bridged mode on my QNAP. Thanks again for all your replies. For the record, "router" mode works fine. Now we just need a fully automatic zyclonite/zerotier:bridge version as well 😄
bfg100k
commented
1 year ago Late to the party but here's my 2 cents. As Phil said, given you can reach other services on the same IP (both 192.168.50.2 and 192.168.50.160, its no longer a route issue.
1) For QNAP web admin and Homebridge - Have you checked if they are IP restricted (i.e. they only accept connections from the same subnet which is why it works when your ipad is on the home wifi)?
2) Can't access SteamLink - Not familiar with SteamLink but based on a quick google, it appears other folks have the same issue when clients and hosts are on separate subnets. https://steamcommunity.com/groups/homestream/discussions/0/606068060815627379/ ?
Some possible solutions are suggested here https://steamcommunity.com/groups/homestream/discussions/0/606068060815627379/?ctp=2#c1697174779865867407 https://steamcommunity.com/groups/homestream/discussions/0/606068060815627379/?ctp=2#c1735462352476358319 https://steamcommunity.com/groups/homestream/discussions/0/606068060815627379/?ctp=2#c3393916911748284544
IMHO, trying to bridge over WAN connections (i.e. have remote machines share the same subnet as local ones) isn't a good idea for control, security and performance reasons.
On Mon, 6 Feb 2023 at 05:02, Kael Rowan @.***> wrote:
FYI the official zerotier/zerotier:latest https://hub.docker.com/r/zerotier/zerotier was a non-starter since it couldn't access /dev/net/tun on my QNAP, since the zerotier/zerotier container runs as user 999.
[image: image] https://user-images.githubusercontent.com/1703361/216831644-3009796e-5c9a-4f7c-9d14-a772237ad5c5.png
Manually ssh-ing into the QNAP and doing sudo chmod go+rw /dev/net/tun fixes that problem, but unfortunately it resets back on reboot. I tried changing the compose file to specify user: root and/or user: 0:0, but zerotier/zerotier:latest https://hub.docker.com/r/zerotier/zerotier still insists on running as 999 no matter what I do.
zyclonite/zerotier:latest https://hub.docker.com/r/zyclonite/zerotier doesn't have that problem even though it also runs as 999 (good job, @zyclonite https://github.com/zyclonite!), so I've been doing my best to get things working with zyclonite's version.
I've tried different permutations of "Allow Ethernet Bridging", assigning an IP, not assigning an IP, setting allowManaged=0, setting back to allowManaged=1, but I think it all boils down to there's something I need to manually do on the QNAP to bridge my zt5u4qe64t with eth0. I found I could use brctl to do it in Busybox, but whenever I try to brctl addif br0 eth0 zt5u4qe64t, the network goes down and I have to reboot. I can't tell you how many times I've rebooted my QNAP in the past 24 hours. And the thing takes 25 minutes to boot up since it has full-disk encryption 😭
At this point I'm just going to buy a Raspberry Pi and give up trying to get ZeroTier to work in bridged mode on my QNAP. Thanks again for all your replies. For the record, "router" mode works fine. Now we just need a fully automatic zyclonite/zerotier:bridge https://hub.docker.com/r/zyclonite/zerotier/tags?name=bridge version as well 😄
— Reply to this email directly, view it on GitHub https://github.com/zyclonite/zerotier-docker/pull/12#issuecomment-1418202629, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQWGN7NWWADJF555Q3ZCCTWV7TKVANCNFSM54Q6OTHQ . You are receiving this because you were mentioned.Message ID: @.***>
Paraphraser
commented
1 year ago I was going to make the same point as @bfg100k about the wisdom (or otherwise) of bridging across a WAN link. That's something I'd never do.
Why? The whole point of routers (the reason they were created in the first place) is that they segment networks into Broadcast Domains.
A broadcast domain is another word for a subnet but it is a better way of thinking about what is going on - the domain describes the boundaries for the propagation of broadcast traffic (or, more precisely, non-unicast traffic).
The reason routers were invented to do that is because non-unicast traffic is the key limiting factor on scaling networks.
Sure, a "switch" or a "bridge" (same thing, the former is just a marketing term which means the bridging is done in silicon rather than software) will get you some distance down the scaling highway because it lets you split collision domains to limit the propagation of unicast traffic but non-unicast traffic still has to flood.
Networks have a lot more non-unicast traffic than you might think and the last thing you want is that squirting around all over the place - particularly if your "remote" is on a cellular link where you can wind up paying for each extra byte.
If you don't have tcpdump that's solved with something like:
$ sudo apt update
$ sudo apt install -y tcpdumpGoogle is your friend when it comes to constructing tcpdump commands to capture traffic of interest (eg sniff on this interface, only grab traffic to or from that other host, or associated with a particular port).
If you're not experienced with interpreting tcpdump output as it appears in the command line then I'd recommend augmenting each command you come up with by Googling with:
-s 0 -n -w «filename».pcapThe -s 0 says "capture entire packets", the -n says "don't try to resolve IP addresses back to DNS names" (that just adds confusion) and the -w tells tcpdump where to save the packet captures.
For example, to capture MQTT traffic (port 1883) arriving on my Raspberry Pi, I would use:
$ sudo tcpdump -i eth0 -s 0 -n -w mqtt.pcap dst port 1883Use control+C to terminate. Then, I'd move the mqtt.pcap file to a host where I had WireShark installed (Mac, Windows).
If you find yourself wondering what a container can see, there's nothing stopping you from installing tcpdump inside the container and running it there. Once installed, it will persist until you recreate the container.
You can either write tcpdump output to the container's persistent store (ie so you can see the file from outside the container) or just write to any old place inside the container and use docker cp to retrieve it.
My guess is that all your problems will go away once you get ZeroTier-router running on a Pi.
If you visualise a packet coming from your remote iPad, it will have the source IP address allocated to the iPad in the ZeroTier cloud. It will reach the ZeroTier-router and pass through NAT so it's source IP address will then look like it's the Raspberry Pi (ie on the same subnet as all your other devices). Those other devices will reply to the Raspberry Pi and then NAT will rewrite the packet on its way back to the iPad.
Also, please don't think you need to dedicate a Raspberry Pi to this. Here's my primary Pi:
NAMES CREATED STATUS
wireguard 4 days ago Up 4 days
pihole 4 days ago Up 4 days (healthy)
influxdb 11 days ago Up 11 hours (healthy)
zerotier 4 weeks ago Up 4 days
nodered 4 weeks ago Up 4 days (healthy)
mosquitto 4 weeks ago Up 4 days (healthy)
portainer-ce 4 weeks ago Up 4 days
grafana 4 weeks ago Up 4 days (healthy)Happily running both WireGuard and ZeroTier side by side. From a remote ZeroTier client I can reach anything on that Pi (including the other containers - most commonly Grafana). I can reach anything else on my home network. No restrictions, no compromises.
When the remote is a Mac laptop, it gets DNS from PiHole by virtue of the "Allow DNS Configuration" option in the ZeroTier Client menu running on the laptop.
It would be nice if remote iOS devices could do that too but, at the moment, there is no support for that in the iOS app.
When any remote is a WireGuard client, they all get DNS from PiHole. But WireGuard depends on not being CGNATted (I'm lucky enough to have an ISP that hasn't done that to me - yet - but it's probably a matter of time, which is why I run both).
bfg100k
commented
1 year ago offtopic and specifically addressing Phill's comment - " It would be nice if remote iOS devices could do that too but, at the moment, there is no support for that in the iOS app."
Yes there is support for this in iOS - just that its not a checkbox option as found in the clients for other platforms. First it’s not possible to enable/disable allowDNS in the iOS client after the network is added. The network first has to be removed, then re-added, at which point allowDNS can be set.
On Mon, 6 Feb 2023 at 13:47, Phill @.***> wrote:
I was going to make the same point as @bfg100k https://github.com/bfg100k about the wisdom (or otherwise) of bridging across a WAN link. That's something I'd never do.
Why? The whole point of routers (the reason they were created in the first place) is that they segment networks into Broadcast Domains.
A broadcast domain is another word for a subnet but it is a better way of thinking about what is going on - the domain describes the boundaries for the propagation of broadcast traffic (or, more precisely, non-unicast traffic).
The reason routers were invented to do that is because non-unicast traffic is the key limiting factor on scaling networks.
Sure, a "switch" or a "bridge" (same thing, the former is just a marketing term which means the bridging is done in silicon rather than software) will get you some distance down the scaling highway because it lets you split collision domains to limit the propagation of unicast traffic but non-unicast traffic still has to flood.
Networks have a lot more non-unicast traffic than you might think and the last thing you want is that squirting around all over the place - particularly if your "remote" is on a cellular link where you can wind up paying for each extra byte.
If you don't have tcpdump that's solved with something like:
$ sudo apt update
$ sudo apt install -y tcpdump
Google is your friend when it comes to constructing tcpdump commands to capture traffic of interest (eg sniff on this interface, only grab traffic to or from that other host, or associated with a particular port).
If you're not experienced with interpreting tcpdump output as it appears in the command line then I'd recommend augmenting each command you come up with by Googling with:
-s 0 -n -w «filename».pcap
The -s 0 says "capture entire packets", the -n says "don't try to resolve IP addresses back to DNS names" (that just adds confusion) and the -w tells tcpdump where to save the packet captures.
For example, to capture MQTT traffic (port 1883) arriving on my Raspberry Pi, I would use:
$ sudo tcpdump -i eth0 -s 0 -n -w mqtt.pcap dst port 1883
Use control+C to terminate. Then, I'd move the mqtt.pcap file to a host where I had WireShark installed (Mac, Windows).
If you find yourself wondering what a container can see, there's nothing stopping you from installing tcpdump inside the container and running it there. Once installed, it will persist until you recreate the container.
You can either write tcpdump output to the container's persistent store (ie so you can see the file from outside the container) or just write to any old place inside the container and use docker cp to retrieve it.
My guess is that all your problems will go away once you get ZeroTier-router running on a Pi.
If you visualise a packet coming from your remote iPad, it will have the source IP address allocated to the iPad in the ZeroTier cloud. It will reach the ZeroTier-router and pass through NAT so it's source IP address will then look like it's the Raspberry Pi (ie on the same subnet as all your other devices). Those other devices will reply to the Raspberry Pi and then NAT will rewrite the packet on its way back to the iPad.
Also, please don't think you need to dedicate a Raspberry Pi to this. Here's my primary Pi:
NAMES CREATED STATUS
wireguard 4 days ago Up 4 days
pihole 4 days ago Up 4 days (healthy)
influxdb 11 days ago Up 11 hours (healthy)
zerotier 4 weeks ago Up 4 days
nodered 4 weeks ago Up 4 days (healthy)
mosquitto 4 weeks ago Up 4 days (healthy)
portainer-ce 4 weeks ago Up 4 days
grafana 4 weeks ago Up 4 days (healthy)
Happily running both WireGuard and ZeroTier side by side. From a remote ZeroTier client I can reach anything on that Pi (including the other containers - most commonly Grafana). I can reach anything else on my home network. No restrictions, no compromises.
When the remote is a Mac laptop, it gets DNS from PiHole by virtue of the "Allow DNS Configuration" option in the ZeroTier Client menu running on the laptop.
It would be nice if remote iOS devices could do that too but, at the moment, there is no support for that in the iOS app.
When any remote is a WireGuard client, they all get DNS from PiHole. But WireGuard depends on not being CGNATted (I'm lucky enough to have an ISP that hasn't done that to me - yet - but it's probably a matter of time, which is why I run both).
— Reply to this email directly, view it on GitHub https://github.com/zyclonite/zerotier-docker/pull/12#issuecomment-1418424868, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQWGNZPRFXJDBFFKCRINGTWWBQ2XANCNFSM54Q6OTHQ . You are receiving this because you were mentioned.Message ID: @.***>
Paraphraser
commented
1 year ago Well, perhaps I'm misreading but the iOS client seems to have this kind of functionality in its UI.
You can enable/disable the default route propagation and, yes, if you want to change the setting, you have to tear down and start up the tunnel again.
To me, this would be the logical place for another button which says "accept DNS", just like you can in the other clients (at least Linux and macOS, not sure about Android and Windows).
I take some hope from observing that iOS clients are at 1.8.10, while Linux and macOS are at 1.10.2, and knowing that DNS propagation is a recent feature for ZeroTier. Fingers crossed.
Incidentally, that "DNS servers" field is never populated.
Identifier
commented
1 year ago Well, perhaps I'm misreading but the iOS client seems to have this kind of functionality in its UI.
You can enable/disable the default route propagation and, yes, if you want to change the setting, you have to tear down and start up the tunnel again.
Hi @Paraphraser, I think what @bfg100k means is that you have to choose to allow ZeroTier to control the DNS when you join the network in the iOS app. In other words, you'll need to click the Edit button on the top left of the iOS app, then Delete that "row", then click the + button to Join your network again. Underneath the textbox you enter your network ID, you'll see settings for DNS options.
By the way, I have some half-good news. I was finally able to get ZeroTier (zyclonite/zerotier:latest) working in Bridge mode on my QNAP. Since my QNAP has two physical ethernet ports, I added a cable from eth1 to my router (so now I have two physical cables going from my QNAP to my router), and then I was able to bridge eth1 to zt5u4qe64t.
sudo brctl addbr br1
sudo brctl addif br1 eth1 zt5u4qe64t
sudo ifconfig br1 upThese are my settings in ZeroTier Central:
I'm using manually-assigned IPs for the moment, which overlap with my home router's address space on purpose, but I limited my home router to only dole out DHCP addresses between 192.168.50.10 and 192.168.50.199 so there's no collisions.
Theoretically my iPad could get its IP address directly from my home router's DHCP server, but I'm not sure how to tell the ZeroTier client on iOS to do that.
But, even after all that, Steam Link still doesn't work 😭 It still has the same behavior as it did when I was using ZeroTier as a "router" instead of a "bridge". Is there something additional that needs to be done to get multicast packets to go over bridges? We'll see if I have any better luck when the Raspberry Pi arrives. I've always wanted to play with one anyway, so it should be fun.
Paraphraser
commented
1 year ago Oh! Thank you so much for clarifying that!! I had been silently cursing the good folks at ZeroTier for this apparent "omission" and now it turns out to be PEBKAC. 🤬
Having re-joined as instructed with "Network DNS" enabled, I'm seeing two things:
I had half expected that deleting and re-creating the connection would result in needing to re-authorise the client in ZeroTier Central but it kept on trucking.
Now I'm off to fix my other iOS clients. Again, thanks to both of you!
I spin up bridges like that on a Pi equipped with two USB-to-Ethernet dongles so I can stick the Pi between my router and the FTTC modem (ie I can sniff the WAN traffic via the regular Ethernet port). But I've found it somewhat unreliable. I've never been too sure what's going on but the bridge can sometimes stop forwarding for no good reason. Because I'm only using it for occasional diagnostics I haven't drilled into it. It may be the way I'm going about it. It might be a Raspbian thing. It might be the dongles I'm using. It's just weird enough to place a permanent question mark in my mind.
Yes, I gave up on the somewhat poxy DHCP support in my home router. I'm running both DHCPD and BIND9 in containers on a pair of Raspberry Pis. True, it's only a small home network but old habits with redundancy die hard. Having a decent DHCP server with no arbitrary restrictions and where I can segment clients into groups for different treatment (things that need ad-blocking go to PiHole, things that don't go to BIND9) eases my life.
Hard to say. When it comes to off-the-shelf switches my experience has been that if the data sheet doesn't mention IGMP snooping then a multicast packet will be treated like any other non-unicast packet and flooded out of all connected interfaces. If it does mention IGMP snooping then it will depend on settings and implementation decisions (ie it might default to blocking unless you enable it, or it might default to flooding until you enable it). I've never experimented with a quick and dirty bridge like the one I mentioned above to see what happens.
I do, of course, see multicast traffic (mDNS and the occasional IPv6 advertisements - until I clobber them) but otherwise the traffic behaves as expected and stops at the router.
Well, maybe it's just me but I rediscovered the joys of computing when I got my first Raspberry Pi. I'm an old guy - grew up on the command line. Not quite old enough for paper tape but definitely old enough for 80 column Hollerith punch cards.
While you are waiting for yours to arrive, mosey on over to PiBuilder and wrap your brain around the readme. If you use that to build your Pi you'll get a rock solid platform with IOTstack installed. Just run the menu, select ZeroTier-router, edit the compose file with your ZeroTier network ID and you'll be up and away.
A word to the wise, though. The IOTstack menu is good for getting started. My view is that, once you have an initial set of containers, you're better off studying how it all hangs together and, thereafter, edit the compose file by hand to achieve what you want. I field quite a few queries from people where the root cause of the problem is expecting all-singing-and-dancing performance from the menu. It helps if you start from the presumption that the menu is as dumb as a post.
The screen shot is my primary Pi. Notice that up time. Pretty much the only time I reboot the sucker is if an apt upgrade says "you should reboot". The 5 days on the containers is the by-product of an apt upgrade to container.io. The 5 hours on InfluxDB is the result of a daily cron job sending the container a restart message (I think the container has a memory leak and that's my way of keeping it under control).
Paraphraser
commented
1 year ago Yes - on Discord. See IOTstack Wiki. There's an invite link in the first set of dot points on that page.
Reboot after 04 script is usually 30 seconds (all my Pi4s). PiBuilder assumes SSH (headless). I have an HDMI screen that I use from time to time. It does go black on every reboot and I have to power-cycle the monitor. But that doesn't explain pings.
Will definitely help on Discord. I'm at UCT+11 so please keep that in mind (ie slow response usually means asleep).
Identifier
commented
1 year ago Just to end this thread: Good news! I got bridge mode to work on my newly arrived Raspberry Pi, and Steam Link works now too!
I installed ZeroTier directly on the Pi (not in a docker container), and followed the steps on https://zerotier.atlassian.net/wiki/spaces/SD/pages/193134593/Bridge+your+ZeroTier+and+local+network+with+a+RaspberryPi
But I was only half way there. I could communicate from my iPad (LTE) to my Raspberry Pi at home, but I couldn't access any LAN devices. I also tried the commands at the end of that guide (echo "0" > /proc/sys/net/bridge/bridge-nf-call-iptables and iptables -A FORWARD -p all -i br0 -j ACCEPT), but it wasn't enough. It turns out that since I had installed Docker, it had added additional rules in iptables:
identifier@Pi:~ $ sudo iptables --list
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:3000
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhereSo I asked ChatGPT how to reset the rules to allow everything, and it told me to run the following:
identifier@Pi:~ $ sudo iptables -F
identifier@Pi:~ $ sudo iptables -P INPUT ACCEPT
identifier@Pi:~ $ sudo iptables -P OUTPUT ACCEPT
identifier@Pi:~ $ sudo iptables -P FORWARD ACCEPTImmediately I could access all home LAN devices from my iPad, and Steam Link works too! 🥳
I probably messed up my docker somehow, but I don't care, at least I made it to the light at the end of the tunnel!
Next steps would be for someone to add a zyclonite/zerotier:bridge version that has all of this configured already 😁
Paraphraser
commented
1 year ago Well, on the basis of "you break it, you buy it", the general idea is that myffic someone would be you. 😎
Although it's aimed at helping people prepare pull requests for IOTstack, this gist plus a bit of search+replace is generally applicable to any GitHub repo.
Identifier
commented
1 year ago One last follow up here for anyone else trying to accomplish the same thing. I noticed docker messes with the iptables again after every reboot, so those changes were lost. I fixed this by adding these lines to my /etc/rc.local file:
printf "Waiting for docker to start before setting default IPTABLES FORWARD policy to ACCEPT"
sleep 10
iptables -P FORWARD ACCEPTI don't know how exactly to know when docker has finished its abuse of the iptables, but 10 seconds seemed sufficient on my Raspberry Pi 4B
Paraphraser
commented
1 year ago I might be wrong but my understanding of rc.local is the system isn't ready for business until that completes. Rather than hold up the wheels of progress, I think I'd write a script that did the sleep and iptables mod (maybe using logger to get the message into the log) and then have rc.local launch that script in the background.
That said, I don't think rc.local is the place for this (directly or indirectly). Most of the iteration in this PR was about getting the entry-point script "right" - the goal being to add iptables rules as the "router" container came up, then remove them again as the container came down. I think that should be the goal here too - a container should be self-contained, not depend on externalities it can't control. If you need help building a test environment with a local Dockerfile, the basics of what I did for this PR are in #10 - anything else, please ask me on Discord.
I'm not sure I'd describe what Docker does as "abuse" **. Sure, Docker adds a whole bunch of rules and they look a bit messy when you list them out. I'm not going to claim that I understand exactly what they do because I've never taken the time to study them carefully. I just observe that they seem to be necessary and leave the design and aesthetics up to people who clearly know one heck of a lot more than I do.
As to when Docker "finishes", I don't think it ever does. I've seen rules come and go each time a container comes up or is taken down.
* dem's fightin' words to those ordinarily peace-lovin' dudes livin' over on the Docker Ranch*
Enhancements to router (formerly bridge) functionality
Rename "bridge" to "router"
Rename files with "bridge" in the name to "router".
Edit file contents referring to "bridge" so they refer to "router".
point to the correct link in the ZeroTier documentation (existing link points to "bridge" but the actual implementation with iptables rules is that of the "router").
The problem I'm foreseeing is what happens if there's a need to implement an actual bridge variant?
timezone support
tzdatato Dockerfile.router activatesTZenvironment variable. Avoids need for mental gymnastics when examining logs.support environment variables:
PUID+PGID. Default to 999 and 994, respectively, mimicking what happens on a "native" install of ZeroTier-One on a Raspberry Pi. Only used to perform unconditional reset of ownership (PUID:PGID) throughout persistent store on each launch. Self-repair mitigates permission problems that can sometimes occur in docker environments, which can send containers into restart loops.ZEROTIER_ONE_LOCAL_PHYS. Allows for overriding towlan0(eg Raspberry Pi Zero 2W), or botheth0 wlan0to support multiple subnets or failover modes, or similar situations. Defaults toeth0to maintain backwards compatibility.ZEROTIER_ONE_USE_IPTABLES_NFT. Iftrue, substitutesiptables-nft. This definitely seems to be necessary on Raspberry Pi running Bullseye. Default isfalsewhich usesiptablesand maintains backwards compatibility.ZEROTIER_ONE_NETWORK_IDS. Supports passing one or more default network IDs. Intended as an alternative to waiting for a container to initialise and then issuing one or morejoincommands. Also means container can fail safe if its persistent storage is erased (will look like a new identity but can be authorised in ZeroTier Central and will then be reachable for additional configuration).improve router entry-point script:
basic problem is iptables rules do not get cleaned-up when container quits. This can lead to multiple duplicate rules.
launches ZeroTier daemon as a detached (nohup) process, with an asynchronous tail as an intermediary to bring any stdout/stderr messaging into PID=1.
traps INT, TERM and HUP to run termination handler to clean up iptables rules (TERM is the common case).
suspends on ZeroTier daemon then runs termination handler on normal exit.
tested correct handling of:
docker commands to restart, terminate, recreate, stack down, etc.
reboots while the container is running (assuming "unless stopped" in force).
external action (eg user killall) clobbering the ZeroTier daemon.
add example docker-compose service definition for router variant.
refactor documentation to separate zerotier-router into its own readme.