search

TheNewNormal / corectl

CoreOS over macOS made very simple
Apache License 2.0
286 stars 21 forks source link

port redirects broken when corectld is running corectld #105

Open vitaly opened 7 years ago

vitaly commented 7 years ago

I have a custom anchor for my port redirects (port 80 to mehserve), but once corectld is running those rules disappear.

I have a script to print PF rules: https://gist.github.com/vitaly/cd0024d232a3a134b00e5496e4e779b8

here is the output w/o corectld: https://gist.github.com/vitaly/10b9f4cd467370c226daf78424f02816 here is the output with corectld: https://gist.github.com/vitaly/b40d230cc72b88fbcb7c1ade87a12163

The difference is this:

--- 1   2016-11-30 15:10:05.000000000 +0100
+++ 2   2016-11-30 15:10:27.000000000 +0100
@@ -1,9 +1,6 @@
-                                        r     scrub-anchor "com.apple/*" all fragment reassemble
-                                        r     anchor "com.apple/*" all
-                                        n     nat-anchor "com.apple/*" all
-                                        n     nat-anchor "custom" all
-                                        n     rdr-anchor "com.apple/*" all
-                                        n     rdr-anchor "custom" all
+                                        n     nat-anchor "com.apple.internet-sharing" all
+                                        n     rdr-anchor "corectl-dns-forwarding" all
+                                        n     rdr-anchor "com.apple.internet-sharing" all
                                         A       com.apple
                                         A       com.apple.internet-sharing
                                         A       corectl-dns-forwarding

Note that not only my 'custom' anchors are skipped but the default apple ones as well.

vitaly commented 7 years ago

I don't know go, so I'd rather not try to change the code (for now ;)). but I think I do know what needs to be done in terms of PF.

right now it does pfctl -F pfC_file which in turn loads pfR_file. pfC declares 2 anchors: corectl-dns-forwarding, and com.apple.internet-sharing, and then loads corectl-dns-forwarding from pfR. The problem is that it clears the default "root" config (e.g. all com.apple/* anchors).

but default config already uses rdr rules from com.apple/*, so instead we can just load pfR directly into this namespace:

pfctl -a com.apple/corectl-dns-forwarding -f pfR_file

and to remove it:

pfctl -a com.apple/corectl-dns-forwarding -Fa

AntonioMeireles commented 7 years ago

It'a a bug, my fault - you're spot on. I' m on the road ATM, will fix it tomorrow. And many, many thanks!

vitaly commented 7 years ago

@AntonioMeireles it looks like kube-solo also messes up with it. Not sure if directly or through corectl though.

AntonioMeireles commented 7 years ago

@vitaly

Hi again, could you please fetch latest tip, built it, and confirm that it fixes this for you ? Thanks in advance!

vitaly commented 7 years ago

@AntonioMeireles please see my comment on https://github.com/TheNewNormal/corectl/issues/106

couldn't build it, not a golang user.

vitaly commented 7 years ago

OK, so I finally got it to compile using hints from /Applications/corectl.app/Contents/Resources/get_go_binaries/corectl_compile_from_source_corectl.sh

DNS port configuration seems to be working, but PF rules are still messed up. Actually it seems to be worse, as I can't even connect outside from a VM.

Note: I'm again using my pf rules dump script, as it produces a much shorter output and much easier to diff. You can see the script here: https://gist.github.com/vitaly/cd0024d232a3a134b00e5496e4e779b8

rules after firewall reset:

                                        r     scrub-anchor "com.apple/*" all fragment reassemble
                                        r     anchor "com.apple/*" all
                                        n     nat-anchor "com.apple/*" all
                                        n     rdr-anchor "com.apple/*" all
                                        A       com.apple
                                        A       com.apple.internet-sharing
com.apple                               r     anchor "200.AirDrop/*" all
com.apple                               r     anchor "250.ApplicationFirewall/*" all
com.apple                               A       com.apple/200.AirDrop
com.apple                               A       com.apple/250.ApplicationFirewall
com.apple                               A       com.apple/mehserve
com.apple/200.AirDrop                   A       com.apple/200.AirDrop/Bonjour
com.apple/200.AirDrop/Bonjour           r     pass in on p2p0 inet6 proto udp from any to any port = 5353 keep state
com.apple/200.AirDrop/Bonjour           r     pass out on p2p0 proto tcp all flags any keep state
com.apple/250.ApplicationFirewall       r     scrub in all fragment reassemble
com.apple/250.ApplicationFirewall       r     block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall       r     block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall       r     block drop in inet6 proto ipv6-icmp all icmp6-type echoreq
com.apple/mehserve                      n     rdr pass inet proto tcp from any to 127.0.0.1 port = 80 -> 127.0.0.1 port 12439

rules after corectld started:

                                        r     scrub-anchor "com.apple/*" all fragment reassemble
                                        r     anchor "com.apple/*" all
                                        n     nat-anchor "com.apple/*" all
                                        n     rdr-anchor "com.apple/*" all
                                        A       com.apple
                                        A       com.apple.internet-sharing
com.apple                               r     anchor "200.AirDrop/*" all
com.apple                               r     anchor "250.ApplicationFirewall/*" all
com.apple                               A       com.apple/200.AirDrop
com.apple                               A       com.apple/250.ApplicationFirewall
com.apple                               A       com.apple/corectl-dns-forwarding
com.apple                               A       com.apple/mehserve
com.apple/200.AirDrop                   A       com.apple/200.AirDrop/Bonjour
com.apple/200.AirDrop/Bonjour           r     pass in on p2p0 inet6 proto udp from any to any port = 5353 keep state
com.apple/200.AirDrop/Bonjour           r     pass out on p2p0 proto tcp all flags any keep state
com.apple/250.ApplicationFirewall       r     scrub in all fragment reassemble
com.apple/250.ApplicationFirewall       r     block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall       r     block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall       r     block drop in inet6 proto ipv6-icmp all icmp6-type echoreq
com.apple/corectl-dns-forwarding        n     nat-anchor "com.apple.internet-sharing" all
com.apple/corectl-dns-forwarding        n     rdr-anchor "corectl-dns-forwarding" all
com.apple/corectl-dns-forwarding        n     rdr-anchor "com.apple.internet-sharing" all
com.apple/corectl-dns-forwarding        A       com.apple/corectl-dns-forwarding/com.apple.internet-sharing
com.apple/corectl-dns-forwarding        A       com.apple/corectl-dns-forwarding/corectl-dns-forwarding
com.apple/mehserve                      n     rdr pass inet proto tcp from any to 127.0.0.1 port = 80 -> 127.0.0.1 port 12439

here is the diff:

--- 1-after-reset   2016-12-02 03:49:00.000000000 +0100
+++ 2-after-corectld-started    2016-12-02 03:50:50.000000000 +0100
@@ -9,6 +9,7 @@
 com.apple                               r     anchor "250.ApplicationFirewall/*" all
 com.apple                               A       com.apple/200.AirDrop
 com.apple                               A       com.apple/250.ApplicationFirewall
+com.apple                               A       com.apple/corectl-dns-forwarding
 com.apple                               A       com.apple/mehserve
 com.apple/200.AirDrop                   A       com.apple/200.AirDrop/Bonjour
 com.apple/200.AirDrop/Bonjour           r     pass in on p2p0 inet6 proto udp from any to any port = 5353 keep state
@@ -17,4 +18,9 @@
 com.apple/250.ApplicationFirewall       r     block drop in inet proto icmp all icmp-type echoreq
 com.apple/250.ApplicationFirewall       r     block drop in inet proto icmp all icmp-type echoreq
 com.apple/250.ApplicationFirewall       r     block drop in inet6 proto ipv6-icmp all icmp6-type echoreq
+com.apple/corectl-dns-forwarding        n     nat-anchor "com.apple.internet-sharing" all
+com.apple/corectl-dns-forwarding        n     rdr-anchor "corectl-dns-forwarding" all
+com.apple/corectl-dns-forwarding        n     rdr-anchor "com.apple.internet-sharing" all
+com.apple/corectl-dns-forwarding        A       com.apple/corectl-dns-forwarding/com.apple.internet-sharing
+com.apple/corectl-dns-forwarding        A       com.apple/corectl-dns-forwarding/corectl-dns-forwarding
 com.apple/mehserve                      n     rdr pass inet proto tcp from any to 127.0.0.1 port = 80 -> 127.0.0.1 port 12439

Note the weird anchors: com.apple/corectl-dns-forwarding/com.apple.internet-sharing, and com.apple/corectl-dns-forwarding/corectl-dns-forwarding.

Once you properly nest your rules inside com.apple/*, there is no need to declare the anchors, just the rules.

Since at this stage you don't yet add any commands, I think instead of adding nat-anchor and rdr-anchor you only need to reset them (Also note the renamed internet-sharing anchor to properly nest it inside com.apple/):

sudo pfctl -a com.apple/corectl-dns-forwarding -Fa
sudo pfctl -a com.apple/internet-sharing -Fa

But at least at this point my mehserve is still working.

Now, here is how it looks once I run a VM with it:

                                        r     scrub-anchor "com.apple/*" all fragment reassemble
                                        r     scrub-anchor "com.apple.internet-sharing" all fragment reassemble
                                        r     anchor "com.apple/*" all
                                        r     anchor "com.apple.internet-sharing" all
                                        n     nat-anchor "com.apple/*" all
                                        n     nat-anchor "com.apple.internet-sharing" all
                                        n     rdr-anchor "com.apple/*" all
                                        n     rdr-anchor "com.apple.internet-sharing" all
                                        A       com.apple
                                        A       com.apple.internet-sharing
com.apple                               r     anchor "200.AirDrop/*" all
com.apple                               r     anchor "250.ApplicationFirewall/*" all
com.apple                               A       com.apple/200.AirDrop
com.apple                               A       com.apple/250.ApplicationFirewall
com.apple                               A       com.apple/corectl-dns-forwarding
com.apple                               A       com.apple/mehserve
com.apple/200.AirDrop                   A       com.apple/200.AirDrop/Bonjour
com.apple/200.AirDrop/Bonjour           r     pass in on p2p0 inet6 proto udp from any to any port = 5353 keep state
com.apple/200.AirDrop/Bonjour           r     pass out on p2p0 proto tcp all flags any keep state
com.apple/250.ApplicationFirewall       r     scrub in all fragment reassemble
com.apple/250.ApplicationFirewall       r     block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall       r     block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall       r     block drop in inet6 proto ipv6-icmp all icmp6-type echoreq
com.apple/corectl-dns-forwarding        n     nat-anchor "com.apple.internet-sharing" all
com.apple/corectl-dns-forwarding        n     rdr-anchor "corectl-dns-forwarding" all
com.apple/corectl-dns-forwarding        n     rdr-anchor "com.apple.internet-sharing" all
com.apple/corectl-dns-forwarding        A       com.apple/corectl-dns-forwarding/com.apple.internet-sharing
com.apple/corectl-dns-forwarding        A       com.apple/corectl-dns-forwarding/corectl-dns-forwarding
com.apple/mehserve                      n     rdr pass inet proto tcp from any to 127.0.0.1 port = 80 -> 127.0.0.1 port 12439
com.apple.internet-sharing              r     scrub-anchor "shared_v4" all fragment reassemble
com.apple.internet-sharing              r     anchor "shared_v4" all
com.apple.internet-sharing              n     nat-anchor "shared_v4" all
com.apple.internet-sharing              n     rdr-anchor "shared_v4" all
com.apple.internet-sharing              A       com.apple.internet-sharing/shared_v4
com.apple.internet-sharing/shared_v4    r     scrub on en0 all no-df fragment reassemble
com.apple.internet-sharing/shared_v4    r     scrub on bridge100 all no-df max-mss 1460 fragment reassemble
com.apple.internet-sharing/shared_v4    r     scrub on bridge100 proto esp all no-df fragment reassemble
com.apple.internet-sharing/shared_v4    r     pass on en0 all flags any keep state
com.apple.internet-sharing/shared_v4    r     pass on en0 proto esp all no state
com.apple.internet-sharing/shared_v4    n     nat on en0 inet from 192.168.64.0/24 to any -> (en0:0) extfilter ei
com.apple.internet-sharing/shared_v4    n     no nat on bridge100 inet from 192.168.64.1 to 192.168.64.0/24

here is the diff:

--- 2-after-corectld-started    2016-12-02 03:50:50.000000000 +0100
+++ 3-after-corectl-run 2016-12-02 03:53:11.000000000 +0100
@@ -1,8 +1,12 @@
 ./pfdump
                                         r     scrub-anchor "com.apple/*" all fragment reassemble
+                                        r     scrub-anchor "com.apple.internet-sharing" all fragment reassemble
                                         r     anchor "com.apple/*" all
+                                        r     anchor "com.apple.internet-sharing" all
                                         n     nat-anchor "com.apple/*" all
+                                        n     nat-anchor "com.apple.internet-sharing" all
                                         n     rdr-anchor "com.apple/*" all
+                                        n     rdr-anchor "com.apple.internet-sharing" all
                                         A       com.apple
                                         A       com.apple.internet-sharing
 com.apple                               r     anchor "200.AirDrop/*" all
@@ -24,3 +28,15 @@
 com.apple/corectl-dns-forwarding        A       com.apple/corectl-dns-forwarding/com.apple.internet-sharing
 com.apple/corectl-dns-forwarding        A       com.apple/corectl-dns-forwarding/corectl-dns-forwarding
 com.apple/mehserve                      n     rdr pass inet proto tcp from any to 127.0.0.1 port = 80 -> 127.0.0.1 port 12439
+com.apple.internet-sharing              r     scrub-anchor "shared_v4" all fragment reassemble
+com.apple.internet-sharing              r     anchor "shared_v4" all
+com.apple.internet-sharing              n     nat-anchor "shared_v4" all
+com.apple.internet-sharing              n     rdr-anchor "shared_v4" all
+com.apple.internet-sharing              A       com.apple.internet-sharing/shared_v4
+com.apple.internet-sharing/shared_v4    r     scrub on en0 all no-df fragment reassemble
+com.apple.internet-sharing/shared_v4    r     scrub on bridge100 all no-df max-mss 1460 fragment reassemble
+com.apple.internet-sharing/shared_v4    r     scrub on bridge100 proto esp all no-df fragment reassemble
+com.apple.internet-sharing/shared_v4    r     pass on en0 all flags any keep state
+com.apple.internet-sharing/shared_v4    r     pass on en0 proto esp all no state
+com.apple.internet-sharing/shared_v4    n     nat on en0 inet from 192.168.64.0/24 to any -> (en0:0) extfilter ei
+com.apple.internet-sharing/shared_v4    n     no nat on bridge100 inet from 192.168.64.1 to 192.168.64.0/24

Seems like it is rewriting the global scope again, which is not needed if you just nest the rules inside com.apple/.

At this stage when I corectl ssh into the VM I can't even ping 8.8.8.8, where previously I could ping it, just couldn't resolve any hostnames (due to DNS port conflict, the forwarding was working)

Note that the DNS rules are missing now. I tried adding them, but it didn't help (din't expect it to, it has nothing to do with ping)

I tried to mess with the rules for a while, but couldn't get it to work, so I reverted to the old version for now.

AntonioMeireles commented 7 years ago

@vitaly

sorry lag, can you try again against latest tip ? (and many thanks for your patience!)

vitaly commented 7 years ago

just tried the latest tip.

result:

PF dumps: https://gist.github.com/vitaly/fc0cb6986a2a3d92fb86b2239c8cf1cf

Looking at the current corectl pf config, I see that it was done outside of 'com.apple' on purpose ;). and, to my untrained eye it actually looks OK vs breaking other rules. i.e. I can't see how it can break my redirect, but it does ;(. On the other hand I also don't understand how your corectld-forwarding-dns anchor works. it is not referenced from root or any other anchors (btw, can you please explain this? ;)). it clearly does work though, because I do have DNS inside the VM, but I really can't figure out how it manages to do that. which means I have no idea how this PF thing actually works ;)

vitaly commented 7 years ago

I think I already mentioned this, but it might be worth pointing out again.

My redirect still works once corectld is started and stops working as soon as I run any VM

BUT

it still doesn't work even after I stop VM and corectd, even though PF dump looks exactly like before corectld was started

only once I run sudo pfctl -f /etc/pf.conf it starts working again, but I can not detect any difference in PF config before and after I do that. The only idea that I have is that its some kind of transient state that is getting cleared when I re–run the root scope config.

Hopefully this helps ... ;)

AntonioMeireles commented 7 years ago

interesting... can you paste your exact redirect rules and how you are loading them please... (and once again MANY thanks for your patience!)

genevera commented 7 years ago

Pinging as I have a similar problem. corectl and sshuttle also don't get along so well.

genevera commented 7 years ago

@vitaly and @AntonioMeireles maybe pfctl needs to be called with -E, the token stored somewhere, then torn down with -X + that token? That looks like that might solve this particular problem.