search

sshuttle / sshuttle

Transparent proxy server that works as a poor man's VPN. Forwards over ssh. Doesn't require admin. Works with Linux and MacOS. Supports DNS tunneling.
GNU Lesser General Public License v2.1
11.61k stars 727 forks source link

Sshuttle not working in combination with WireGuard on macOS Ventura 13.4.1 #864

Open tbosnjak opened 1 year ago

tbosnjak commented 1 year ago

My setup is as fallows:

#wg.conf
[Interface]
PrivateKey = XXXXXXXXX
Address = 192.168.100.3/32
DNS = 192.168.100.1

[Peer]
PublicKey = XXXXXXXXXX
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = XXXXXXX:13231

sshuttle has no special configuration and it is in version:

# sshuttle --version                                                                                                                                                     17:09:10
1.1.1

I'm using kuberntes pod as bastion and using kuttle

The bastion pod image is alpine based with python version:

# python -V
Python 3.11.4

The command used to start sshuttle is:

sshuttle --dns -r ${sshuttlePo} --exclude 44.205.64.79/32 -e kuttle 44.0.0.0/8 10.250.0.0/16 10.251.0.0/16

Sudoers is set to allow starting of sshutlle without password.

I can confirm that WireGuard works as expected and sshuttle when there is no WireGurad works as expected. When I connect first to WireGurad and then start sshuttle, dns resolution starts to fail.

I did a quick WireShark check and found out that dns request goes throught the WG interface and dns reply gets back, but the reply doens't get back to lo0 interface.

# sudo pfctl -s all                                                                                                                                                  12:36:17
Password:
No ALTQ support in kernel
ALTQ related functions disabled
TRANSLATION RULES:
nat-anchor "com.apple/*" all
rdr-anchor "com.apple/*" all
rdr-anchor "sshuttle6-12300" all
rdr-anchor "sshuttle-12300" all

FILTER RULES:
scrub-anchor "com.apple/*" all fragment reassemble
anchor "com.apple/*" all
anchor "sshuttle6-12300" all
anchor "sshuttle-12300" all

DUMMYNET RULES:
dummynet-anchor "com.apple/*" all

STATES:
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:61931       MULTIPLE:MULTIPLE
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:64670       MULTIPLE:MULTIPLE
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:53038       MULTIPLE:MULTIPLE
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:55589       MULTIPLE:MULTIPLE
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:56887       MULTIPLE:MULTIPLE
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:53058       MULTIPLE:MULTIPLE
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:51126       MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:55009 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:55009       MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:55198 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:55198       MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:62635 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:62635       MULTIPLE:MULTIPLE
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:64880       MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:54363 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:54363       MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:51397 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:51397       MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:56766 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:56766       MULTIPLE:MULTIPLE
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:52968       MULTIPLE:MULTIPLE
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:58557       MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:56887 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 192.168.100.3:53058 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 192.168.100.3:51126 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 192.168.100.3:52968 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 192.168.100.3:58557 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:53229       MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:61931 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 192.168.100.3:64523 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:64523       MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:52969 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:52969       MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:64690 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:64690       MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:51493 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 127.0.0.1:12299 <- 192.168.100.1:53 <- 192.168.100.3:51493       MULTIPLE:MULTIPLE
ALL udp 192.168.100.3:64880 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 192.168.100.3:64670 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 192.168.100.3:53038 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC
ALL udp 192.168.100.3:55589 -> 192.168.100.1:53       SINGLE:NO_TRAFFIC

INFO:
Status: Enabled for 0 days 00:03:48           Debug: Urgent

State Table                          Total             Rate
  current entries                       41
  searches                         1060000         4649.1/s
  inserts                             6790           29.8/s
  removals                            6749           29.6/s
Counters
  match                             679939         2982.2/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             0            0.0/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                         0            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s
  dummynet                               0            0.0/s
  invalid-port                           0            0.0/s

TIMEOUTS:
tcp.first                   120s
tcp.opening                  30s
tcp.established           86400s
tcp.closing                 900s
tcp.finwait                  45s
tcp.closed                   90s
tcp.tsdiff                   30s
udp.first                    60s
udp.single                   30s
udp.multiple                 60s
icmp.first                   20s
icmp.error                   10s
grev1.first                 120s
grev1.initiating             30s
grev1.estblished           1800s
esp.first                   120s
esp.estblished              900s
other.first                  60s
other.single                 30s
other.multiple               60s
frag                         30s
interval                     10s
adaptive.start             6000 states
adaptive.end              12000 states
src.track                     0s

LIMITS:
states        hard limit    10000
app-states    hard limit    10000
src-nodes     hard limit    10000
frags         hard limit     5000
tables        hard limit     1000
table-entries hard limit   200000

TABLES:

OS FINGERPRINTS:
696 fingerprints loaded
#sudo netstat -nr                                                                                                                                               15s 10:20:03
Routing tables

Internet:
Destination        Gateway            Flags           Netif Expire
default            link#18            UCSg            utun3
default            172.20.10.1        UGScIg            en0
8.8.8.8            link#18            UHW3Ig          utun3     11
20.50.46.239       link#18            UHWIig          utun3
23.89.10.203       link#18            UHW3Ig          utun3     40
23.89.56.117       link#18            UHW3Ig          utun3     42
23.89.82.86        link#18            UHW3Ig          utun3     40
23.89.83.47        link#18            UHW3Ig          utun3      9
69.26.167.48       link#18            UHW3Ig          utun3      9
92.123.213.105     link#18            UHW3Ig          utun3    100
92.123.213.155     link#18            UHWIig          utun3
108.138.36.89      link#18            UHWIig          utun3
127                127.0.0.1          UCS               lo0
127.0.0.1          127.0.0.1          UH                lo0
142.251.5.188      link#18            UHWIig          utun3
169.254            link#11            UCS               en0      !
170.72.4.197       link#18            UHW3Ig          utun3     10
170.72.23.12       link#18            UHW3Ig          utun3     10
170.72.41.43       link#18            UHW3Ig          utun3     40
170.72.74.201      link#18            UHW3Ig          utun3     41
170.72.75.212      link#18            UHW3Ig          utun3     41
170.72.134.40      link#18            UHW3Ig          utun3     40
170.72.148.152     link#18            UHW3Ig          utun3     42
170.72.165.102     link#18            UHW3Ig          utun3     42
170.72.166.240     link#18            UHW3Ig          utun3     42
170.72.234.3       link#18            UHW3Ig          utun3    100
172.20.10/28       link#11            UCS               en0      !
172.20.10.1/32     link#11            UCS               en0      !
172.20.10.1        16:c8:8b:66:ca:64  UHLWIir           en0    363
172.20.10.3/32     link#11            UCS               en0      !
172.20.10.3        90:9c:4a:d0:1:76   UHLWI             lo0
192.168.100.1      link#18            UHWIig          utun3
192.168.100.3      192.168.100.3      UH              utun3
224.0.0/4          link#18            UmCS            utun3
224.0.0/4          link#11            UmCSI             en0      !
239.255.255.250    1:0:5e:7f:ff:fa    UHmLWI            en0
239.255.255.250    link#18            UHmW3I          utun3    129
255.255.255.255/32 link#18            UCS             utun3
255.255.255.255/32 link#11            UCSI              en0      !

Internet6:
Destination                             Gateway                         Flags           Netif Expire
default                                 fe80::14c8:8bff:fe66:ca64%en0   UGcIg             en0
default                                 fe80::%utun0                    UGcIg           utun0
default                                 fe80::%utun1                    UGcIg           utun1
default                                 fe80::%utun2                    UGcIg           utun2
::1                                     ::1                             UHL               lo0
2a01:599:a17:49fb::/64                  link#11                         UC                en0
2a01:599:a17:49fb:46a:fc92:d95d:cfe0    90:9c:4a:d0:1:76                UHL               lo0
2a01:599:a17:49fb:95c1:69c7:794d:30e6   16:c8:8b:66:ca:64               UHLWI             en0
2a01:599:a17:49fb:d17d:7a7c:2782:6da4   90:9c:4a:d0:1:76                UHL               lo0
fe80::%lo0/64                           fe80::1%lo0                     UcI               lo0
fe80::1%lo0                             link#1                          UHLI              lo0
fe80::%en5/64                           link#4                          UCI               en5
fe80::aede:48ff:fe00:1122%en5           ac:de:48:0:11:22                UHLI              lo0
fe80::aede:48ff:fe33:4455%en5           ac:de:48:33:44:55               UHLWIi            en5
fe80::%en0/64                           link#11                         UCI               en0
fe80::14c8:8bff:fe66:ca64%en0           16:c8:8b:66:ca:64               UHLWIir           en0
fe80::18d2:bd00:ffab:3ccb%en0           90:9c:4a:d0:1:76                UHLI              lo0
fe80::bcb2:a8ff:fe15:708d%awdl0         be:b2:a8:15:70:8d               UHLI              lo0
fe80::bcb2:a8ff:fe15:708d%llw0          be:b2:a8:15:70:8d               UHLI              lo0
fe80::%utun0/64                         fe80::99b:f4ad:44dd:eae2%utun0  UcI             utun0
fe80::99b:f4ad:44dd:eae2%utun0          link#14                         UHLI              lo0
fe80::%utun1/64                         fe80::e75c:af9d:6831:9791%utun1 UcI             utun1
fe80::e75c:af9d:6831:9791%utun1         link#15                         UHLI              lo0
fe80::%utun2/64                         fe80::ce81:b1c:bd2c:69e%utun2   UcI             utun2
fe80::ce81:b1c:bd2c:69e%utun2           link#16                         UHLI              lo0
ff00::/8                                ::1                             UmCI              lo0
ff00::/8                                link#4                          UmCI              en5
ff00::/8                                link#11                         UmCI              en0
ff00::/8                                link#12                         UmCI            awdl0
ff00::/8                                link#13                         UmCI             llw0
ff00::/8                                fe80::99b:f4ad:44dd:eae2%utun0  UmCI            utun0
ff00::/8                                fe80::e75c:af9d:6831:9791%utun1 UmCI            utun1
ff00::/8                                fe80::ce81:b1c:bd2c:69e%utun2   UmCI            utun2
ff01::%lo0/32                           ::1                             UmCI              lo0
ff01::%en5/32                           link#4                          UmCI              en5
ff01::%en0/32                           link#11                         UmCI              en0
ff01::%utun0/32                         fe80::99b:f4ad:44dd:eae2%utun0  UmCI            utun0
ff01::%utun1/32                         fe80::e75c:af9d:6831:9791%utun1 UmCI            utun1
ff01::%utun2/32                         fe80::ce81:b1c:bd2c:69e%utun2   UmCI            utun2
ff02::%lo0/32                           ::1                             UmCI              lo0
ff02::%en5/32                           link#4                          UmCI              en5
ff02::%en0/32                           link#11                         UmCI              en0
ff02::%utun0/32                         fe80::99b:f4ad:44dd:eae2%utun0  UmCI            utun0
ff02::%utun1/32                         fe80::e75c:af9d:6831:9791%utun1 UmCI            utun1
ff02::%utun2/32                         fe80::ce81:b1c:bd2c:69e%utun2   UmCI            utun2

Just to be transparent, I have no clue about MacOS networking stack.

I checked the following issues, without any success: https://github.com/sshuttle/sshuttle/issues/706 https://github.com/sshuttle/sshuttle/issues/563

I tried to apply the patch proposed in: https://github.com/azolotko/sshuttle/pull/1/files, no luck neither.

Any help will be appreciated.

MatthieuBarthel commented 5 months ago

I've just had a similar issue on Linux using wg-quick (I had no issue before with standard wireguard config with nmcli), I fixed it by not routing all my traffic through wireguard (just needed to specifiy in AllowedIPs the subnets I want to route, including the one where my sshuttle server is).