search

docker / for-mac

Bug reports for Docker Desktop for Mac
https://www.docker.com/products/docker#/mac
2.43k stars 117 forks source link

Broken TCP packets after version 4.22.0 #7360

Open petr-ujezdsky opened 3 months ago

petr-ujezdsky commented 3 months ago

Description

I am using wg-easy successfully inside Docker for Mac version 4.22.0. However newer versions of Docker for Mac break the TCP layer making wg-easy not working at all. I have tried the newer Docker for Mac version on both Intel and M3 MacBook with the same result.

Note: To install tcpdump and ethtool inside wg-easy container run apk update && apk add tcpdump ethtool.

Docker for Mac 4.22.0 (working)

client - another macbook

$ curl -vvv cheat.sh
<web page content>
$ ifconfig utun4
utun4: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1420
    options=6460<TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
    inet 10.0.4.2 --> 10.0.4.2 netmask 0xffffff00
$ sudo tcpdump -i utun4 -v | grep cheat
tcpdump: listening on utun4, link-type RAW (Raw IP), snapshot length 524288 bytes
...
    10.0.4.2.53066 > cheat.sh.http: Flags [S], cksum 0x52bf (correct), seq 3148994044, win 65535, options [mss 1380,nop,wscale 6,nop,nop,TS val 3063072233 ecr 0,sackOK,eol], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [S.], cksum 0x1fd9 (correct), seq 4170317243, ack 3148994045, win 65504, options [mss 65495,nop,nop,TS val 1634596043 ecr 3063072233,nop,wscale 5], length 0
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0x3da0 (correct), ack 1, win 2052, options [nop,nop,TS val 3063072236 ecr 1634596043], length 0
    10.0.4.2.53066 > cheat.sh.http: Flags [P.], cksum 0x99f9 (correct), seq 1:72, ack 1, win 2052, options [nop,nop,TS val 3063072236 ecr 1634596043], length 71: HTTP, length: 71
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0x3d5f (correct), ack 72, win 2044, options [nop,nop,TS val 1634596045 ecr 3063072236], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0x29bd (correct), seq 1:1369, ack 72, win 16384, options [nop,nop,TS val 1634596122 ecr 3063072236], length 1368: HTTP, length: 1368
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0xa974 (correct), seq 1369:2737, ack 72, win 16384, options [nop,nop,TS val 1634596123 ecr 3063072236], length 1368: HTTP
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0x3778 (correct), ack 1369, win 2030, options [nop,nop,TS val 3063072316 ecr 1634596122], length 0
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0x3234 (correct), ack 2737, win 2009, options [nop,nop,TS val 3063072316 ecr 1634596123], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0x390c (correct), seq 2737:4105, ack 72, win 16384, options [nop,nop,TS val 1634596123 ecr 3063072236], length 1368: HTTP
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0x5a0b (correct), seq 4105:5473, ack 72, win 16384, options [nop,nop,TS val 1634596123 ecr 3063072236], length 1368: HTTP
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0x2788 (correct), ack 5473, win 2005, options [nop,nop,TS val 3063072316 ecr 1634596123], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0x691b (correct), seq 5473:6841, ack 72, win 16384, options [nop,nop,TS val 1634596123 ecr 3063072236], length 1368: HTTP
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0x2246 (correct), ack 6841, win 1983, options [nop,nop,TS val 3063072316 ecr 1634596123], length 0
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0x2205 (correct), ack 6841, win 2048, options [nop,nop,TS val 3063072316 ecr 1634596123], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0x8f40 (correct), seq 6841:8209, ack 72, win 16384, options [nop,nop,TS val 1634596123 ecr 3063072236], length 1368: HTTP
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0x1cc3 (correct), ack 8209, win 2026, options [nop,nop,TS val 3063072316 ecr 1634596123], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0xad47 (correct), seq 8209:9577, ack 72, win 16384, options [nop,nop,TS val 1634596123 ecr 3063072236], length 1368: HTTP
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0x176b (correct), ack 9577, win 2026, options [nop,nop,TS val 3063072316 ecr 1634596123], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0xc67f (correct), seq 9577:10945, ack 72, win 16384, options [nop,nop,TS val 1634596123 ecr 3063072236], length 1368: HTTP
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0x1228 (correct), ack 10945, win 2005, options [nop,nop,TS val 3063072316 ecr 1634596123], length 0
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0x11fd (correct), ack 10945, win 2048, options [nop,nop,TS val 3063072316 ecr 1634596123], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0x3fc5 (correct), seq 10945:12313, ack 72, win 16384, options [nop,nop,TS val 1634596123 ecr 3063072236], length 1368: HTTP
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0x0cba (correct), ack 12313, win 2026, options [nop,nop,TS val 3063072317 ecr 1634596123], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0x6947 (correct), seq 12313:13681, ack 72, win 16384, options [nop,nop,TS val 1634596123 ecr 3063072236], length 1368: HTTP
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0x0762 (correct), ack 13681, win 2026, options [nop,nop,TS val 3063072317 ecr 1634596123], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [P.], cksum 0xd6b4 (correct), seq 13681:14481, ack 72, win 16384, options [nop,nop,TS val 1634596127 ecr 3063072316], length 800: HTTP
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0x0432 (correct), ack 14481, win 2035, options [nop,nop,TS val 3063072320 ecr 1634596127], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0x78ad (correct), seq 14481:15849, ack 72, win 16384, options [nop,nop,TS val 1634596135 ecr 3063072320], length 1368: HTTP
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0xfed3 (correct), ack 15849, win 2026, options [nop,nop,TS val 3063072327 ecr 1634596135], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0xc5ab (correct), seq 15849:17217, ack 72, win 16384, options [nop,nop,TS val 1634596135 ecr 3063072320], length 1368: HTTP
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0xf97b (correct), ack 17217, win 2026, options [nop,nop,TS val 3063072327 ecr 1634596135], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0xacc9 (correct), seq 17217:18585, ack 72, win 16384, options [nop,nop,TS val 1634596135 ecr 3063072320], length 1368: HTTP
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0xf423 (correct), ack 18585, win 2026, options [nop,nop,TS val 3063072327 ecr 1634596135], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0x4b8c (correct), seq 18585:19953, ack 72, win 16384, options [nop,nop,TS val 1634596135 ecr 3063072320], length 1368: HTTP
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0xeecb (correct), ack 19953, win 2026, options [nop,nop,TS val 3063072327 ecr 1634596135], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0x3190 (correct), seq 19953:21321, ack 72, win 16384, options [nop,nop,TS val 1634596135 ecr 3063072320], length 1368: HTTP
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0xe973 (correct), ack 21321, win 2026, options [nop,nop,TS val 3063072327 ecr 1634596135], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0xd516 (correct), seq 21321:22689, ack 72, win 16384, options [nop,nop,TS val 1634596135 ecr 3063072320], length 1368: HTTP
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0xc318 (correct), seq 22689:24057, ack 72, win 16384, options [nop,nop,TS val 1634596135 ecr 3063072320], length 1368: HTTP
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0xe405 (correct), ack 22689, win 2048, options [nop,nop,TS val 3063072327 ecr 1634596135], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [P.], cksum 0xeb37 (correct), seq 24057:24617, ack 72, win 16384, options [nop,nop,TS val 1634596135 ecr 3063072320], length 560: HTTP
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0xdc9c (correct), ack 24617, win 2017, options [nop,nop,TS val 3063072327 ecr 1634596135], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [P.], cksum 0x0630 (correct), seq 24617:25785, ack 72, win 16384, options [nop,nop,TS val 1634596139 ecr 3063072327], length 1168: HTTP
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0xd7f7 (correct), ack 25785, win 2029, options [nop,nop,TS val 3063072332 ecr 1634596139], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [F.], cksum 0x9fe8 (correct), seq 25785, ack 72, win 16384, options [nop,nop,TS val 1634596139 ecr 3063072327], length 0
    10.0.4.2.53066 > cheat.sh.http: Flags [.], cksum 0xd7e3 (correct), ack 25786, win 2048, options [nop,nop,TS val 3063072332 ecr 1634596139], length 0
    10.0.4.2.53066 > cheat.sh.http: Flags [F.], cksum 0xd7e2 (correct), seq 72, ack 25786, win 2048, options [nop,nop,TS val 3063072332 ecr 1634596139], length 0
    cheat.sh.http > 10.0.4.2.53066: Flags [.], cksum 0x9fe0 (correct), ack 73, win 16383, options [nop,nop,TS val 1634596142 ecr 3063072332], length 0

server - wg-easy container

# ifconfig wg0
wg0       Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:10.0.4.1  P-t-P:10.0.4.1  Mask:255.255.255.0
          UP POINTOPOINT RUNNING NOARP  MTU:1420  Metric:1
          RX packets:166629 errors:4118 dropped:0 overruns:0 frame:4118
          TX packets:461695 errors:0 dropped:1518 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:65040452 (62.0 MiB)  TX bytes:535859376 (511.0 MiB)
# tcpdump -i eth0 -v | grep cheat
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
...
    83957e66aad1.53066 > cheat.sh.80: Flags [S], cksum 0x51bf (correct), seq 3148994044, win 65535, options [mss 1380,nop,wscale 6,nop,nop,TS val 3063072233 ecr 0,sackOK,eol], length 0
    cheat.sh.80 > 83957e66aad1.53066: Flags [S.], cksum 0x1ed9 (correct), seq 4170317243, ack 3148994045, win 65504, options [mss 65495,nop,nop,TS val 1634596043 ecr 3063072233,nop,wscale 5], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0x3ca0 (correct), ack 1, win 2052, options [nop,nop,TS val 3063072236 ecr 1634596043], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [P.], cksum 0x98f9 (correct), seq 1:72, ack 1, win 2052, options [nop,nop,TS val 3063072236 ecr 1634596043], length 71: HTTP, length: 71
    cheat.sh.80 > 83957e66aad1.53066: Flags [.], cksum 0x3c5f (correct), ack 72, win 2044, options [nop,nop,TS val 1634596045 ecr 3063072236], length 0
    cheat.sh.80 > 83957e66aad1.53066: Flags [.], cksum 0x28bd (correct), seq 1:1369, ack 72, win 16384, options [nop,nop,TS val 1634596122 ecr 3063072236], length 1368: HTTP, length: 1368
    cheat.sh.80 > 83957e66aad1.53066: Flags [.], cksum 0x3806 (incorrect -> 0x1375), seq 1369:13681, ack 72, win 16384, options [nop,nop,TS val 1634596123 ecr 3063072236], length 12312: HTTP
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0x3678 (correct), ack 1369, win 2030, options [nop,nop,TS val 3063072316 ecr 1634596122], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0x3134 (correct), ack 2737, win 2009, options [nop,nop,TS val 3063072316 ecr 1634596123], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0x2688 (correct), ack 5473, win 2005, options [nop,nop,TS val 3063072316 ecr 1634596123], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0x2146 (correct), ack 6841, win 1983, options [nop,nop,TS val 3063072316 ecr 1634596123], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0x2105 (correct), ack 6841, win 2048, options [nop,nop,TS val 3063072316 ecr 1634596123], length 0
    cheat.sh.80 > 83957e66aad1.53066: Flags [P.], cksum 0xd5b4 (correct), seq 13681:14481, ack 72, win 16384, options [nop,nop,TS val 1634596127 ecr 3063072316], length 800: HTTP
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0x1bc3 (correct), ack 8209, win 2026, options [nop,nop,TS val 3063072316 ecr 1634596123], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0x166b (correct), ack 9577, win 2026, options [nop,nop,TS val 3063072316 ecr 1634596123], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0x1128 (correct), ack 10945, win 2005, options [nop,nop,TS val 3063072316 ecr 1634596123], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0x10fd (correct), ack 10945, win 2048, options [nop,nop,TS val 3063072316 ecr 1634596123], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0x0bba (correct), ack 12313, win 2026, options [nop,nop,TS val 3063072317 ecr 1634596123], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0x0662 (correct), ack 13681, win 2026, options [nop,nop,TS val 3063072317 ecr 1634596123], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0x0332 (correct), ack 14481, win 2035, options [nop,nop,TS val 3063072320 ecr 1634596127], length 0
    cheat.sh.80 > 83957e66aad1.53066: Flags [.], cksum 0x129e (incorrect -> 0x7694), seq 14481:17217, ack 72, win 16384, options [nop,nop,TS val 1634596135 ecr 3063072320], length 2736: HTTP
    cheat.sh.80 > 83957e66aad1.53066: Flags [P.], cksum 0x24d6 (incorrect -> 0x3556), seq 17217:24617, ack 72, win 16384, options [nop,nop,TS val 1634596135 ecr 3063072320], length 7400: HTTP
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0xfdd3 (correct), ack 15849, win 2026, options [nop,nop,TS val 3063072327 ecr 1634596135], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0xf87b (correct), ack 17217, win 2026, options [nop,nop,TS val 3063072327 ecr 1634596135], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0xf323 (correct), ack 18585, win 2026, options [nop,nop,TS val 3063072327 ecr 1634596135], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0xedcb (correct), ack 19953, win 2026, options [nop,nop,TS val 3063072327 ecr 1634596135], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0xe873 (correct), ack 21321, win 2026, options [nop,nop,TS val 3063072327 ecr 1634596135], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0xe305 (correct), ack 22689, win 2048, options [nop,nop,TS val 3063072327 ecr 1634596135], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0xdb9c (correct), ack 24617, win 2017, options [nop,nop,TS val 3063072327 ecr 1634596135], length 0
    cheat.sh.80 > 83957e66aad1.53066: Flags [P.], cksum 0x0530 (correct), seq 24617:25785, ack 72, win 16384, options [nop,nop,TS val 1634596139 ecr 3063072327], length 1168: HTTP
    cheat.sh.80 > 83957e66aad1.53066: Flags [F.], cksum 0x9ee8 (correct), seq 25785, ack 72, win 16384, options [nop,nop,TS val 1634596139 ecr 3063072327], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0xd6f7 (correct), ack 25785, win 2029, options [nop,nop,TS val 3063072332 ecr 1634596139], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [.], cksum 0xd6e3 (correct), ack 25786, win 2048, options [nop,nop,TS val 3063072332 ecr 1634596139], length 0
    83957e66aad1.53066 > cheat.sh.80: Flags [F.], cksum 0xd6e2 (correct), seq 72, ack 25786, win 2048, options [nop,nop,TS val 3063072332 ecr 1634596139], length 0
    cheat.sh.80 > 83957e66aad1.53066: Flags [.], cksum 0x9ee0 (correct), ack 73, win 16383, options [nop,nop,TS val 1634596142 ecr 3063072332], length 0

TCP is successfully established in the beginning - S -> S. -> . (3-way handshake, see documentation). Then follows the web page data.

Docker for Mac 4.32.0 (broken)

client - another macbook

$ curl -vvv cheat.sh
* Host cheat.sh:80 was resolved.
* IPv6: (none)
* IPv4: 5.9.243.188
*   Trying 5.9.243.188:80...
* connect to 5.9.243.188 port 80 from 10.0.4.2 port 55544 failed: Operation timed out
* Failed to connect to cheat.sh port 80 after 75011 ms: Couldn't connect to server
* Closing connection
curl: (28) Failed to connect to cheat.sh port 80 after 75011 ms: Couldn't connect to server
$ ifconfig utun4
utun4: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1420
    options=6460<TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
    inet 10.0.4.2 --> 10.0.4.2 netmask 0xffffff00
$ sudo tcpdump -i utun4 -v | grep cheat
tcpdump: listening on utun4, link-type RAW (Raw IP), snapshot length 524288 bytes
...
    10.0.4.2.52544 > cheat.sh.http: Flags [SEW], cksum 0x3e35 (correct), seq 2571259145, win 65535, options [mss 1380,nop,wscale 6,nop,nop,TS val 1537466895 ecr 0,sackOK,eol], length 0
    cheat.sh.http > 10.0.4.2.52544: Flags [S.], cksum 0xf3a9 (incorrect -> 0x43d3), seq 2401073651, ack 2571259146, win 65408, options [mss 65495,nop,nop,TS val 1650895273 ecr 1537466895,nop,wscale 7], length 0
    cheat.sh.http > 10.0.4.2.52544: Flags [S.], cksum 0xf3a9 (incorrect -> 0x43d3), seq 2401073651, ack 2571259146, win 65408, options [mss 65495,nop,nop,TS val 1650895273 ecr 1537466895,nop,wscale 7], length 0
    cheat.sh.http > 10.0.4.2.52544: Flags [S.], cksum 0xf3a9 (incorrect -> 0x43d3), seq 2401073651, ack 2571259146, win 65408, options [mss 65495,nop,nop,TS val 1650895273 ecr 1537466895,nop,wscale 7], length 0
    cheat.sh.http > 10.0.4.2.52544: Flags [S.], cksum 0xf3a9 (incorrect -> 0x43d3), seq 2401073651, ack 2571259146, win 65408, options [mss 65495,nop,nop,TS val 1650895273 ecr 1537466895,nop,wscale 7], length 0
    cheat.sh.http > 10.0.4.2.52315: Flags [S.], cksum 0xf3a9 (incorrect -> 0x09bc), seq 4203615612, ack 1431375901, win 65408, options [mss 65495,nop,nop,TS val 1650844888 ecr 4173577889,nop,wscale 7], length 0
    cheat.sh.http > 10.0.4.2.52544: Flags [S.], cksum 0xf3a9 (incorrect -> 0x43d3), seq 2401073651, ack 2571259146, win 65408, options [mss 65495,nop,nop,TS val 1650895273 ecr 1537466895,nop,wscale 7], length 0

server - wg-easy container

# ifconfig wg0
wg0       Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
          inet addr:10.0.4.1  P-t-P:10.0.4.1  Mask:255.255.255.0
          UP POINTOPOINT RUNNING NOARP  MTU:65455  Metric:1
          RX packets:5490 errors:1582 dropped:0 overruns:0 frame:1582
          TX packets:6953 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:858572 (838.4 KiB)  TX bytes:1053276 (1.0 MiB)
# tcpdump -i eth0 -v | grep cheat
tcpdump: listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
...
    706a40b241c9.52544 > cheat.sh.80: Flags [SEW], cksum 0xa023 (correct), seq 2571259145, win 65535, options [mss 1380,nop,wscale 6,nop,nop,TS val 1537466895 ecr 0,sackOK,eol], length 0
    cheat.sh.80 > 706a40b241c9.52544: Flags [S.], cksum 0x5598 (incorrect -> 0xa5c1), seq 2401073651, ack 2571259146, win 65408, options [mss 65495,nop,nop,TS val 1650895273 ecr 1537466895,nop,wscale 7], length 0
    192.168.65.7.53 > 706a40b241c9.46412: 52544 1/0/0 188.243.9.5.in-addr.arpa. PTR cheat.sh. (88)
    cheat.sh.80 > 706a40b241c9.52544: Flags [S.], cksum 0x5598 (incorrect -> 0xa5c1), seq 2401073651, ack 2571259146, win 65408, options [mss 65495,nop,nop,TS val 1650895273 ecr 1537466895,nop,wscale 7], length 0
    cheat.sh.80 > 706a40b241c9.52544: Flags [S.], cksum 0x5598 (incorrect -> 0xa5c1), seq 2401073651, ack 2571259146, win 65408, options [mss 65495,nop,nop,TS val 1650895273 ecr 1537466895,nop,wscale 7], length 0
    cheat.sh.80 > 706a40b241c9.52544: Flags [S.], cksum 0x5598 (incorrect -> 0xa5c1), seq 2401073651, ack 2571259146, win 65408, options [mss 65495,nop,nop,TS val 1650895273 ecr 1537466895,nop,wscale 7], length 0
    cheat.sh.80 > 706a40b241c9.52315: Flags [S.], cksum 0x5598 (incorrect -> 0x6baa), seq 4203615612, ack 1431375901, win 65408, options [mss 65495,nop,nop,TS val 1650844888 ecr 4173577889,nop,wscale 7], length 0
    cheat.sh.80 > 706a40b241c9.52544: Flags [S.], cksum 0x5598 (incorrect -> 0xa5c1), seq 2401073651, ack 2571259146, win 65408, options [mss 65495,nop,nop,TS val 1650895273 ecr 1537466895,nop,wscale 7], length 0
    cheat.sh.80 > 706a40b241c9.52544: Flags [S.], cksum 0x5598 (incorrect -> 0xa5c1), seq 2401073651, ack 2571259146, win 65408, options [mss 65495,nop,nop,TS val 1650895273 ecr 1537466895,nop,wscale 7], length 0

There seems to be problem with the S. packet delivery to curl from website. curl then can not complete the 3-way handshake and times out after a while. I have read that the cksum incorrect can be false positive due to computation offloading to NIC, but it does seem like it is the cause.

What I have tried

Inspect NAT and routing

Before tcpdump I was looking at iptables -vxn -L and iptables -vxn -L -t nat for some routing issues. But the tcpdump shows that the packets are routed properly all the way to the curl.

Inspect checksum computation

I have compared output of ethtool -k wg0 and ethtool -k eth0 from inside the wg-easy container for both Docker for Mac versions and they are the same.

I have tried to disable checksum on macbook client

$ sudo sysctl -w net.link.generic.system.hwcksum_tx=0
net.link.generic.system.hwcksum_tx: 1 -> 0

$ sudo sysctl -w net.link.generic.system.hwcksum_rx=0
net.link.generic.system.hwcksum_rx: 1 -> 0

Direct communication using nc

I can successfully comunicate between client and wg-easy container using nc client

$ nc -l 9999
Hello

wg-easy container

# echo Hello | nc 10.0.4.2 9999

Both TCP and UDP (-u) is working.

Docker setting Use kernel networking for UDP

I have tried the switch Resources > Network > Use kernel networking for UDP in both ON and OFF positions (with restart).

Versions

Client

MacBook M1 Sonoma 14.5 WireGuard app from App Store, version 1.0.16 (27)

Server

MacBook Intel Sonoma 14.4.1 WireGuard app from App Store, version 1.0.16 (27) Docker for Mac 4.22.0 / 4.32.0

MacBook M3 Sonoma 14.5 WireGuard app from App Store, version 1.0.16 (27) Docker for Mac 4.32.0

Reproduce

  1. Install recent version of Docker for Mac
  2. Configure and run wg-easy container (see below)
  3. Add VPN client in wg-easy web administration at http://0.0.0.0:51821
  4. Install Wireguard client on another device (I have tested iPhone and another macbook)
  5. Add configuration via QR code (iPhone) or via download + import (macbook)
  6. Turn on VPN tunnel

wg-easy start command

See documentation

docker run -d \
  --name=wg-easy \
  -e WG_HOST=my-host.local \
  -e WG_PORT=51820 \
  -e WG_DEFAULT_ADDRESS=10.0.4.x \
  -e WG_DEFAULT_DNS=<redacted> \
  -e PASSWORD=<redacted> \
  -v ./data:/etc/wireguard \
  -p 0.0.0.0:51820:51820/udp \
  -p 0.0.0.0:51821:51821/tcp \
  --cap-add=NET_ADMIN \
  --cap-add=SYS_MODULE \
  --sysctl="net.ipv4.conf.all.src_valid_mark=1" \
  --sysctl="net.ipv4.ip_forward=1" \
  --restart always \
  ghcr.io/wg-easy/wg-easy:13

wg0.conf (generated after startup and when adding new clients, at ./data/wg0.conf)

# Note: Do not edit this file directly.
# Your changes will be overwritten!

# Server
[Interface]
PrivateKey = <redacted>
Address = 10.0.4.1/24
ListenPort = 51820
PreUp = 
PostUp =  iptables -t nat -A POSTROUTING -s 10.0.4.0/24 -o eth0 -j MASQUERADE; iptables -A INPUT -p udp -m udp --dport 51820 -j ACCEPT; iptables -A FORWARD -i wg0 -j ACCEPT; iptables -A FORWARD -o wg0 -j ACCEPT; 
PreDown = 
PostDown =  iptables -t nat -D POSTROUTING -s 10.0.4.0/24 -o eth0 -j MASQUERADE; iptables -D INPUT -p udp -m udp --dport 51820 -j ACCEPT; iptables -D FORWARD -i wg0 -j ACCEPT; iptables -D FORWARD -o wg0 -j ACCEPT; 

# Client: MacBook M1
[Peer]
PublicKey = <redacted>
PresharedKey = <redacted>
AllowedIPs = 10.0.4.2/32

Client configuration (QR code / downloaded via web administration)

[Interface]
PrivateKey = <redacted>
Address = 10.0.4.2/24
DNS = <redacted>

[Peer]
PublicKey = <redacted>
PresharedKey = <redacted>
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = my-host.local
PersistentKeepalive = 0

Expected behavior

I expect the network to work (eg. internet web pages are loaded), being routed properly through the VPN tunnel.

docker version

Client:
 Version:           27.0.3
 API version:       1.46
 Go version:        go1.21.11
 Git commit:        7d4bcd8
 Built:             Fri Jun 28 23:59:41 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.32.0 (157355)
 Engine:
  Version:          27.0.3
  API version:      1.46 (minimum version 1.24)
  Go version:       go1.21.11
  Git commit:       662f78c
  Built:            Sat Jun 29 00:02:44 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.18
  GitCommit:        ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
 runc:
  Version:          1.7.18
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    27.0.3
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.15.1-desktop.1
    Path:     /Users/ujezdsky/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.28.1-desktop.1
    Path:     /Users/ujezdsky/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.32
    Path:     /Users/ujezdsky/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.14
    Path:     /Users/ujezdsky/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/ujezdsky/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     /Users/ujezdsky/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/ujezdsky/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     /Users/ujezdsky/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/ujezdsky/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.10.0
    Path:     /Users/ujezdsky/.docker/cli-plugins/docker-scout

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 5
 Server Version: 27.0.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.6.32-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 19.51GiB
 Name: docker-desktop
 ID: a121339e-8e8e-4752-a854-d387acf1c3fc
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/ujezdsky/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

257AADD6-96D6-41D9-87E7-605D420CD0E8/20240718105011

Additional Info

Information for older and working Docker for Mac installation (on Intel MacBook, I have not found version 4.22.0 for Apple Silicon).

Diagnostics ID

6FB77661-8FDA-427A-926F-CF4FC04C2C6A/20240718105247

docker version

Client:
 Cloud integration: v1.0.35-desktop+001
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 20:32:30 2023
 OS/Arch:           darwin/amd64
 Context:           desktop-linux

Server: Docker Desktop 4.22.0 (117440)
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 20:35:45 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.5
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2-desktop.1
    Path:     /Users/ujezdsky/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2-desktop.1
    Path:     /Users/ujezdsky/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/ujezdsky/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.20
    Path:     /Users/ujezdsky/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.6
    Path:     /Users/ujezdsky/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/ujezdsky/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/ujezdsky/.docker/cli-plugins/docker-scan
  scout: Command line tool for Docker Scout (Docker Inc.)
    Version:  0.20.0
    Path:     /Users/ujezdsky/.docker/cli-plugins/docker-scout
WARNING: Plugin "/Users/ujezdsky/.docker/cli-plugins/docker-debug" is not valid: failed to fetch metadata: fork/exec /Users/ujezdsky/.docker/cli-plugins/docker-debug: no such file or directory
WARNING: Plugin "/Users/ujezdsky/.docker/cli-plugins/docker-feedback" is not valid: failed to fetch metadata: fork/exec /Users/ujezdsky/.docker/cli-plugins/docker-feedback: no such file or directory

Server:
 Containers: 3
  Running: 2
  Paused: 0
  Stopped: 1
 Images: 91
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 5.15.49-linuxkit-pr
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 19.55GiB
 Name: docker-desktop
 ID: c83038e4-bf16-4723-8add-55b750b74864
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile
petr-ujezdsky commented 3 months ago

Just found out official download link for 4.22.0 for Apple Silicon thanks to this gist and reproduced this issue.

So 4.22.0 works with both Intel and Apple Silicon. 4.32.0 and now 4.33.0 are still broken on both Intel and Apple Silicon.

I will try to bisect the versions to find the first bad one.

petr-ujezdsky commented 3 months ago

Found it. The last working version is 4.22.1 (download for Apple Silicon). Versions 4.23.0 (download for Apple Silicon) and newer are broken.