search

tailscale / tailscale

The easiest, most secure way to use WireGuard and 2FA.
https://tailscale.com
BSD 3-Clause "New" or "Revised" License
19.32k stars 1.51k forks source link

Segmentation Fault and iptables Issues on WD MyCloud EX2 Ultra with Tailscale Docker Container #12787

Closed bamit99 closed 3 months ago

bamit99 commented 4 months ago

What is the issue?

While attempting to run Tailscale on a WD MyCloud EX2 Ultra using Docker, the process encounters repeated segmentation faults and issues related to iptables. This appears to be due to incompatibilities with the kernel and iptables implementation on the NAS device. Despite using the --privileged flag and ensuring the necessary kernel modules are loaded, the issue persists.

Steps to reproduce

Steps to Reproduce SSH into the WD MyCloud EX2 Ultra. Install and configure Docker. Run the Tailscale Docker container with elevated privileges:

docker run -d \ --name=tailscale \ --net=host \ --privileged \ -v /mnt/docker/tailscale:/var/lib/tailscale \ tailscale/tailscale:unstable tailscaled

Check the Docker logs for errors: docker logs tailscale

Are there any recent changes that introduced the issue?

No specific changes on the NAS or Docker environment, though this might be a longstanding compatibility issue with the specific kernel version and iptables implementation on the WD MyCloud EX2 Ultra.

OS

Linux, Other

OS version

Linux (WD MyCloud EX2 Ultra) 4.14.22-armada-18.09.3

Tailscale version

v1.69.144-t8d7b78f3f

Other software

2024/07/11 17:37:45 linuxfw: clear iptables: could not get iptables version: signal: segmentation fault 2024/07/11 17:37:45 linuxfw: clear ip6tables: could not get iptables version: signal: segmentation fault 2024/07/11 17:37:45 cleanup: list tables: socket: protocol not supported 2024/07/11 17:37:45 wgengine.NewUserspaceEngine(tun "tailscale0") ... 2024/07/11 17:37:45 setting link attributes: setsockopt: protocol not available 2024/07/11 17:37:45 router: error querying IP rules (does kernel have IP_MULTIPLE_TABLES?): operation not supported 2024/07/11 17:37:45 router: warning: running without policy routing 2024/07/11 17:37:45 dns: [rc=unknown ret=direct] 2024/07/11 17:37:45 dns: using "direct" mode 2024/07/11 17:37:45 dns: using dns.directManager 2024/07/11 17:37:45 link state: interfaces.State{defaultRoute=egiga0 ifs={docker0:[172.17.0.1/16] egiga0:[192.168.0.100/24]} v4=true v6=false} 2024/07/11 17:37:45 onPortUpdate(port=37497, network=udp6) 2024/07/11 17:37:45 router: using firewall mode pref 2024/07/11 17:37:45 router: default choosing iptables 2024/07/11 17:37:45 UpdateMagicsockPort(port=37497, network=udp6) failed: %!w(fmt.wrapError=&{could not setup netfilter: could not create new netfilter: could not get iptables version: signal: segmentation fault 0x150a050}) ... panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x12 pc=0x4b9dc8] goroutine 149 [running]: tailscale.com/util/linuxfw.(iptablesRunner).HasIPV6Filter(...) tailscale.com/util/linuxfw/iptables_runner.go:163 tailscale.com/util/linuxfw.(iptablesRunner).getTables(...) tailscale.com/util/linuxfw/iptables_runner.go:209 tailscale.com/util/linuxfw.(iptablesRunner).AddChains(0x0) tailscale.com/util/linuxfw/iptables_runner.go:282 +0x14 tailscale.com/wgengine/router.(linuxRouter).setNetfilterMode(0x14821c0, 0x2) tailscale.com/wgengine/router/router_linux.go:641 +0x664 tailscale.com/wgengine/router.(linuxRouter).Set(0x14821c0, 0x43f66e0) tailscale.com/wgengine/router/router_linux.go:386 +0x270 tailscale.com/wgengine.(userspaceEngine).Reconfig(0x14d6208, 0x43702c8, 0x43f66e0, 0x45305d0) tailscale.com/wgengine/userspace.go:986 +0x145c tailscale.com/wgengine.(watchdogEngine).Reconfig.func1() tailscale.com/wgengine/watchdog.go:124 +0x40 tailscale.com/wgengine.(watchdogEngine).watchdogErr.func2() tailscale.com/wgengine/watchdog.go:84 +0x28 created by tailscale.com/wgengine.(*watchdogEngine).watchdogErr in goroutine 119 tailscale.com/wgengine/watchdog.go:83 +0x1c4

Additional Information The TUN module is manually loaded and /dev/net/tun is created. Using the --privileged flag in Docker. The same issue persists with tailscale/tailscale:unstable and other images.

Bug report

iptables v1.8.7 (legacy)

No response

irbekrm commented 4 months ago

Hi, thanks for opening the issue.

I do see that there is an earlier issue about using Tailscale on MyCloud https://github.com/tailscale/tailscale/issues/4622#issuecomment-1147402933 related to iptables.We will take a look at whether we should perhaps disable netfilter/iptables mode on these distros if it seems that iptables/netfilter cannot be used.

A couple clarifying questions:

iptables v1.8.7 (legacy)

is this the output of iptables --version on the device?

What is the output of lsmod on that device?

Do you know whether nftables are available on the device? Could you try running tailscale with netfilter in nftables mode? You can do that by setting TS_DEBUG_FIREWALL_MODE=nftables env var to the container.

If nftables does not work, are you able to run it in userspace mode? This is the default form containers, so I assume you explicitly run it in kernel mode for reasons?

bamit99 commented 4 months ago

Yes, iptables v1.8.7 (legacy) is actual version of iptables on my NAS. lsmod below. It does not support nftables. I tried Userspace mode too and it failed unfortunately. This is why I shifted to Kernel mode.

root@MyCloudEX2Ultra ~ # lsmod Module Size Used by br_netfilter 131072 0 bridge 196608 1 br_netfilter stp 131072 1 bridge llc 131072 2 bridge,stp iptable_nat 131072 1 ipt_MASQUERADE 131072 2 nf_nat_masquerade_ipv4 131072 1 ipt_MASQUERADE xt_nat 131072 1 xt_addrtype 131072 2 nf_nat_ipv4 131072 1 iptable_nat nf_nat 131072 3 xt_nat,nf_nat_masquerade_ipv4,nf_nat_ipv4 netatop 131072 0 iscsi_tcp 131072 0 libiscsi_tcp 131072 1 iscsi_tcp libiscsi 131072 2 libiscsi_tcp,iscsi_tcp scsi_transport_iscsi 131072 3 libiscsi,iscsi_tcp nf_log_ipv6 131072 0 nf_conntrack_ipv6 131072 0 nf_defrag_ipv6 131072 1 nf_conntrack_ipv6 ip6table_filter 131072 1 ip6_tables 131072 1 ip6table_filter nf_log_ipv4 131072 0 nf_conntrack_ipv4 131072 4 nf_defrag_ipv4 131072 1 nf_conntrack_ipv4 iptable_filter 131072 1 ip_tables 131072 2 iptable_filter,iptable_nat xt_limit 131072 0 nf_log_common 131072 2 nf_log_ipv6,nf_log_ipv4 xt_LOG 131072 0 xt_recent 131072 0 xt_tcpudp 131072 2 xt_state 131072 0 xt_conntrack 131072 0 nf_conntrack 163840 9 xt_state,xt_nat,nf_conntrack_ipv6,nf_conntrack_ipv4,ipt_MASQUERADE,nf_nat_masquerade_ipv4,xt _conntrack,nf_nat_ipv4,nf_nat x_tables 131072 13 xt_state,xt_LOG,xt_nat,ip_tables,iptable_filter,xt_tcpudp,ipt_MASQUERADE,xt_limit,xt_recent ,ip6table_filter,xt_addrtype,xt_conntrack,ip6_tables xfrm6_mode_tunnel 131072 0 xfrm6_mode_transport 131072 0 xfrm6_mode_beet 131072 0 ip6_tunnel 131072 0 tunnel6 131072 1 ip6_tunnel sit 131072 0 ip_tunnel 131072 1 sit tunnel4 131072 1 sit ipv6 393216 76 nf_conntrack_ipv6,sit,xfrm6_mode_tunnel,xfrm6_mode_beet,nf_defrag_ipv6,ip6_tunnel,bridge exfat 163840 0 tun 131072 0 ufsd 819200 0 jnl 131072 1 ufsd cifs 458752 0 udf 163840 0 nls_utf8 131072 0 nls_iso8859_2 131072 0 nls_iso8859_1 131072 0 nls_cp850 131072 0 nfsd 163840 0 nfsv3 131072 0 nfsv2 131072 0 nfs 229376 2 nfsv2,nfsv3 lockd 163840 4 nfsd,nfsv2,nfs,nfsv3 grace 131072 1 lockd nfs_acl 131072 1 nfsd isofs 163840 0 hfs 163840 0 vfat 131072 0 msdos 131072 0 fat 163840 2 msdos,vfat raid456 196608 0 raid10 163840 0 raid1 131072 2 raid0 131072 0 linear 131072 0 dm_thin_pool 163840 0 dm_persistent_data 131072 1 dm_thin_pool dm_snapshot 131072 0 dm_crypt 131072 0 dm_bufio 131072 2 dm_persistent_data,dm_snapshot dm_bio_prison 131072 1 dm_thin_pool dm_mod 163840 4 dm_bufio,dm_crypt,dm_thin_pool,dm_snapshot rpcsec_gss_krb5 131072 0 auth_rpcgss 131072 1 rpcsec_gss_krb5 sunrpc 262144 8 auth_rpcgss,nfsd,rpcsec_gss_krb5,nfsv2,nfs_acl,lockd,nfs,nfsv3 async_raid6_recov 131072 1 raid456 async_pq 131072 2 raid456,async_raid6_recov async_memcpy 131072 2 raid456,async_raid6_recov async_xor 131072 3 async_pq,raid456,async_raid6_recov async_tx 131072 5 async_xor,async_pq,raid456,async_memcpy,async_raid6_recov xor 131072 1 async_xor raid6_pq 229376 3 async_pq,raid456,async_raid6_recov oid_registry 131072 1 auth_rpcgss

irbekrm commented 3 months ago

Thank you for the additional info.

Yes, iptables v1.8.7 (legacy) is actual version of iptables on my NAS

So it appears that you are able to run iptables --version on the NAS directly using the iptables binary on the host, but when tailscale in container runs the same command using the iptables binary packaged in the image that results in segfault (this is where this line in logs come from):

2024/07/11 17:37:45 linuxfw: clear iptables: could not get iptables version: signal: segmentation fault

I've also tried running the latest tailscale/tailscale image on a VM with 4.x kernel and did not observe any issues with iptables.

That seems to suggest that perhaps somehow the particular iptables binary that comes with the image that you pulled does not work with your NAS, maybe architecture mismatch?

Would you be able to provide

Additionally, it would be interesting what is the output of file /sbin/xtables-nft-multi (or whatever is the actual path of the iptables binary) on the NAS host?

bamit99 commented 3 months ago

For now, this worked. I made quite a few changes to the OS so just did a clean install using firmware image then ran the command below. It's holding up. No crashes etc.

docker run -d \ --name=tailscale \ --restart=unless-stopped \ --network=host \ --privileged \ -v /mnt/HD/HD_a2/Nas_Prog/tailscale/data:/var/lib/tailscale \ -e TS_STATE_DIR=/var/lib/tailscale \ tailscale/tailscale:latest \ tailscaled --tun=userspace-networking