search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.58k stars 13.74k forks source link

Rootless podman/slirp4netns sandbox blocks dns #231191

Open KenMacD opened 1 year ago

KenMacD commented 1 year ago

Describe the bug

Slirp4netns does not resolve hostnames because the chain of /etc/resolv.conf passes through /nix/store.

On my NixOS system my /etc/resolv.con is a symlink to /etc/static/resolv.conf. The path to /etc/static is in the Nix store though, /nix/store/...-etc/etc.

Steps To Reproduce

Steps to reproduce the behavior:

  1. Install the podman package
  2. As non-root run a container and do a query, for example: podman run --rm -it busybox busybox nslookup nixos.org 10.0.2.3
  3. See no servers could be reached
  4. Link /etc/resolv.conf directly: sudo ln -rsf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf
  5. Run container again, and see that query works.

Additional context

In the process list it can be seen that podman started slirp4netns with the argument --enable-sandbox. The docs and sandbox code make it clear DNS will not work if /etc/resolv.conf is outside of /etc and /run. There's even a warning message that should tell you about it. The warning doesn't work in this case because it uses realpath() which skips over the intermediate links.

Notify maintainers

@adisbladis @saschagrunert @vdemeester @zowoq

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.3.1, NixOS, 23.05 (Stoat), 23.05.19700101.dirty`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.3`
 - channels(root): `""`
 - channels(kenny): `""`
 - nixpkgs: `/nix/store/zjpd6dn9kza8zl3l17cwgyifsxbvi3nb-source`
KenMacD commented 1 year ago

As a workaround I'm currently using environment.etc."resolv.conf".mode = "direct-symlink"; in my config.

Ovyerus commented 11 months ago

Ran into this today with Docker Rootless as well! The resolv.conf fix above worked.

KiruyaMomochi commented 9 months ago

There has been a merged PR in slirp4netns, which added support for escaping resolv.conf symlinks: https://github.com/rootless-containers/slirp4netns/pull/318. The problem is that they only checked the final resolved path of /etc/resolv.conf. However in our case it's more complicated. You can see my comment at https://github.com/rootless-containers/slirp4netns/issues/333 for a detailed explanation.

JustinLex commented 8 months ago

Encountered this on rootless docker + systemd-resolved too.

Docker logs correctly noted detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf, but the sandbox was breaking Docker's 10.0.2.3 nameserver.

The environment.etc."resolv.conf".mode workaround worked for me, thanks!

SebTM commented 5 months ago

Hey, I have a similar issue but can't resolve it with the "environment.etc(...)"-solution. The issue is that dns inside the container is broken as soon as I'm connected to any vpn (before creating/starting them does not matter) - my setup:

network-setup from my compose.yaml:

networks:
    default:
        driver: bridge
        ipam:
            config:
                # Take fixed subnet within containers default_subnet-range
                - subnet: 10.200.70.0/24

podman network inspect:

[
     {
          "name": "xyz_default",
          "id": "d5b712d2c6a309f819dabeb4767c049e3901b4a517533b3c790fcf93b5c17635",
          "driver": "bridge",
          "network_interface": "podman1",
          "created": "2024-04-22T15:42:51.374248457+02:00",
          "subnets": [
               {
                    "subnet": "10.200.70.0/24",
                    "gateway": "10.200.70.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": true,
          "labels": {
               "com.docker.compose.network": "default",
               "com.docker.compose.project": "xyz",
               "com.docker.compose.version": "2.26.1"
          },
          "options": {
               "isolate": "true"
          },
          "ipam_options": {
               "driver": "host-local"
          },
          "containers": {
               "0dcf7de3f5710edeb5061a0cd87e1ed1765eb8a5434706138b6513b7fde52021": {
                    "name": "xyz-challenge",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.200.70.11/24",
                                        "gateway": "10.200.70.1"
                                   }
                              ],
                              "mac_address": "ce:df:7a:02:40:6a"
                         }
                    }
               },
               "16bef19d6dc0d86749a8d4e6de3ffe1b37e5cde8456af45cf37b935bb0f756eb": {
                    "name": "xyz-admin-1",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.200.70.12/24",
                                        "gateway": "10.200.70.1"
                                   }
                              ],
                              "mac_address": "66:dd:c5:13:94:97"
                         }
                    }
               },
               "2f43614cfa9a9e47c6fb9717695706a4631ab3d0cd27ad7b232983f2e4edba0a": {
                    "name": "xyz-api-1",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.200.70.18/24",
                                        "gateway": "10.200.70.1"
                                   }
                              ],
                              "mac_address": "be:24:a6:d5:cd:83"
                         }
                    }
               },
               "30051d212afc01e120498db9351d211b62f3bc7d492ba36736690bed089cec86": {
                    "name": "xyz-redis",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.200.70.7/24",
                                        "gateway": "10.200.70.1"
                                   }
                              ],
                              "mac_address": "ca:94:45:ba:dc:f5"
                         }
                    }
               },
               "4cb91ef0d197887f4607971ca55376269cf5827d66da81b6c087ca1924096a08": {
                    "name": "xyz-stripe-mock-1",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.200.70.6/24",
                                        "gateway": "10.200.70.1"
                                   }
                              ],
                              "mac_address": "56:62:5c:96:4d:0b"
                         }
                    }
               },
               "594e622f20b5c12f686c24f84459208704db84b6e5a8a57db57bf4963dbe3972": {
                    "name": "xyz-test-encryption",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.200.70.16/24",
                                        "gateway": "10.200.70.1"
                                   }
                              ],
                              "mac_address": "4e:28:ce:d0:7f:6b"
                         }
                    }
               },
               "6227b63df367b47585b969899f22f99c7f82288f8c1e25674da41535d7908730": {
                    "name": "xyz-pma-1",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.200.70.10/24",
                                        "gateway": "10.200.70.1"
                                   }
                              ],
                              "mac_address": "8e:e1:76:91:5a:bf"
                         }
                    }
               },
               "6c4437ba87841f7825b5d1ce82a25f4d0aa7ba6c5f108620c15faf171f7da7af": {
                    "name": "xyz-redisCommander-1",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.200.70.17/24",
                                        "gateway": "10.200.70.1"
                                   }
                              ],
                              "mac_address": "7a:4e:14:c9:bf:1e"
                         }
                    }
               },
               "74799d7ee5473814214bfccfcfe4e8819d852e52cd61f09498ec41c88a0c062f": {
                    "name": "xyz-payment-token-1",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.200.70.13/24",
                                        "gateway": "10.200.70.1"
                                   }
                              ],
                              "mac_address": "ee:fa:f6:47:42:71"
                         }
                    }
               },
               "7dad82b070afcc6d49feb817ccb08ca6b1bbc05df30d624e8c8989e3cbd598c5": {
                    "name": "xyz-sphinx",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.200.70.21/24",
                                        "gateway": "10.200.70.1"
                                   }
                              ],
                              "mac_address": "9e:7e:af:e0:79:59"
                         }
                    }
               },
               "819b4765a5a52528399fc2d31138ba773a99d74ccfc7a1f0ec22a6271c8627e3": {
                    "name": "xyz-haproxy-1",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.200.70.9/24",
                                        "gateway": "10.200.70.1"
                                   }
                              ],
                              "mac_address": "52:19:c6:d9:ce:d4"
                         }
                    }
               },
               "820b8bbf53b5ad6deaeb5af82dbb6ab22a08c102da0350edc263ff9ed13b38b2": {
                    "name": "xyz-selenium",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.200.70.5/24",
                                        "gateway": "10.200.70.1"
                                   }
                              ],
                              "mac_address": "5a:6a:8f:28:3e:c2"
                         }
                    }
               },
               "8e67465bd5a28719cc970b088ca3555049bdc778a478ed56f97e4adbe9a285c0": {
                    "name": "xyz-mock-http-server-1",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.200.70.4/24",
                                        "gateway": "10.200.70.1"
                                   }
                              ],
                              "mac_address": "de:74:f9:06:b8:ef"
                         }
                    }
               },
               "a791522b62cbb6838dfe9c6f75d26d03e6592009ee8660bdf06560e9f32efb02": {
                    "name": "xyz-ceph-api",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.200.70.2/24",
                                        "gateway": "10.200.70.1"
                                   }
                              ],
                              "mac_address": "f2:4b:3e:a9:25:43"
                         }
                    }
               },
               "cbf3bce68898e055a014402fb16403e7eac1e0599a7010bdc08c4d78ecfefadd": {
                    "name": "xyz-mock-imap-1",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.200.70.15/24",
                                        "gateway": "10.200.70.1"
                                   }
                              ],
                              "mac_address": "e6:5e:4d:6c:0b:80"
                         }
                    }
               },
               "db0bf795c6fd57cb432d0ecb9a92782ebe97ec65339d709e4d630c99c9e99f10": {
                    "name": "xyz-mysql",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.200.70.3/24",
                                        "gateway": "10.200.70.1"
                                   }
                              ],
                              "mac_address": "92:e0:f2:c5:6b:6c"
                         }
                    }
               },
               "e33040a558d328dc51869caebac7842aa6d6985f6ad20b68cb32cad0b102a048": {
                    "name": "xyz-mailhog",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.200.70.14/24",
                                        "gateway": "10.200.70.1"
                                   }
                              ],
                              "mac_address": "da:95:5c:fc:8f:09"
                         }
                    }
               },
               "f3f1bd092e118dff2f946403d031dba224989fa704381997c72fc5eecbf6a36e": {
                    "name": "xyz-gopenpgp-service",
                    "interfaces": {
                         "eth0": {
                              "subnets": [
                                   {
                                        "ipnet": "10.200.70.8/24",
                                        "gateway": "10.200.70.1"
                                   }
                              ],
                              "mac_address": "92:32:6e:45:7b:8f"
                         }
                    }
               }
          }
     }
]

podman run --network proton_default --rm -it busybox busybox sh:

ip a:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if26: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue qlen 1000
    link/ether 42:bd:a2:72:c1:ad brd ff:ff:ff:ff:ff:ff
    inet 10.200.70.24/24 brd 10.200.70.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::40bd:a2ff:fe72:c1ad/64 scope link
       valid_lft forever preferred_lft forever
/ # nslookup google.de
;; connection timed out; no servers could be reached

/ # nslookup google.de 10.200.70.1
;; connection timed out; no servers could be reached
❯ ls -al /etc/resolv.conf
Permissions Size User Date Modified Name
lrwxrwxrwx     - root 22 Apr 15:11   /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf

cat /run/systemd/resolve/stub-resolv.conf

# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

ip a (host):

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: wlp166s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether xx:f8:xx:75:18:xx brd ff:ff:ff:ff:ff:ff
    inet 10.10.27.195/24 brd 10.10.27.255 scope global dynamic noprefixroute wlp166s0
       valid_lft 39874sec preferred_lft 39874sec
    inet6 2xxx:a61:xxx:d20:6ee5:25f7:64d8:68e2/64 scope global temporary dynamic
       valid_lft 4953sec preferred_lft 1352sec
    inet6 2xxx:a61:xxx:d20:8ef8:c5ff:fe75:1877/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 4953sec preferred_lft 1352sec
    inet6 fd27:1:1:20:3f7a:ec98:a975:e69c/64 scope global temporary dynamic
       valid_lft 601476sec preferred_lft 82910sec
    inet6 fd27:1:1:20:8ef8:c5ff:fe75:1877/64 scope global mngtmpaddr noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::8ef8:c5ff:fe75:1877/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
5: ipv6leakintrf0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether xx:b9:xx:40:ef:xx brd ff:ff:ff:ff:ff:ff
    inet6 fdeb:446c:912d:8da::/64 scope global noprefixroute
       valid_lft forever preferred_lft forever
    inet6 fe80::690b:9bf5:7867:a3ae/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
6: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 10.96.0.5/16 brd 10.96.255.255 scope global noprefixroute tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::321b:7efe:6f82:2f8/64 scope link stable-privacy proto kernel_ll
       valid_lft forever preferred_lft forever

resolvectl (host):

Global
         Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub
        DNS Domain: int ~.

Link 2 (wlp166s0)
    Current Scopes: LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 5 (ipv6leakintrf0)
    Current Scopes: none
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 6 (tun0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: +DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.96.0.1
       DNS Servers: 10.96.0.1
        DNS Domain: ~.

Any idea/hint? 🙏🏻

SebTM commented 5 months ago

For anyone having the same issue: Appears podman 4 -> 5 switched from slirp4netns to pasta by default which breaks it for me as pasta uses the ipv6leak interface instead of tun0 - will look further into pasta-config but for now this solves my issues: https://github.com/containers/podman/issues/22044#issuecomment-2009255504

Solution to my issue with new default pasta-network: https://github.com/containers/podman/issues/22320#issuecomment-2072368902

KiruyaMomochi commented 4 months ago

I almost forgot this one...

It seems the new mount API have ability to create mountpoint without following symlink, but I can only find few examples. I have managed to patch slirp4netns with this new API before. You can find my patch at https://github.com/KiruyaMomochi/nix/blob/main/packages/slirp4netns.patch.

I still somehow think it's a hack so I'm not sure if I should contribute this...