search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.83k stars 2.42k forks source link

With podman 5.0.0-rc6, rootless containers can no longer resolve DNS using corporate VPN servers #22044

Closed jiridanek closed 8 months ago

jiridanek commented 8 months ago

Issue Description

After updating, I can no longer resolve domain names using DNS servers on my corporation's VPN.

$ podman run --rm -it quay.io/fedora/fedora:38-x86_64 curl http://download-node-02.eng.bos.redhat.com
curl: (6) Could not resolve host: download-node-02.eng.bos.redhat.com

My DNS setup looks like this

$ resolvectl 
Global
         Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub
Current DNS Server: 127.0.0.1
       DNS Servers: 127.0.0.1

Link 2 (enp9s0u2u1u2)
    Current Scopes: none
         Protocols: -DefaultRoute LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (wlp0s20f3)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.42.42.42
         DNS Servers: 10.m.n.o 10.p.q.r

Link 4 (tun0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: -DefaultRoute LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.8.8.8
       DNS Servers: 10.y.z.ž 10.a.b.c
        DNS Domain: redhat.com

Steps to reproduce the issue

Steps to reproduce the issue

$ podman run --rm -it quay.io/fedora/fedora:38-x86_64 curl http://download-node-02.eng.bos.redhat.com
curl: (6) Could not resolve host: download-node-02.eng.bos.redhat.com

Describe the results you received

$ podman run --rm -it quay.io/fedora/fedora:38-x86_64 cat /etc/resolv.conf
search redhat.com
nameserver 10.200.0.245
nameserver 10.192.206.245

Describe the results you expected

When I add --network=slirp4netns, things start to work.

$ podman run --network=slirp4netns --rm -it quay.io/fedora/fedora:38-x86_64 curl http://download-node-02.eng.bos.redhat.com
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
 <head>
  <title>Index of /</title>
...
$ podman run --network=slirp4netns --rm -it quay.io/fedora/fedora:38-x86_64 cat /etc/resolv.conf
search redhat.com
nameserver 10.0.2.3
nameserver 10.200.0.245
nameserver 10.192.206.245

podman info output

$ podman info
host:
  arch: amd64
  buildahVersion: 1.35.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.8-4.fc40.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 93.3
    systemPercent: 1.15
    userPercent: 5.55
  cpus: 12
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: workstation
    version: "40"
  eventLogger: journald
  freeLocks: 2047
  hostname: fedora
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    - container_id: 65537
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
  kernel: 6.8.0-63.fc40.1.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 394874880
  memTotal: 33395396608
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc40.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-2.fc40.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.fc40.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240220.g1e6f92b-1.fc40.x86_64
    version: |
      pasta 0^20240220.g1e6f92b-1.fc40.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8324640768
  swapTotal: 8589930496
  uptime: 3h 2m 15.00s (Approximately 0.12 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  image-registry.openshift-image-registry.svc:
    Blocked: false
    Insecure: false
    Location: default-route-openshift-image-registry.apps-crc.testing
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: image-registry.openshift-image-registry.svc
    PullFromMirror: ""
store:
  configFile: /home/jdanek/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/jdanek/.local/share/containers/storage
  graphRootAllocated: 510389125120
  graphRootUsed: 473875283968
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/jdanek/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.0-rc6
  Built: 1710288000
  BuiltTime: Wed Mar 13 01:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.0
  Os: linux
  OsArch: linux/amd64
  Version: 5.0.0-rc6

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

$ podman version Client: Podman Engine Version: 5.0.0-rc6 API Version: 5.0.0-rc6 Go Version: go1.22.0 Built: Wed Mar 13 01:00:00 2024 OS/Arch: linux/amd64

$ rpm -q podman podman-5.0.0~rc6-2.fc40.x86_64

$ cat /etc/redhat-release Fedora release 40 (Forty)

Prerelease, updated to today's versions of packages.

Additional information

This problem is similar in symptoms to my previous issue

Luap99 commented 8 months ago

I am already working on a fix which should restore the slirp4netns behaviour with pasta (https://github.com/containers/podman/pull/22043, if you could build and test it that would be great)

However what I do not understand is why dns is not resolving:

nameserver 10.200.0.245
nameserver 10.192.206.245

These are still valid servers, no? I assume the issue is that you have split dns setup and the container of course will just route everything to one address.

jiridanek commented 8 months ago

I realize I sort of screwed up the report my inconsistent censorship. Let's show the real IPs. They are just meaningless IPs.

nameserver 10.200.0.245
nameserver 10.192.206.245

These are valid servers, but they are the ones that DHCP running on my WiFi assigned me.

What I want to be using is one of the DNS servers on the VPN.

Current DNS Server: 10.45.248.15
       DNS Servers: 10.45.248.15 10.38.5.26

This is what resolvectl gives me:

Link 3 (wlp0s20f3)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.192.206.245
       DNS Servers: 10.200.0.245 10.192.206.245

Link 4 (tun0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: -DefaultRoute LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 10.45.248.15
       DNS Servers: 10.45.248.15 10.38.5.26
        DNS Domain: redhat.com
jiridanek commented 8 months ago

@Luap99 Let me know when https://github.com/containers/podman/pull/22043 builds cleanly in Packit, I'll install it from there then!

Hmm, never mind, it does actually build already, https://download.copr.fedorainfracloud.org/results/packit/containers-podman-22043/fedora-40-x86_64/07159051-podman/ Trying to install that.

Luap99 commented 8 months ago

yeah the container has no idea about the split dns setup, my PR will almost certainly fix this as it will insert a new ip in resolv.conf which will be used by pasta to proxy.

Actually you do not need a build, just add --network pasta:--dns-forward,169.254.0.1 --dns 169.254.0.1 to the podman run command, that should result in the same thing, my changes only make something like that the default basically

jiridanek commented 8 months ago

Ok, tried your command and also the updated pkg versions, both work

$ podman run --network pasta:--dns-forward,169.254.0.1 --dns 169.254.0.1  --rm -it quay.io/fedora/fedora:38-x86_64 curl http://download-node-02.eng.bos.redhat.com
jiridanek commented 8 months ago

Forgot to post workaround, so here it is, if anyone is still interested. Switch from pasta back to slirp4netns.

$ cat ~/.config/containers/containers.conf 
[containers]

[engine]

[machine]

[network]
default_rootless_network_cmd="slirp4netns"

[secrets]

[configmaps]
Luap99 commented 8 months ago

FYI the fix is not in 5.0.0 however I work on backporting it so that will be included in the next 5.0.1 release

jiridanek commented 8 months ago

Good I put up my workaround. I might need to point my fellow early-adopter colleagues to it, if they get Fedora 40 before podman 5.0.1 is in.

francoism90 commented 7 months ago

@jiridanek Thanks for your workaround! I searched all the day, but couldn't understand why DNS inside the container(s) didn't work at all after upgrading Podman 5.0 on Silverblue.