[Bug]: DNS randomly will not work after reboot

asagecty commented 1 year ago

Issue Description

I have set up a container for redis and oauth2-proxy to run rootless and start on reboot. Randomly (it seems) DNS will fail on the oauth2-proxy container. When this happens, there are extra logs in journalctl --user -u oauth2-proxy.service relating to dnsmasq that I do not see when everything is working normally.

Jan 09 20:40:09 oauth2-proxytest dnsmasq[6336]: started, version 2.86 cachesize 150
Jan 09 20:40:09 oauth2-proxytest dnsmasq[6336]: compile time options: IPv6 GNU-getopt DBus no-UBus i18n IDN2 DHCP DHCPv6 no-Lua TFTP conntrack ipset auth cryptohash DNSSEC loop-detect inotify dumpfile
Jan 09 20:40:09 oauth2-proxytest dnsmasq[6336]: using only locally-known addresses for dns.podman
Jan 09 20:40:09 oauth2-proxytest dnsmasq[6336]: reading /etc/resolv.conf
Jan 09 20:40:09 oauth2-proxytest dnsmasq[6336]: using nameserver 10.0.2.3#53
Jan 09 20:40:09 oauth2-proxytest dnsmasq[6336]: using nameserver 8.8.8.8#53
Jan 09 20:40:09 oauth2-proxytest dnsmasq[6336]: using nameserver 8.8.4.4#53
Jan 09 20:40:09 oauth2-proxytest dnsmasq[6336]: using only locally-known addresses for dns.podman
Jan 09 20:40:09 oauth2-proxytest dnsmasq[6336]: read /run/user/1000/containers/cni/dnsname/oauth2-proxy/addnhosts - 2 addresses

To get things working again, I can run podman network reload --all and restart the service and it will start working again. When I reload the networks, I get the following error:

~$ podman network reload --all
time="2023-01-09T20:40:36Z" level=error msg="open /run/user/1000/containers/cni/dnsname/oauth2-proxy/pidfile: no such file or directory"
ERRO[0000] failed to kill slirp4netns process: no such process
55f42ec5480d65020fb6ab71a893e9e17e77c660deea3d7cf1956dac2fa63f92

After that I can start the service again and everything is fine until a random reboot breaks again

Steps to reproduce the issue

Reboot until the service doesn't start
Check journalctl --user -u oauth2-proxy.service and see log etries for dnsmasq
Run podman network reload --all then restart the service and everything works again

Describe the results you received

DNS is broken at random on system reboots

Describe the results you expected

DNS should work on every reboot

podman info output

~$ podman version
Version:      3.4.4
API Version:  3.4.4
Go Version:   go1.17.3
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

~$ podman info
host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/bin/conmon'
    path: /usr/bin/conmon
    version: 'conmon version 2.0.25, commit: unknown'
  cpus: 2
  distribution:
    codename: jammy
    distribution: ubuntu
    version: "22.04"
  eventLogger: journald
  hostname: hostname
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.15.0-57-generic
  linkmode: dynamic
  logDriver: journald
  memFree: 7629377536
  memTotal: 8336162816
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17
      commit: 0e9229ae34caaebcb86f1fde18de3acaf18c6d9a
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.6.1
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 1m 52.04s
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/user/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 2
  runRoot: /run/user/1000/containers
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.4
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.17.3
  OsArch: linux/amd64
  Version: 3.4.4

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

Luap99 commented 1 year ago

3.4 is not supported upstream, I am very sure this was fixed in 4.0.

asagecty commented 1 year ago

3.4 is the current version on Jammy. https://packages.ubuntu.com/jammy/podman Do you know when the repo will be updated? We would rather stay away from manual builds on production servers.

asagecty commented 1 year ago

@Luap99 any idea on when the Ubuntu repos will be updated?

Luap99 commented 1 year ago

You need to ask this the ubuntu/debian packager(s), upstream has no control over this. From experience I would guess it will only be updated in new debian/ubuntu versions, bookworm has 4.3.1 at the moment: https://packages.debian.org/bookworm/podman

containers / podman