search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
22.77k stars 2.32k forks source link

Networking issues (particularly domain name resolution) #18810

Closed BetaChess closed 11 months ago

BetaChess commented 1 year ago

Issue Description

I am experiencing issues with domain name resolution and have set up a test case to illustrate the problem.

I've created two containers named alp1 and alp2, which are connected to a bridge network titled networktesting_bridge. These containers are able to access the internet without any issues and can successfully ping each other. However, they are only able to resolve their own names. For instance, alp1 can perform ping alp1, but not ping alp2. They are also unable to resolve external hostnames like "google.com". However if I ping the IP address directly, it works without a hitch.

Interestingly, everything functions flawlessly when using the default Podman network bridge, except for the ability to ping container names, which is a known limitation and therefore expected. When alp1 and alp2 are connected to the default bridge, commands like ping google.com execute perfectly.

In addition to these issues, opening ports will, for some reason, only allow traffic through my tailscale vpn, which I find severely confusing. The containers are inaccesible on both LAN and over the internet via port forwarding, even after specifically making ALLOW rules in ufw.

Steps to reproduce the issue

Steps to reproduce the issue

  1. I am using the follwing compose file: 
    
version: "2.1"

networks: bridge: driver: bridge internal: false

services: alpine1: name: alp1 image: alpine:latest tty: true networks: bridge: restart: "no" alpine2: name: alp2 image: alpine:latest tty: true networks: bridge: restart: "no"

2. Run the compose file with `podman-compose up -d`. Containers start successfully.
3. Go into the shell and try to ping any domain name.

### Describe the results you received

When I use the shell of alp2 to run some tests, I get the following results:

/ # ping google.com ping: bad address 'google.com' / # ping alp1 ping: bad address 'alp1' / # ping alp2 -c 2 PING alp2 (10.89.1.3): 56 data bytes 64 bytes from 10.89.1.3: seq=0 ttl=42 time=0.065 ms 64 bytes from 10.89.1.3: seq=1 ttl=42 time=0.079 ms

--- alp2 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.065/0.072/0.079 ms / # ping 10.89.1.2 -c 2 PING 10.89.1.2 (10.89.1.2): 56 data bytes 64 bytes from 10.89.1.2: seq=0 ttl=42 time=0.081 ms 64 bytes from 10.89.1.2: seq=1 ttl=42 time=0.097 ms

--- 10.89.1.2 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.081/0.089/0.097 ms / # ping 9.9.9.9 -c 2 PING 9.9.9.9 (9.9.9.9): 56 data bytes 64 bytes from 9.9.9.9: seq=0 ttl=42 time=27.298 ms 64 bytes from 9.9.9.9: seq=1 ttl=42 time=25.353 ms

--- 9.9.9.9 ping statistics --- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 25.353/26.325/27.298 ms / #


### Describe the results you expected

I expected a reply from all `ping` commands.

### podman info output

```yaml
host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-2.fc37.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 99.65
    systemPercent: 0.16
    userPercent: 0.19
  cpus: 20
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: server
    version: "37"
  eventLogger: journald
  hostname: betachess-server
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.2.15-200.fc37.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 22455615488
  memTotal: 33491095552
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.4-1.fc37.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.4
      commit: 5a8fa99a5e41facba2eda4af12fa26313918805b
      rundir: /run/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8589930496
  swapTotal: 8589930496
  uptime: 23h 47m 20.00s (Approximately 0.96 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 4
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /mnt/largeraid/containers/storage
  graphRootAllocated: 4000797868032
  graphRootUsed: 430091313152
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 7
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /mnt/largeraid/containers/storage/volumes
version:
  APIVersion: 4.5.0
  Built: 1681486976
  BuiltTime: Fri Apr 14 17:42:56 2023
  GitCommit: ""
  GoVersion: go1.19.7
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.0

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

No

Additional environment details

No response

Additional information

This is the contents of /etc/resolv.conf in alp2:

search dns.podman
nameserver 10.89.1.1

I would also like to clarify that the issue is not specific to Alpine (I am running Nextcloud with the same issues), I just chose this as a example because of easy testing.

in addition I am runnning the latest version of Tailscale.

Luap99 commented 1 year ago

Is aardvark-dns running when the containers are up? Do you have something else already running on 0.0.0.0:53?

BetaChess commented 1 year ago

When I check with lsof I find the following:

systemd-r  1079 systemd-resolve   16u  IPv4   19901      0t0  UDP 127.0.0.53:53
systemd-r  1079 systemd-resolve   17u  IPv4   19902      0t0  TCP 127.0.0.53:53 (LISTEN)
systemd-r  1079 systemd-resolve   18u  IPv4   19903      0t0  UDP 127.0.0.54:53
systemd-r  1079 systemd-resolve   19u  IPv4   19904      0t0  TCP 127.0.0.54:53 (LISTEN)
aardvark-  1595            root   12u  IPv4   55515      0t0  UDP 10.5.0.1:53
aardvark-  1595            root   13u  IPv4   43983      0t0  UDP 10.89.1.1:53
Luap99 commented 1 year ago

It shows aardvark-dns listening on 10.89.1.1:53 so that is good, I would assume things should just work. Do you see any aardvark-dns errors reported in journald?

BetaChess commented 1 year ago

So I grepped for aardvark and there's obviously a lot, so I'm just gonna post a snippet, but it definitely seems like something is wrong here, but I'm quite new to linux, so I'm a bit perplexed as to what exactly:

May 05 21:49:44 betachess-server podman[3879]: [INFO  netavark::commands::setup] dns disabled because aardvark-dns path does not exists
May 05 21:49:53 betachess-server podman[3971]: [INFO  netavark::commands::setup] dns disabled because aardvark-dns path does not exists
May 05 21:49:55 betachess-server podman[4055]: [INFO  netavark::commands::setup] dns disabled because aardvark-dns path does not exists
May 05 21:49:56 betachess-server podman[4149]: [INFO  netavark::commands::setup] dns disabled because aardvark-dns path does not exists
May 05 21:50:41 betachess-server podman[4497]: [INFO  netavark::commands::setup] dns disabled because aardvark-dns path does not exists
May 05 21:54:04 betachess-server podman[4943]: [INFO  netavark::commands::setup] dns disabled because aardvark-dns path does not exists
May 05 22:01:16 betachess-server podman[5559]: [INFO  netavark::commands::setup] dns disabled because aardvark-dns path does not exists
May 05 22:02:06 betachess-server podman[5704]: [INFO  netavark::commands::setup] dns disabled because aardvark-dns path does not exists
May 05 22:54:27 betachess-server systemd[6888]: Started run-r2aa73ef4ae874fb2a6132b99244121a2.scope - /usr/libexec/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run.
May 05 22:58:20 betachess-server systemd[6888]: Started run-rb55e71d428c044d083bb193dab4c8e3a.scope - /usr/libexec/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run.
May 05 23:47:02 betachess-server sudo[23352]: betachess : TTY=pts/0 ; PWD=/home/betachess/compose-files/nextcloud_copy ; USER=root ; COMMAND=/usr/bin/dnf install dnf install podman python3-pip netavark aardvark-dns
May 05 23:47:10 betachess-server sudo[23368]: betachess : TTY=pts/0 ; PWD=/home/betachess/compose-files/nextcloud_copy ; USER=root ; COMMAND=/usr/bin/dnf install podman python3-pip netavark aardvark-dns
May 06 00:01:17 betachess-server systemd[6888]: Stopping run-rb55e71d428c044d083bb193dab4c8e3a.scope - /usr/libexec/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run...
May 06 00:01:17 betachess-server systemd[6888]: Stopped run-rb55e71d428c044d083bb193dab4c8e3a.scope - /usr/libexec/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run.
May 15 19:56:20 betachess-server systemd[1]: Stopping run-r32ae8aafa911421185f4b6371e61b2f2.scope - /usr/libexec/podman/aardvark-dns --config /run/containers/networks/aardvark-dns -p 53 run...
May 15 19:56:20 betachess-server systemd[1]: Stopped run-r32ae8aafa911421185f4b6371e61b2f2.scope - /usr/libexec/podman/aardvark-dns --config /run/containers/networks/aardvark-dns -p 53 run.
May 15 19:56:30 betachess-server podman-compose[96101]: time="2023-05-15T19:56:30+02:00" level=error msg="Unable to clean up network for container f46381f099d75569082f863ce8051d7a9f5f82aa137494cfca37f3baf6c9ebd4: \"tearing down network namespace configuration for container f46381f099d75569082f863ce8051d7a9f5f82aa137494cfca37f3baf6c9ebd4: netavark: IO error: aardvark pid not found\""
May 15 19:56:35 betachess-server podman-compose[96965]: time="2023-05-15T19:56:35+02:00" level=error msg="Unable to clean up network for container 190ab697d0681952da2ee2c5a4f6e4bfac6618480e8467afa9d0ef4009d2dc56: \"tearing down network namespace configuration for container 190ab697d0681952da2ee2c5a4f6e4bfac6618480e8467afa9d0ef4009d2dc56: netavark: IO error: aardvark pid not found\""
May 15 19:58:21 betachess-server systemd[1]: Started run-r114cf9e2f869445da19d7b2a2f449dff.scope - /usr/libexec/podman/aardvark-dns --config /run/containers/networks/aardvark-dns -p 53 run.
May 26 19:29:15 betachess-server systemd[1]: Stopping run-r114cf9e2f869445da19d7b2a2f449dff.scope - /usr/libexec/podman/aardvark-dns --config /run/containers/networks/aardvark-dns -p 53 run...
May 26 19:29:15 betachess-server systemd[1]: Stopped run-r114cf9e2f869445da19d7b2a2f449dff.scope - /usr/libexec/podman/aardvark-dns --config /run/containers/networks/aardvark-dns -p 53 run.
May 26 19:29:26 betachess-server podman-compose[362995]: time="2023-05-26T19:29:26+02:00" level=error msg="Unable to clean up network for container 9c8a6d51abf773d0a67669822ab2eb26484352a23d205322b475872b68414a02: \"tearing down network namespace configuration for container 9c8a6d51abf773d0a67669822ab2eb26484352a23d205322b475872b68414a02: netavark: IO error: aardvark pid not found\""
May 26 19:29:31 betachess-server podman-compose[363480]: time="2023-05-26T19:29:31+02:00" level=error msg="Unable to clean up network for container d79faf152a90423703f3fe9ad747ceee3aa861d1b8c92a5d4d89ad9da1c02e6d: \"tearing down network namespace configuration for container d79faf152a90423703f3fe9ad747ceee3aa861d1b8c92a5d4d89ad9da1c02e6d: netavark: IO error: aardvark pid not found\""
May 26 19:30:19 betachess-server systemd[1]: Started run-r23da2d64025e49ba9dede9ffce91722d.scope - /usr/libexec/podman/aardvark-dns --config /run/containers/networks/aardvark-dns -p 53 run.
Jun 03 14:49:37 betachess-server systemd[1]: Stopping run-r23da2d64025e49ba9dede9ffce91722d.scope - /usr/libexec/podman/aardvark-dns --config /run/containers/networks/aardvark-dns -p 53 run...
Jun 03 14:49:37 betachess-server systemd[1]: Stopped run-r23da2d64025e49ba9dede9ffce91722d.scope - /usr/libexec/podman/aardvark-dns --config /run/containers/networks/aardvark-dns -p 53 run.
Jun 03 14:49:47 betachess-server podman-compose[51158]: time="2023-06-03T14:49:47+02:00" level=error msg="Unable to clean up network for container ab744ffa6a1952f759fdf8742934b7f7da0a227c92908ad242292a7187200b8e: \"tearing down network namespace configuration for container ab744ffa6a1952f759fdf8742934b7f7da0a227c92908ad242292a7187200b8e: netavark: IO error: aardvark pid not found\""
Jun 03 14:49:52 betachess-server podman-compose[51604]: time="2023-06-03T14:49:52+02:00" level=error msg="Unable to clean up network for container 0ca69341d7d4345f8469df9be963df85faa26592c275dd0eb7509d545ff493ab: \"tearing down network namespace configuration for container 0ca69341d7d4345f8469df9be963df85faa26592c275dd0eb7509d545ff493ab: netavark: IO error: aardvark pid not found\""
Jun 03 14:50:41 betachess-server systemd[1]: Started run-rce74a4f071cf49cb949c3f9ace1d61bb.scope - /usr/libexec/podman/aardvark-dns --config /run/containers/networks/aardvark-dns -p 53 run.
Jun 03 15:33:19 betachess-server systemd[3198]: Started run-r07aa68de6cf743d283b0e96fc7d442d8.scope - /usr/libexec/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run.
Jun 03 15:33:21 betachess-server systemd[3198]: Started run-r175c85c04ed64ca7aa639b277e67844b.scope - /usr/libexec/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run.
Jun 03 15:33:36 betachess-server systemd[3198]: Started run-r140d53a407584db8973949b2cd91fd45.scope - /usr/libexec/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run.
Jun 03 15:33:36 betachess-server aardvark-dns[12140]: Received SIGHUP will refresh servers: 1
Jun 03 15:33:36 betachess-server aardvark-dns[12140]: No configuration found stopping the sever
Jun 03 15:36:16 betachess-server systemd[3198]: Started run-r59c15b86186842db9d06e5f81ea69994.scope - /usr/libexec/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run.
Jun 03 15:38:32 betachess-server systemd[3198]: Started run-ra9fd04f704b9483c8bc82ee480ad4cef.scope - /usr/libexec/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run.
Jun 03 15:38:33 betachess-server systemd[3198]: Started run-r7c8e72645d044a12b3f7b7bd3c109c69.scope - /usr/libexec/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run.
Jun 03 15:38:44 betachess-server systemd[3198]: Started run-r1f936b0f135d4e148cd7c9753ebe6302.scope - /usr/libexec/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run.
Jun 03 15:38:44 betachess-server aardvark-dns[13235]: Received SIGHUP will refresh servers: 1
Jun 03 15:38:44 betachess-server aardvark-dns[13235]: No configuration found stopping the sever
Jun 03 15:40:37 betachess-server systemd[3198]: Started run-r76360ab8fc50487bab70d0bbd756a283.scope - /usr/libexec/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run.
Jun 03 16:39:54 betachess-server systemd[1]: Stopping run-rce74a4f071cf49cb949c3f9ace1d61bb.scope - /usr/libexec/podman/aardvark-dns --config /run/containers/networks/aardvark-dns -p 53 run...
Jun 03 16:39:55 betachess-server systemd[1]: Stopped run-rce74a4f071cf49cb949c3f9ace1d61bb.scope - /usr/libexec/podman/aardvark-dns --config /run/containers/networks/aardvark-dns -p 53 run.
Jun 03 16:40:05 betachess-server podman-compose[18747]: time="2023-06-03T16:40:05+02:00" level=error msg="Unable to clean up network for container eab7aba9696b246e6eeb7d6dc7be4d7d12aa6bbb8320622a22f737f0ee698bc6: \"tearing down network namespace configuration for container eab7aba9696b246e6eeb7d6dc7be4d7d12aa6bbb8320622a22f737f0ee698bc6: netavark: IO error: aardvark pid not found\""
Jun 03 16:40:09 betachess-server podman-compose[19297]: time="2023-06-03T16:40:09+02:00" level=error msg="Unable to clean up network for container 9c8cf9945a0c5e430fb293d3e9661bb0671f7b735e839c08607ff1f9ded0b3db: \"tearing down network namespace configuration for container 9c8cf9945a0c5e430fb293d3e9661bb0671f7b735e839c08607ff1f9ded0b3db: netavark: IO error: aardvark pid not found\""
Jun 03 16:41:54 betachess-server systemd[1]: Started run-r7a07bfc111264b1f9123c13dbceccb2c.scope - /usr/libexec/podman/aardvark-dns --config /run/containers/networks/aardvark-dns -p 53 run.
Jun 05 22:41:24 betachess-server systemd[31045]: Started run-rc8c38893fec54ef2951059cc22b81c79.scope - /usr/libexec/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run.
Jun 05 22:42:30 betachess-server systemd[31045]: Started run-r8ecdeecace714e3a8a02f9d375115062.scope - /usr/libexec/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run.
Jun 05 22:44:52 betachess-server podman[33002]: [INFO  netavark::commands::setup] dns disabled because aardvark-dns path does not exists
Jun 05 22:49:19 betachess-server systemd[31045]: Started run-rec9e499427cb43e4a1d20222c2da2098.scope - /usr/libexec/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run.
Jun 05 22:57:07 betachess-server systemd[1]: Stopping run-r7a07bfc111264b1f9123c13dbceccb2c.scope - /usr/libexec/podman/aardvark-dns --config /run/containers/networks/aardvark-dns -p 53 run...
Jun 05 22:57:08 betachess-server systemd[1]: Stopped run-r7a07bfc111264b1f9123c13dbceccb2c.scope - /usr/libexec/podman/aardvark-dns --config /run/containers/networks/aardvark-dns -p 53 run.
Jun 05 22:57:18 betachess-server podman-compose[37297]: time="2023-06-05T22:57:18+02:00" level=error msg="Unable to clean up network for container 23b0a5e1a54780536ff2eb93c167b4e62cf92e81a760c7a26f780d6f61a5b58d: \"tearing down network namespace configuration for container 23b0a5e1a54780536ff2eb93c167b4e62cf92e81a760c7a26f780d6f61a5b58d: netavark: IO error: aardvark pid not found\""
Jun 05 22:57:22 betachess-server podman-compose[37843]: time="2023-06-05T22:57:22+02:00" level=error msg="Unable to clean up network for container cf4d9d2c003b92837f4b8c108c8090362364fa78914cc8fc05daf2ed51df7203: \"tearing down network namespace configuration for container cf4d9d2c003b92837f4b8c108c8090362364fa78914cc8fc05daf2ed51df7203: netavark: IO error: aardvark pid not found\""
Jun 05 22:58:12 betachess-server systemd[1]: Started run-r775ff09ce3ad479e9512d8171715f626.scope - /usr/libexec/podman/aardvark-dns --config /run/containers/networks/aardvark-dns -p 53 run.
BetaChess commented 1 year ago

Is there anything I can do to fix the issue on my end?

Luap99 commented 1 year ago

Can you reproduce this with plain podman commands? Or rather since podman-compose is execing podman can show the commands. Does it reproduce when you use docker-compose instead?

From your log I see dns disabled because aardvark-dns path does not exists although this is a very old entry and below it shows dns is started, if you do podman network inspect does it say dns_enabled: true?

What you can try is when the containers are running check the content of /run/containers/networks/aardvark-dns There should be afile with your network name and it should contain all dns entries for the containers.

BetaChess commented 1 year ago

For full clarity, I've removed all containers and networks and started fresh. Then I ran the following commands:

[root@betachess-server betachess]# podman ps
CONTAINER ID  IMAGE       COMMAND     CREATED     STATUS      PORTS       NAMES
[root@betachess-server betachess]# podman network ls
NETWORK ID    NAME        DRIVER
2f259bab93aa  podman      bridge
[root@betachess-server betachess]# podman network create alpnet
alpnet
[root@betachess-server betachess]# podman run -itd --network alpnet --name alp1 alpine:latest
8877332aae81f98dc7b93858ff190529904f21a06452c34df3567778cb7d9b2d
[root@betachess-server betachess]# podman run -itd --network alpnet --name alp2 alpine:latest
7626ab9ce4d6c9f9c7950039082f921dfe2068d4a33a21837f6ab8124a7db8a1

Problem persists. Running podman network inspect yields:

[
     {
          "name": "alpnet",
          "id": "5373021c522361c74ee9a7035baeea5a78f86c41687eb17f51c26a2b12d5768f",
          "driver": "bridge",
          "network_interface": "podman1",
          "created": "2023-06-19T17:47:28.03593123+02:00",
          "subnets": [
               {
                    "subnet": "10.89.0.0/24",
                    "gateway": "10.89.0.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": true,
          "ipam_options": {
               "driver": "host-local"
          }
     }
]

As for the contents of /run/containers/networks/aardvark-dns/alpnet, they seem to be correct?

10.89.0.1
8877332aae81f98dc7b93858ff190529904f21a06452c34df3567778cb7d9b2d 10.89.0.2  alp1,8877332aae81
7626ab9ce4d6c9f9c7950039082f921dfe2068d4a33a21837f6ab8124a7db8a1 10.89.0.3  alp2,7626ab9ce4d6

Once again. I am stumped. Trying with another image, busybox (podman run -itd --network alpnet --name busy busybox:latest), doesn't work either.

Luap99 commented 1 year ago

That looks good, can you try to query the resolver from the host just to see if it responds at all, i.e. nslookup apl1 10.89.0.1

BetaChess commented 1 year ago

From alp2:

/ # nslookup apl1 10.89.0.1
;; connection timed out; no servers could be reached
Luap99 commented 1 year ago

Weird and ss -ulpn shows that aardvark-dns is listening there?

BetaChess commented 1 year ago

Certainly looks like it:

[betachess@betachess-server ~]$ sudo ss -ulpn | grep 53
UNCONN 0      0                                                 10.89.0.1:53         0.0.0.0:*    users:(("aardvark-dns",pid=2743,fd=13))
github-actions[bot] commented 1 year ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 1 year ago

@BetaChess @Luap99 is this still an issue?

BetaChess commented 1 year ago

Yes. For now, I have switched to docker, which seems to be working as intented. I have no clue why I couldn't get it to work, but if I find the time, I'll see if I can find a way to replicate the issue from a fresh OS install.

github-actions[bot] commented 11 months ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 11 months ago

Ok closing issue, unless we get a reproducer.

ItalyPaleAle commented 10 months ago

I have the exact same issue as @BetaChess when running on Debian 12. I tried podman from Debian's testing repos as well as from Alvistack.

DNS within a container doesn't work at all when using a custom network (but connecting to an IP works). But it works when using the default bridge.

~ podman info
host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon_100:2.1.8-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 00e08f4a9ca5420de733bf542b930ad58e1a7e7d'
  cpuUtilization:
    idlePercent: 75.82
    systemPercent: 9.09
    userPercent: 15.08
  cpus: 6
  databaseBackend: boltdb
  distribution:
    codename: bookworm
    distribution: debian
    version: "12"
  eventLogger: file
  freeLocks: 2011
  hostname: truffle
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.1.0-13-amd64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 3452153856
  memTotal: 8153362432
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-5_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.4.0
    package: netavark_1.4.0-4_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: crun
    package: crun_100:1.9.2-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.9.2
      commit: 35274d346d2e9ffeacb22cc11590b0266a23d634
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: true
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: true
    capabilities: CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_MKNOD,CAP_NET_BIND_SERVICE,CAP_NET_RAW,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_100:1.2.2-1_amd64
    version: |-
      slirp4netns version 1.2.2
      commit: unknown
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 7998533632
  swapTotal: 7998533632
  uptime: 0h 7m 46.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 22
    paused: 0
    running: 21
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 244939624448
  graphRootUsed: 24350150656
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 13
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.7.1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.21.3
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.1
ItalyPaleAle commented 10 months ago

An update is that switching to CNI as network backend works, and with the DNS plugin resolution works internally and externally