netavark dns resolves container fqdn on only one network when multiple networks connected

ykuksenko commented 2 years ago

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When a container is connected to multiple networks only one network can resolve containers by fqdn.

Steps to reproduce the issue:

create two networks

podman network create --subnet 192.168.55.0/24 network1
podman network create --subnet 192.168.56.0/24 network2

start up a container that will be resolved in dns attached only to network1 and have it do nothing
```
podman run --detach --rm -ti --name container1 --network network1 alpine sleep 9000
```

create a container attached to both networks and run dig to resolve the fqdn of the first container against both network dns servers in resolv.conf

podman run --rm -ti --name container2 --network network1,network2 alpine sh -c "cat /etc/resolv.conf; apk add bind-tools > /dev/null; echo '<<<<<<<<<<< network1 dns test'; dig container1.dns.podman @192.168.55.1; echo '<<<<<<<<<<< network2 dns test'; dig container1.dns.podman @192.168.56.1"

repeat number 3 with short names

podman run --rm -ti --name container2 --network network1,network2 alpine sh -c "cat /etc/resolv.conf; apk add bind-tools > /dev/null; echo '<<<<<<<<<<< network1 dns test'; dig container1 @192.168.55.1; echo '<<<<<<<<<<< network2 dns test'; dig container1 @192.168.56.1"

Describe the results you received: When resolving the fqdn of container1 only one name server responds correctly.

search dns.podman dns.podman
nameserver 192.168.55.1
nameserver 192.168.56.1
nameserver 192.168.121.1
<<<<<<<<<<< network1 dns test

... (clipped for clarity)
;; QUESTION SECTION:
;container1.dns.podman.         IN      A

;; ANSWER SECTION:
container1.dns.podman.  86400   IN      A       192.168.55.2

;; Query time: 1 msec
;; SERVER: 192.168.55.1#53(192.168.55.1)
;; WHEN: Tue May 24 14:37:03 UTC 2022
;; MSG SIZE  rcvd: 78

<<<<<<<<<<< network2 dns test

... (clipped for clarity)
;; QUESTION SECTION:
;container1.dns.podman.         IN      A

;; Query time: 3 msec
;; SERVER: 192.168.56.1#53(192.168.56.1)
;; WHEN: Tue May 24 14:37:03 UTC 2022
;; MSG SIZE  rcvd: 62

When resolving the short name of the container one both name server respond correctly

search dns.podman dns.podman
nameserver 192.168.56.1
nameserver 192.168.55.1
nameserver 192.168.121.1
<<<<<<<<<<< network1 dns test

... (clipped for clarity)
;; QUESTION SECTION:
;container1.                    IN      A

;; ANSWER SECTION:
container1.             86400   IN      A       192.168.55.2

;; Query time: 2 msec
;; SERVER: 192.168.55.1#53(192.168.55.1)
;; WHEN: Tue May 24 14:38:01 UTC 2022
;; MSG SIZE  rcvd: 67

<<<<<<<<<<< network2 dns test

... (clipped for clarity)
;; QUESTION SECTION:
;container1.                    IN      A

;; ANSWER SECTION:
container1.             86400   IN      A       192.168.55.2

;; Query time: 3 msec
;; SERVER: 192.168.56.1#53(192.168.56.1)
;; WHEN: Tue May 24 14:38:01 UTC 2022
;; MSG SIZE  rcvd: 67

Describe the results you expected: Both name servers should respond to both the shortname and fqdn queries.

Additional information you deem important (e.g. issue happens only occasionally):

/etc/resolv.conf entries show up unsorted so if a user uses fqdns to reference containers they will have intermittent issues in applications. ( see https://github.com/containers/podman/issues/14262 )
It is also notable that /etc/resolv.conf lists the dns suffix dns.podman twice - this probably does not harm anything but should probably be deduplicated by podman.

Output of podman version:

# podman version
Client:       Podman Engine
Version:      4.1.0
API Version:  4.1.0
Go Version:   go1.18
Built:        Fri May  6 16:15:54 2022
OS/Arch:      linux/amd64

Output of podman info --debug:

# podman info  --debug
host:
  arch: amd64
  buildahVersion: 1.26.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.0-2.fc36.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpuUtilization:
    idlePercent: 97.24
    systemPercent: 0.93
    userPercent: 1.83
  cpus: 2
  distribution:
    distribution: fedora
    variant: cloud
    version: "36"
  eventLogger: journald
  hostname: container.redacted
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.17.5-300.fc36.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 148381696
  memTotal: 6217089024
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.4-1.fc36.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.4
      commit: 6521fcc5806f20f6187eb933f9f45130c86da230
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64
    version: |-
      slirp4netns version 1.2.0-beta.0
      commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 5851836416
  swapTotal: 6217003008
  uptime: 15h 16m 15.62s (Approximately 0.62 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
    number: 19
    paused: 0
    running: 19
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 41788899328
  graphRootUsed: 9318744064
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 67
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.1.0
  Built: 1651853754
  BuiltTime: Fri May  6 16:15:54 2022
  GitCommit: ""
  GoVersion: go1.18
  Os: linux
  OsArch: linux/amd64
  Version: 4.1.0

Package info (e.g. output of rpm -q podman or apt list podman):

# rpm -q netavark podman
netavark-1.0.3-3.fc36.x86_64
podman-4.1.0-1.fc36.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.): Libvirt VM

mheon commented 2 years ago

@flouthoc PTAL

flouthoc commented 2 years ago

@ykuksenko I think container1 is only connected to network1 please connect it to network2 as well in order to get it resolved from nameserver in network2. So podman run --detach --rm -ti --name container1 --network network1,network2 alpine sleep 9000 should be the right thing here.

OTOH you could say that bug is that short name should not get resolved by nameserver if container is not connected to the network of nameserver.

ykuksenko commented 2 years ago

container1 is intentionally connected to only network1 for isolation. For some additional context of where I ran into this issue, I use the following scenario so that the database is more isolated from the load balancer:

Load Balancer  ------------> Application -------------> Database
                (network1)                (network2)

In this scenarios the application container sometimes fails to resolve the database when addressed with FQDN. I prefer using FQDNs for names because it makes it really obvious what the term refers to and that the name is meant to be internally resolved, and not by my main DNS server.

I suppose the biggest thing I would like to see is consistent behavior:

If shortnames should be resolvable universally, then FQDNs should be too. (preferred solution)
If FQDNs should not be resolvable universally then shortnames should not be too.

I do not see any reason for shortnames to resolve and FQDNs to fail or vise versa. If there is such a reason could you share it?

I prefer having both FQDNs and shortnames to be resolvable because:

the autogenerated /etc/resolv.conf file randomly orders podman network name servers which causes unpredictability
FQDNs avoid name ambiguity if the short name is defined in the host DNS servers. Or if DNS suffixes are utilized (not sure if these are configurable in netavark/aardvark)
it is convenient
it avoids having tweaking /etc/resolv.conf to try multiple DNS servers in case the first one cannot respond and response durations. (With the alpine image, if the first server cannot respond pings simply fail saying bad address, same with Keycloak trying to connect to a database)
Podman container names are unique so any server being able to respond on behalf of any attached network for internal resources seems sound. (This brings up the question of why have more than one server if they act as one anyway but this is not important)
I am not aware of any down side to this. I am missing one?

I am under the impression that netavark and aardvark-dns should combine DNS results from any network a container is connected to according to @mheon (https://github.com/containers/podman/issues/14262#issuecomment-1127977648)

flouthoc commented 2 years ago

@ykuksenko Does the requested feature works on docker cause this seems a bug a too me I think we should fix this otherway around and make short names also not resolvable if they are not part of the network.

baude commented 2 years ago

should this be moved to nv?

ykuksenko commented 2 years ago

Sorry for the delay. I do not usually use docker these days.

Docker works a bit differently in this area but it does work intuitively, at least for me.

Docker seems to only allow creating a container with one network attached. After creating you can separately run docker network connect $network_name $container_name to add the second network. I was not able to find another way with a quick look.
Docker provides only one dns server in /etc/resolv.conf: even when multiple networks are attached. This is the same across all containers
```
/ # cat /etc/resolv.conf
search mydomain.com vagrant.mydomain.com
nameserver 127.0.0.11
options edns0 trust-ad ndots:0
```
The original container automatically gets its short name and a long name that is specific to its network. Not a generic name like podman provides. So container1 from this example resolves at container1 and container1.network1.
Since container1 is only connected to one network I created both container2 and container3 that are both connected to both network1 and network2 and tested resolving container1 and container2 from container3. As there is only one dns server this is simpler and avoids having the OS figure out which nameserver is responsible for what as well as any timeouts associated with that.

From container3: the following names resolve: container1 -> 192.168.55.2 container1.network1 -> 192.168.55.2 container1.network2 -> bad address (as it should be as container1 is not connected to network2 container2 -> 192.168.55.3 container2.network1 -> 192.168.55.3 contaner2.network2 -> 192.168.56.2

From container1 things also resolve as expected. Namely only other containers in network1 with both shortnames and fqdns resolve successfully.

Every name resolved to one and only one IP. A short name never returned both network1 and network2 names from any container.

Here is the list of commands I ran for setup.

docker network create --subnet 192.168.55.0/24 network1
docker network create --subnet 192.168.56.0/24 network2
docker run --detach --rm -ti --name container1 --network network1 alpine sleep 9000
docker run --rm -ti --name container2 -d --network network1 alpine sleep 9000
docker network connect network2 container2
docker run --rm -ti --name container3 -d --network network1 alpine sleep 9000
docker network connect network2 container3

From there I used the following commands to test:

docker exec -ti container1 apk add bind-tools
docker exec -ti container2 apk add bind-tools
docker exec -ti container3 apk add bind-tools
docker exec -ti container1 cat /etc/resolv.conf     #can see above
docker exec -ti container2 cat /etc/resolv.conf     #can see above
docker exec -ti container3 cat /etc/resolv.conf     #can see above

docker exec -ti container1 dig +short container1             #-> 192.168.55.2
docker exec -ti container1 dig +short container1.network1    #-> 192.168.55.2
docker exec -ti container1 dig +short container1.network2    #-> blank (entry should not exist)
docker exec -ti container1 dig +short container3.network2    #-> blank (container1 is not in network2 so access  is denied)

docker exec -ti container3 dig +short container1             #-> 192.168.55.2
docker exec -ti container3 dig +short container1.network1    #-> 192.168.55.2
docker exec -ti container3 dig +short container1.network2    #-> blank (entry should not exist)
docker exec -ti container3 dig +short container2             #-> 192.168.55.3
docker exec -ti container3 dig +short container2.network1    #-> 192.168.55.3
docker exec -ti container3 dig +short container2.network2    #-> 192.168.56.2
docker exec -ti container3 dig +short container3             #-> 192.168.55.4

Hopefully this covers everything.

edit: my docker version: Docker version 20.10.16, build aa7e414 on fedora 36 edit2: change second sentence to be less presumptuous.

github-actions[bot] commented 2 years ago

A friendly reminder that this issue had no activity for 30 days.

felixsanz commented 1 year ago

Bump. Happening to me and it's very annoying! @flouthoc I can confirm this works as expected in docker, been using this networking stuff for years.

In fact this issue was first reported early 2021 here: containers/podman#9492

openshift-ci[bot] commented 11 months ago

@ykuksenko: The label(s) kind/bug cannot be applied, because the repository doesn't have them.

In response to [this](https://github.com/containers/aardvark-dns/issues/403): > > >**Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)** > >/kind bug > >**Description** > >When a container is connected to multiple networks only one network can resolve containers by fqdn. > >**Steps to reproduce the issue:** > >1. create two networks >``` >podman network create --subnet 192.168.55.0/24 network1 >podman network create --subnet 192.168.56.0/24 network2 >``` >2. start up a container that will be resolved in dns attached only to network1 and have it do nothing >``` >podman run --detach --rm -ti --name container1 --network network1 alpine sleep 9000 >``` >3. create a container attached to both networks and run dig to resolve the fqdn of the first container against both network dns servers in resolv.conf >``` >podman run --rm -ti --name container2 --network network1,network2 alpine sh -c "cat /etc/resolv.conf; apk add bind-tools > /dev/null; echo '<<<<<<<<<<< network1 dns test'; dig container1.dns.podman @192.168.55.1; echo '<<<<<<<<<<< network2 dns test'; dig container1.dns.podman @192.168.56.1" >``` >4. repeat number 3 with short names >``` >podman run --rm -ti --name container2 --network network1,network2 alpine sh -c "cat /etc/resolv.conf; apk add bind-tools > /dev/null; echo '<<<<<<<<<<< network1 dns test'; dig container1 @192.168.55.1; echo '<<<<<<<<<<< network2 dns test'; dig container1 @192.168.56.1" >``` > >**Describe the results you received:** >When resolving the fqdn of container1 only one name server responds correctly. >``` >search dns.podman dns.podman >nameserver 192.168.55.1 >nameserver 192.168.56.1 >nameserver 192.168.121.1 ><<<<<<<<<<< network1 dns test > >... (clipped for clarity) >;; QUESTION SECTION: >;container1.dns.podman. IN A > >;; ANSWER SECTION: >container1.dns.podman. 86400 IN A 192.168.55.2 > >;; Query time: 1 msec >;; SERVER: 192.168.55.1#53(192.168.55.1) >;; WHEN: Tue May 24 14:37:03 UTC 2022 >;; MSG SIZE rcvd: 78 > ><<<<<<<<<<< network2 dns test > >... (clipped for clarity) >;; QUESTION SECTION: >;container1.dns.podman. IN A > >;; Query time: 3 msec >;; SERVER: 192.168.56.1#53(192.168.56.1) >;; WHEN: Tue May 24 14:37:03 UTC 2022 >;; MSG SIZE rcvd: 62 > >``` > >When resolving the short name of the container one both name server respond correctly > >``` >search dns.podman dns.podman >nameserver 192.168.56.1 >nameserver 192.168.55.1 >nameserver 192.168.121.1 ><<<<<<<<<<< network1 dns test > >... (clipped for clarity) >;; QUESTION SECTION: >;container1. IN A > >;; ANSWER SECTION: >container1. 86400 IN A 192.168.55.2 > >;; Query time: 2 msec >;; SERVER: 192.168.55.1#53(192.168.55.1) >;; WHEN: Tue May 24 14:38:01 UTC 2022 >;; MSG SIZE rcvd: 67 > ><<<<<<<<<<< network2 dns test > >... (clipped for clarity) >;; QUESTION SECTION: >;container1. IN A > >;; ANSWER SECTION: >container1. 86400 IN A 192.168.55.2 > >;; Query time: 3 msec >;; SERVER: 192.168.56.1#53(192.168.56.1) >;; WHEN: Tue May 24 14:38:01 UTC 2022 >;; MSG SIZE rcvd: 67 > >``` > >**Describe the results you expected:** >Both name servers should respond to both the shortname and fqdn queries. > > >**Additional information you deem important (e.g. issue happens only occasionally):** >- /etc/resolv.conf entries show up unsorted so if a user uses fqdns to reference containers they will have intermittent issues in applications. ( see https://github.com/containers/podman/issues/14262 ) >- It is also notable that /etc/resolv.conf lists the dns suffix dns.podman twice - this probably does not harm anything but should probably be deduplicated by podman. > >**Output of `podman version`:** > >``` ># podman version >Client: Podman Engine >Version: 4.1.0 >API Version: 4.1.0 >Go Version: go1.18 >Built: Fri May 6 16:15:54 2022 >OS/Arch: linux/amd64 >``` > >**Output of `podman info --debug`:** > >``` ># podman info --debug >host: > arch: amd64 > buildahVersion: 1.26.1 > cgroupControllers: > - cpuset > - cpu > - io > - memory > - hugetlb > - pids > - misc > cgroupManager: systemd > cgroupVersion: v2 > conmon: > package: conmon-2.1.0-2.fc36.x86_64 > path: /usr/bin/conmon > version: 'conmon version 2.1.0, commit: ' > cpuUtilization: > idlePercent: 97.24 > systemPercent: 0.93 > userPercent: 1.83 > cpus: 2 > distribution: > distribution: fedora > variant: cloud > version: "36" > eventLogger: journald > hostname: container.redacted > idMappings: > gidmap: null > uidmap: null > kernel: 5.17.5-300.fc36.x86_64 > linkmode: dynamic > logDriver: journald > memFree: 148381696 > memTotal: 6217089024 > networkBackend: netavark > ociRuntime: > name: crun > package: crun-1.4.4-1.fc36.x86_64 > path: /usr/bin/crun > version: |- > crun version 1.4.4 > commit: 6521fcc5806f20f6187eb933f9f45130c86da230 > spec: 1.0.0 > +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL > os: linux > remoteSocket: > path: /run/podman/podman.sock > security: > apparmorEnabled: false > capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT > rootless: false > seccompEnabled: true > seccompProfilePath: /usr/share/containers/seccomp.json > selinuxEnabled: true > serviceIsRemote: false > slirp4netns: > executable: /usr/bin/slirp4netns > package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64 > version: |- > slirp4netns version 1.2.0-beta.0 > commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64 > libslirp: 4.6.1 > SLIRP_CONFIG_VERSION_MAX: 3 > libseccomp: 2.5.3 > swapFree: 5851836416 > swapTotal: 6217003008 > uptime: 15h 16m 15.62s (Approximately 0.62 days) >plugins: > log: > - k8s-file > - none > - passthrough > - journald > network: > - bridge > - macvlan > volume: > - local >registries: > search: > - docker.io >store: > configFile: /usr/share/containers/storage.conf > containerStore: > number: 19 > paused: 0 > running: 19 > stopped: 0 > graphDriverName: overlay > graphOptions: > overlay.mountopt: nodev,metacopy=on > graphRoot: /var/lib/containers/storage > graphRootAllocated: 41788899328 > graphRootUsed: 9318744064 > graphStatus: > Backing Filesystem: btrfs > Native Overlay Diff: "false" > Supports d_type: "true" > Using metacopy: "true" > imageCopyTmpDir: /var/tmp > imageStore: > number: 67 > runRoot: /run/containers/storage > volumePath: /var/lib/containers/storage/volumes >version: > APIVersion: 4.1.0 > Built: 1651853754 > BuiltTime: Fri May 6 16:15:54 2022 > GitCommit: "" > GoVersion: go1.18 > Os: linux > OsArch: linux/amd64 > Version: 4.1.0 >``` > >**Package info (e.g. output of `rpm -q podman` or `apt list podman`):** > >``` ># rpm -q netavark podman >netavark-1.0.3-3.fc36.x86_64 >podman-4.1.0-1.fc36.x86_64 >``` > >**Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)** > > >Yes > >**Additional environment details (AWS, VirtualBox, physical, etc.):** >Libvirt VM Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

rhatdan commented 1 year ago

@flouthoc @Luap99 @felixsanz is @ykuksenko Is this still an issue?

Luap99 commented 11 months ago

I move this to aardvark-dns, looks like this is still a problem

containers / aardvark-dns

netavark dns resolves container fqdn on only one network when multiple networks connected #403