podman: userspace networking DNS broken when network has dns_enabled=true

[acuteaura]

Describe the bug

Userspace networking DNS does not work when running Podman in rootless mode and a pod is attached to a network with dns_enabled: true. This is not usually the default network, but using compose will create one with it enabled by default.

Logs show messages like:

aardvark-dns[1173831]: 59974 dns request got empty response

Steps To Reproduce

  virtualisation.containers.enable = true;
  virtualisation.podman = {
    enable = true;
    package = unstable.podman;
    dockerSocket.enable = true;
  };

  environment.systemPackages = with pkgs; [
    docker-client
    docker-compose
  ];

  networking.firewall.enable = false;

podman network create test
podman run --rm -it --network test alpine
wget google.com # broken
wget 1.1.1.1 # works

Expected behavior

Networking works.

Additional context

Problem also reported here: https://discourse.nixos.org/t/no-dns-resolution-in-user-rootless-podman-container/38665

Might be related to using resolved, but I haven't extensively tested this yet.

Notify maintainers

@marsam @adisbladis @saschagrunert @vdemeester

Metadata

 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.17, NixOS, 23.11 (Tapir), 23.11.20240218.e0da498`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

(podman package is locked to unstable, b98a4e1746acceb92c509bc496ef3d0e5ad8d4aa)

---

Add a :+1: reaction to issues you find important.

[hrenard]

I've hit this issue after enabling networking.useNetworkd. I guess aardvark have trouble connecting to systemd's dns server.

What is your network/dns setup @acuteaura ?

[acuteaura]

I think I switched to docker. I don't remember, I don't actively use NixOS anymore. For reasons.