No DNS inside podman containers of gitea-actions-runner #340479

Closed malikwirin closed 1 month ago

Describe the bug

No Domain is reachable when the podman Container is started by the gitea-actions-runner service. Therefore all workflows fail.

Steps To Reproduce

Steps to reproduce the behavior:

Configure DNS, Podman, and forgejo-actions-runner like described in the wiki
Write a workflow
See it failing inside your forgejo instance because of missing dns

Expected behavior

Normal name resolution inside the container

Additional context

My forgejo-runner configurration:

{ pkgs, config, ... }:

{
  services.gitea-actions-runner = {
    package = pkgs.forgejo-actions-runner;
    instances.default = {
      enable = true;
      name = config.networking.hostName;
      url = "https://codeberg.org";
      token = "******************************************";
      labels = [
        "ubuntu-latest:docker://node:16-bullseye"
        "nix:docker://nixos/nix:latest"
        ## optionally provide native execution on the host:
        # "native:host"
      ];
    };
  };
}

My podman configuration:

{ ... }:

{
  virtualisation = {
    containers.enable = true;

    podman = {
      enable = true;
      # Create a `docker` alias for podman, to use it as a drop-in replacement
      dockerCompat = true;
      # Replace docker socket with podman
      dockerSocket.enable = true;

      defaultNetwork.settings = {
        # Required for containers under podman-compose to be able to talk to each other.
        dns_enabled = true;
      };
    };
  };
}

My Networking configuration:

networking = {
    hostName = "********";
    nameservers = [ "127.0.0.1" "::1" ];
    enableIPv6 = true;
    networkmanager = {
      enable = true;
      dns = "none";
    };

    dhcpcd.extraConfig = "nohook resolv.conf";
    resolvconf.useLocalResolver = true;

    firewall = {
      enable = true;
      allowedTCPPorts = [
        # config.services.headscale.port
      ] ++ config.services.openssh.ports;
    };
  };

My DNS configuration:

{
  services.dnscrypt-proxy2 = {
    enable = true;
    settings = {
      ipv6_servers = true;
      require_dnssec = true;

      sources.public-resolvers = {
        urls = [
          "https://raw.githubusercontent.com/DNSCrypt/dnscrypt-resolvers/master/v3/public-resolvers.md"
          "https://download.dnscrypt.info/resolvers-list/v3/public-resolvers.md"
        ];
        cache_file = "/var/lib/dnscrypt-proxy2/public-resolvers.md";
        minisign_key = "RWQf6LRCGA9i53mlYecO4IzT51TGPpvWucNSCh1CBM0QTaLn73Y7GFO3";
      };

      # You can choose a specific set of servers from https://github.com/DNSCrypt/dnscrypt-resolvers/blob/master/v3/public-resolvers.md
      # server_names = [ ... ];
    };
  };

  systemd.services.dnscrypt-proxy2.serviceConfig = {
    StateDirectory = "dnscrypt-proxy";
  };
}

Notify maintainers

@adamcstephens @emilylange @mweinelt @SuperSandro2000 @zowoq

Add a :+1: reaction to issues you find important.

I am extremely confident that this is because you set networking.nameservers = [ "127.0.0.1" "::1" ]; which podman discards because 127.0.0.1 in the container is the container itself and not your host's nameserver.

You can configure different nameservers just for podman. Please refer to its documentation instead.

I am extremely confident that this is because you set networking.nameservers = [ "127.0.0.1" "::1" ]; which podman discards because 127.0.0.1 in the container is the container itself and not your host's nameserver.

You can configure different nameservers just for podman. Please refer to its documentation instead.

When outputting the content of /etc/resolv.conf inside the container I get the following.

search dns.podman
nameserver 10.89.1.1
options edns0 trust-ad

So it tries to use the dns server of the host propably. Do I maybe have to open the port for it from the host to the podman network?

But at the same time I don't have dns problems when running a container as my normal user.

I am debuggin this for two days already @emilylange of course I also tried setting nameserver to only the cloudflare ip I reboot after every rebuild and before every test

I hope I am not claiming to many peoples time but this has kind of a high priority due to beeing a massive roadblock @afh @CyberShadow @Kranzes

In my experience any DNS server not in the network needs usually some custom config for containers to work. I would suggest to use tcpdump to debug until where the packet is roughly going and then iptables counters to know which rules drops it.

The problem was that even after switching from Podman to docker I still had podman networking settings enabled

NixOS / nixpkgs