search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
17.96k stars 13.98k forks source link

nixos/avahi-daemon: make mdns optional and off by default in nssDatabases.hosts to avoid slow dns resolution #291108

Open networkhermit opened 8 months ago

networkhermit commented 8 months ago

Describe the bug

After enabling nssmdns in avahi, the dns resolution in my nixos vm is very slow (above 5s), I believe it is caused by the last entry mdns in the nss hosts.

Steps To Reproduce

Steps to reproduce the behavior:

  1. Enable nssmdns in services.avahi in nixos configuration.nix:

    services.avahi = {
enable = true;
nssmdns = true;
publish = {
  enable = true;
  addresses = true;
};
};

  2. Run sudo nixos-rebuild switch and check the generated /etc/nsswitch.conf:

    hosts:     mymachines mdns_minimal [NOTFOUND=return] files myhostname dns mdns

  3. Ping some domain for test:

    
❯ time ping -c 1 -q github.com
PING github.com (20.205.243.166) 56(84) bytes of data.

--- github.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 90.951/90.951/90.951/0.000 ms

Executed in 5.47 secs fish external usr time 0.16 millis 159.00 micros 0.00 millis sys time 2.11 millis 186.00 micros 1.93 millis


4. Add some workaround in nixos configuration.nix:
```nix
system.nssDatabases.hosts = lib.mkAfter ["[!UNAVAIL=return]"];

  1. Run sudo nixos-rebuild switch and check the generated /etc/nsswitch.conf:

    hosts:     mymachines mdns_minimal [NOTFOUND=return] files myhostname dns [!UNAVAIL=return] mdns

  2. Ping the same domain again for test:

    
❯ time ping -c 1 -q github.com
PING github.com (20.205.243.166) 56(84) bytes of data.

--- github.com ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 90.492/90.492/90.492/0.000 ms

Executed in 124.60 millis fish external usr time 1.01 millis 312.00 micros 701.00 micros sys time 0.70 millis 0.00 micros 701.00 micros


### Expected behavior
No more slow dns resolution above 5s. Add an option to disable the last `mdns` entry and even make it the default.

### Screenshots
If applicable, add screenshots to help explain your problem.

### Additional context
In nixos-23.11 and nixos-unstable the `mdns` entry is unconditionally appended to the nss hosts:

https://github.com/NixOS/nixpkgs/blob/3cb4ae6689d2aa3f363516234572613b31212b78/nixos/modules/services/networking/avahi-daemon.nix#L269

https://github.com/NixOS/nixpkgs/blob/cbc4211f0afffe6dfd2478a62615dd5175a13f9a/nixos/modules/services/networking/avahi-daemon.nix#L286

### Notify maintainers

<!--
Please @ people who are in the `meta.maintainers` list of the offending package or module.
If in doubt, check `git blame` for whoever last touched something.
-->

### Metadata
Please run `nix-shell -p nix-info --run "nix-info -m"` and paste the result.

```console
[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.78, NixOS, 23.11 (Tapir), 23.11.4621.3cb4ae6689d2`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.1`
 - channels(root): `"nixos-23.11"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Add a :+1: reaction to issues you find important.

SuperSandro2000 commented 8 months ago

No more slow dns resolution above 5s. Add an option to disable the last mdns entry and even make it the default.

The only function of this option is to add that entry. On unstable the option got split into IPv4 and IPv6 and only IPv4 is recommended to be enabled according to upstream as IPv6 often times out, as you have observed.

Since this could be considered a breaking change, it didn't get backported.

networkhermit commented 8 months ago

The only function of this option is to add that entry.

Actually option nssmdns adds two nss host entries: mdns[4,6]_minimal and mdns[4,6]. The mdns_minimal module handles queries for the .local TLD only, and the full mdns module supports other TLDs.

I understand breaking changes won't get backported. But since there are ongoing breaking changes for this module, I think it's good time to add a new option for users to choose mdns[4,6]_minimal with or without the full mdns[4,6].

The following snippet is from nixos-unstable:

https://github.com/NixOS/nixpkgs/blob/548a86b335d7ecd8b57ec617781f5e652ab0c38e/nixos/modules/services/networking/avahi-daemon.nix#L275-L287

P.S.:

Example in Arch and Debian doesn't include the full mdns entry, but Gentoo did.

SuperSandro2000 commented 8 months ago

The dns entry is coming from https://github.com/NixOS/nixpkgs/blob/953cb16a5e703be81cf2ce4fee32465c83c80b11/nixos/modules/config/nsswitch.nix#L131

Example in Arch and Debian doesn't include the full mdns entry, but Gentoo did.

I quickly skimmed over the packages and they all do not contain explanations about why this should or should not be added or what are their thoughts on UNAVAIL=return.

nss-mdns does not recommend this in their example https://github.com/avahi/nss-mdns#activation

In systemd resolved there are more hints to not enable this https://github.com/systemd/systemd/issues/21123#issuecomment-1862931953

I've opened #291821 to remove this behaviour

We also set this currently for resolved https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/system/boot/resolved.nix#L144

I suspect something else is wrong with your DNS/NSS setup. Is the nameserver entry in /etc/resolv.conf working? I suspect that with this change the retry logic from nss switch gets disabled and the DNS resolution gets moved off your machine to the possibly the fallback server.

PS: It would be nice, if you would use coreutils time with less confusing formatting. The different units in time didn't help me better to easier understand the problem at hand.

networkhermit commented 8 months ago

On unstable the option got split into IPv4 and IPv6 and only IPv4 is recommended to be enabled according to upstream as IPv6 often times out, as you have observed.

I just manually tested the nssmdns handling logic of nixos unstable, even if I disable ipv6, it does not work. I firstly reproduced it in my archlinux workstation, then the nixos stable is the same. As nixos is highly reproducible, I doubts that there is something wrong with my DNS/NSS setup, but if it is the router, I can't control it. That said, in my setup the [!UNAVAIL=return] is still needed if I upgraded to nixos 24.05 in the future.

  1. Make nsswitch.con temporarily editable

    cd /etc
sudo mv nsswitch.conf nsswitch.conf.backup
sudo cp nsswitch.conf.backup nsswitch.conf

  2. Use mdns4_minimal and mdns4, emulate the behavior of nixos unstable, the dns resolution is still above 5s

    hosts:     mymachines mdns4_minimal [NOTFOUND=return] files myhostname dns mdns4
$ time ping -c1 -q github.com
PING github.com (20.205.243.166) 56(84) bytes of data.

--- github.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 90.643/90.643/90.643/0.000 ms

real    0m5.125s
user    0m0.001s
sys     0m0.001s
  1. Remove the trailing mdns4 entry, the dns resolution is below 100ms 
    hosts:     mymachines mdns4_minimal [NOTFOUND=return] files myhostname dns
$ time ping -c1 -q github.com
PING github.com (20.205.243.166) 56(84) bytes of data.

--- github.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 93.290/93.290/93.290/0.000 ms

real    0m0.206s
user    0m0.001s
sys     0m0.001s

P.S.

I changed the shell from fish to bash, so the timing format changed.

Some related article:

With Avahi upstream maintainer hat on, I would say it still makes sense to have separate mdns_minimal and mdns? modules. I would say mdns (non-minimal) should be rarely needed, because by default it should be used just for .local names.

Edited:

@SuperSandro2000

I quickly skimmed over the packages and they all do not contain explanations about why this should or should not be added or what are their thoughts on UNAVAIL=return.

I have not request to add [NOTFOUND=return] after dns in the nsswitch, that's just my workaround. What I really want to do is to remove/bypass the last full mdns/mdns4, because that entry greatly slows down my dns resolution. And the minimal version of nssmdns mdns*_minimal satisfies all my needs.

Also, I can confirm that both archlinux and fedora have resolve [!UNAVAIL=return] in their default nsswitch.conf (but not dns with [!UNAVAIL=return]).

SuperSandro2000 commented 7 months ago

@flokli can you help?

Atemu commented 7 months ago

Related: https://github.com/iputils/iputils/issues/491

networkhermit commented 7 months ago

Related: iputils/iputils#491

@Atemu Hello!

Have you noticed similar issue with other tools besides ping? Anyway, as ping is so ubiquitous and the full mdns (non-minimal) nss module should be rarely needed, it is still helpful to disable that on the nixos side.

P.S.

While I use system.nssDatabases.hosts = lib.mkAfter ["[!UNAVAIL=return]"]; to bypass the final mdns entry in nsswitch.conf, I'm still wondering if there is some nix expression to just remove the final mdns entry. Is it possible to achieve that on configuration.nix? (I'm not a long time nix user.)