Navidrome: fails to connect to last.fm with DNS lookup failures

antifuchs commented 2 years ago

Describe the bug

I'm trying to connect navidrome to last.fm, and it fails in a way that makes me suspect the non-cgo DNS lookup method that haunts other golang tools, as well:

time="2021-12-21T11:06:12Z" level=error msg="Could not fetch LastFM session key" error="Get \"https://ws.audioscrobbler.com/2.0/?api_key=<xxx>&api_sig=<xxx>&format=json&method=auth.getSession&token=<xxx>\": dial tcp: lookup ws.audioscrobbler.com on [::1]:53: read udp [::1]:55341->[::1]:53: read: connection refused" requestId=bonnetmaker/daMWUeL5JW-000657 token=<xxx> userId=<xxx>

I am not running a DNS server locally, nor are the various DNS resolution knobs set to [::1]:53. I'm able to resolve ws.audioscrobbler.com on the machine running navidrome on the commandline.

Steps To Reproduce

Steps to reproduce the behavior:

install navidrome
visit <navidrome_install_url>/app/#/personal and select "Scrobble to last.fm"
on last.fm, login with your account & select "connect app".
error message is displayed in the browser, and the above message is logged by navidrome.

Expected behavior

navidrome should be able to resolve ws.audioscrobbler.com.

Additional context

I have two suspicions:

One is that this might be a "pure go" dns resolution problem: https://github.com/golang/go/issues/21906 seems to be a classic of the genre
The other is that there is some systemd isolation knob that's turned on in the navidrome service.

Notify maintainers

@aciceri

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"x86_64-linux"`
 - host os: `Linux 5.10.77, NixOS, 22.05 (Quokka)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.4`
 - channels(root): `"nixos-20.09.2331.7d57790bc12"`
 - nixpkgs: `/nix/store/4g24bds0a05p0vj87jzp3v4yy9nywpf2-source`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: navidrome
# a list of nixos modules affected by the problem
module: services.navidrome

antifuchs commented 2 years ago

Update: This seems to be because the navidrome process can't find /etc/resolv.conf - I imagine something in the filesystem isolation needs to let that file through: If resolv.conf can't be found, go tries some fallback

peperunas commented 2 years ago

I think this is also related to a similar issue I'm encountering, as every request fails with a x509: certificate invalid.

antifuchs commented 2 years ago

Oh yeah, I'd guess if it can't read the SSL certificate store, things would go sideways too. (Edited to note:) I believe the PR referenced above should fix this, too.

kevincox commented 1 year ago

I'm seeing this issue. This is my first time trying the navidrome service but I'm seeing DNS resolution errors for all hosts even though it works on the host.

This is the generated service file:

[Unit]
After=network.target
Description=Navidrome Media Server

[Service]
Environment="IPFS_PATH=/var/lib/ipfs"
Environment="LOCALE_ARCHIVE=/nix/store/k6n7i52ymal0w0s4jgrf6inn2bhmrmiv-glibc-locales-2.35-224/lib/locale/locale-archive"
Environment="PATH=/nix/store/lyicmql3ws929d7azr65h25b2hyakmb6-coreutils-9.1/bin:/nix/store/hk8w4kray4jbc8bbpaxb0dfa95jm7syj-findutils-4.9.0/bin:/nix/store/kp1chfvl8bai7rd2g80lk3xjz4v76j1v-gnugrep-3.7/bin:/nix/store/hdq087rhq35fl5d5p6pw2m7hr2ib0f4h-gnused-4.9/bin:/nix/store/33pkg1f1ad2wh2rkl2c48hf8kfxcjgw5-systemd-253.1/bin:/nix/store/lyicmql3ws929d7azr65h25b2hyakmb6-coreutils-9.1/sbin:/nix/store/hk8w4kray4jbc8bbpaxb0dfa95jm7syj-findutils-4.9.0/sbin:/nix/store/kp1chfvl8bai7rd2g80lk3xjz4v76j1v-gnugrep-3.7/sbin:/nix/store/hdq087rhq35fl5d5p6pw2m7hr2ib0f4h-gnused-4.9/sbin:/nix/store/33pkg1f1ad2wh2rkl2c48hf8kfxcjgw5-systemd-253.1/sbin"
Environment="SSL_CERT_FILE=/nix/store/7yrdb4cb3xqy8xlqmvdn5yalzcjz98r6-nss-cacert-3.86/etc/ssl/certs/ca-bundle.crt"
Environment="TZ=:/etc/zoneinfo/UTC"
Environment="TZDIR=/nix/store/w2swil1dw4fxmvkggcik4l0jiywwp5vp-tzdata-2022g/share/zoneinfo"

BindReadOnlyPaths=/nix/store/7yrdb4cb3xqy8xlqmvdn5yalzcjz98r6-nss-cacert-3.86/etc/ssl/certs/ca-bundle.crt:/etc/ssl/certs/ca-certificates.crt
BindReadOnlyPaths=/nix/store
BindReadOnlyPaths=/etc
BindReadOnlyPaths=/mnt/music
CapabilityBoundingSet=
DynamicUser=true
ExecStart=/nix/store/0llyjdlqzp5l5sa5idd6z5vy9ip98pm7-navidrome-0.49.3/bin/navidrome --configfile /nix/store/nlasb2zfachnq7ifnc4sc53v27iwrzl6-navidrome.json

LockPersonality=true
MemoryDenyWriteExecute=true
PrivateDevices=true
PrivateUsers=true
ProtectClock=true
ProtectControlGroups=true
ProtectHome=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ReadWritePaths=
RestrictAddressFamilies=AF_UNIX
RestrictAddressFamilies=AF_INET
RestrictAddressFamilies=AF_INET6
RestrictNamespaces=true
RestrictRealtime=true
RootDirectory=/run/navidrome
RuntimeDirectory=navidrome
StateDirectory=navidrome
SupplementaryGroups=media-rw
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=~@privileged
UMask=0066
WorkingDirectory=/var/lib/navidrome

kevincox commented 1 year ago

It appears that because when using systemd-resolved /etc/resolv.conf is a symlink.

% realpath /etc/resolv.conf
/run/systemd/resolve/stub-resolv.conf

In fact on most NixOS systems files in /etc are symlinks, so I wonder how this fix helped much at all.

Adding that path to the service config works around this issue.

systemd.services.navidrome.serviceConfig.BindReadOnlyPaths = ["/run/systemd/resolve/stub-resolv.conf"];

NixOS / nixpkgs