search

m13253 / dns-over-https

High performance DNS over HTTPS client & server
https://developers.google.com/speed/public-dns/docs/dns-over-https
MIT License
1.96k stars 221 forks source link

Service doesn't start properly in Arch Linux #150

Closed ndelta0 closed 12 months ago

ndelta0 commented 1 year ago

Every time I boot up my machine the service starts but stops after a few seconds. After manually sudo systemctl start doh-client it, it works fine and has no issues.

OS: Arch

Output of journalctl -b -p 7 -u doh-client:

Aug 15 19:58:02 mango systemd[1]: Started DNS-over-HTTPS Client.
Aug 15 19:58:03 mango systemd[1]: Stopping DNS-over-HTTPS Client...
Aug 15 19:58:03 mango systemd[1]: doh-client.service: Deactivated successfully.
Aug 15 19:58:03 mango systemd[1]: Stopped DNS-over-HTTPS Client.
Aug 15 19:58:03 mango systemd[1]: Started DNS-over-HTTPS Client.
Aug 15 19:58:08 mango doh-client[625]: 2023/08/15 19:58:08 Get "https://[adguard dns uri]": dial tcp: lookup xxxxxxxx.d.adguard-dns.com on [2001:730:3ed2::53]:53: dial udp 94.140.14.49:53: connect: network is unreachable
Aug 15 19:58:08 mango doh-client[625]: 2023/08/15 19:58:08 Get "https://[adguard dns uri]": dial tcp: lookup xxxxxxxx.d.adguard-dns.com on [2001:730:3ed2::53]:53: dial udp 94.140.14.59:53: connect: network is unreachable
Aug 15 19:58:08 mango doh-client[625]: 2023/08/15 19:58:08 Get "https://[adguard dns uri]": dial tcp: lookup xxxxxxxx.d.adguard-dns.com on [2001:730:3ed2::53]:53: dial udp 94.140.14.59:53: connect: network is unreachable
Aug 15 19:58:08 mango systemd[1]: Stopping DNS-over-HTTPS Client...
Aug 15 19:58:08 mango systemd[1]: doh-client.service: Deactivated successfully.
Aug 15 19:58:08 mango systemd[1]: Stopped DNS-over-HTTPS Client.
Aug 15 19:58:08 mango systemd[1]: Started DNS-over-HTTPS Client.
Aug 15 19:58:09 mango systemd[1]: Stopping DNS-over-HTTPS Client...
Aug 15 19:58:09 mango systemd[1]: doh-client.service: Deactivated successfully.
Aug 15 19:58:09 mango systemd[1]: Stopped DNS-over-HTTPS Client.
Aug 15 19:58:09 mango systemd[1]: Started DNS-over-HTTPS Client.
Aug 15 19:58:09 mango systemd[1]: Stopping DNS-over-HTTPS Client...
Aug 15 19:58:09 mango systemd[1]: doh-client.service: Deactivated successfully.
Aug 15 19:58:09 mango systemd[1]: Stopped DNS-over-HTTPS Client.
Aug 15 19:58:09 mango systemd[1]: Started DNS-over-HTTPS Client.
Aug 15 19:58:09 mango systemd[1]: Stopping DNS-over-HTTPS Client...
Aug 15 19:58:09 mango systemd[1]: doh-client.service: Deactivated successfully.
Aug 15 19:58:09 mango systemd[1]: Stopped DNS-over-HTTPS Client.
Aug 15 19:58:09 mango systemd[1]: doh-client.service: Start request repeated too quickly.
Aug 15 19:58:09 mango systemd[1]: doh-client.service: Failed with result 'start-limit-hit'.
Aug 15 19:58:09 mango systemd[1]: Failed to start DNS-over-HTTPS Client.

Note that the dial tcp errors sometimes occur during normal operation, but they don't cause the service to stop.

m13253 commented 1 year ago

This is strange. Can you try running doh-client manually without systemd and see whether it runs?

ndelta0 commented 1 year ago

Will do, in the meantime here's the verbose log from right after booting up:

Aug 22 20:00:26 mango systemd[1]: Started DNS-over-HTTPS Client.
Aug 22 20:00:26 mango doh-client[611]: 2023/08/22 20:00:26 random mode start
Aug 22 20:00:26 mango systemd[1]: Stopping DNS-over-HTTPS Client...
Aug 22 20:00:26 mango systemd[1]: doh-client.service: Deactivated successfully.
Aug 22 20:00:26 mango systemd[1]: Stopped DNS-over-HTTPS Client.
Aug 22 20:00:26 mango systemd[1]: Started DNS-over-HTTPS Client.
Aug 22 20:00:26 mango doh-client[660]: 2023/08/22 20:00:26 random mode start
Aug 22 20:00:31 mango doh-client[660]: [::1]:37533 - - [22/Aug/2023:20:00:31 +0200] "ping.archlinux.org. IN AAAA"
Aug 22 20:00:31 mango doh-client[660]: 2023/08/22 20:00:31 choose upstream: upstream type: IETF, upstream url: https://adguard_id.d.adguard-dns.com/dns-query
Aug 22 20:00:31 mango doh-client[660]: 2023/08/22 20:00:31 Get "https://adguard_id.d.adguard-dns.com/dns-query?ct=application/dns-message&dns=[REDACTED]": dial tcp: lookup adguard_id.d.adguard-dns.com on [2001:730:3ed2::53]:53: dial udp 94.140.14.49:53: connect: network is unreachable
Aug 22 20:00:31 mango doh-client[660]: [::1]:55458 - - [22/Aug/2023:20:00:31 +0200] "ping.archlinux.org. IN AAAA"
Aug 22 20:00:31 mango doh-client[660]: 2023/08/22 20:00:31 choose upstream: upstream type: IETF, upstream url: https://adguard_id.d.adguard-dns.com/dns-query
Aug 22 20:00:31 mango doh-client[660]: 2023/08/22 20:00:31 Get "https://adguard_id.d.adguard-dns.com/dns-query?ct=application/dns-message&dns=[REDACTED]": dial tcp: lookup adguard_id.d.adguard-dns.com on [2001:730:3ed2::53]:53: dial udp 94.140.14.49:53: connect: network is unreachable
Aug 22 20:00:32 mango doh-client[660]: 127.0.0.1:33932 - - [22/Aug/2023:20:00:32 +0200] "ping.archlinux.org. IN AAAA"
Aug 22 20:00:32 mango doh-client[660]: 2023/08/22 20:00:32 choose upstream: upstream type: IETF, upstream url: https://adguard_id.d.adguard-dns.com/dns-query
Aug 22 20:00:32 mango doh-client[660]: 127.0.0.1:46173 - - [22/Aug/2023:20:00:32 +0200] "ping.archlinux.org. IN AAAA"
Aug 22 20:00:32 mango doh-client[660]: 2023/08/22 20:00:32 choose upstream: upstream type: IETF, upstream url: https://adguard_id.d.adguard-dns.com/dns-query
Aug 22 20:00:32 mango doh-client[660]: 127.0.0.1:33932 - - [22/Aug/2023:20:00:32 +0200] "ping.archlinux.org. IN A"
Aug 22 20:00:32 mango doh-client[660]: 127.0.0.1:46173 - - [22/Aug/2023:20:00:32 +0200] "ping.archlinux.org. IN A"
Aug 22 20:00:32 mango doh-client[660]: 2023/08/22 20:00:32 choose upstream: upstream type: IETF, upstream url: https://adguard_id.d.adguard-dns.com/dns-query
Aug 22 20:00:32 mango doh-client[660]: 2023/08/22 20:00:32 choose upstream: upstream type: IETF, upstream url: https://adguard_id.d.adguard-dns.com/dns-query
Aug 22 20:00:32 mango systemd[1]: Stopping DNS-over-HTTPS Client...
Aug 22 20:00:32 mango systemd[1]: doh-client.service: Deactivated successfully.
Aug 22 20:00:32 mango systemd[1]: Stopped DNS-over-HTTPS Client.
Aug 22 20:00:32 mango systemd[1]: Started DNS-over-HTTPS Client.
Aug 22 20:00:32 mango doh-client[774]: 2023/08/22 20:00:32 random mode start
Aug 22 20:00:32 mango systemd[1]: Stopping DNS-over-HTTPS Client...
Aug 22 20:00:32 mango systemd[1]: doh-client.service: Deactivated successfully.
Aug 22 20:00:32 mango systemd[1]: Stopped DNS-over-HTTPS Client.
Aug 22 20:00:32 mango systemd[1]: Started DNS-over-HTTPS Client.
Aug 22 20:00:32 mango doh-client[806]: 2023/08/22 20:00:32 random mode start
Aug 22 20:00:32 mango doh-client[806]: 127.0.0.1:43338 - - [22/Aug/2023:20:00:32 +0200] "archlinux.org. IN AAAA"
Aug 22 20:00:32 mango doh-client[806]: 2023/08/22 20:00:32 choose upstream: upstream type: IETF, upstream url: https://adguard_id.d.adguard-dns.com/dns-query
Aug 22 20:00:32 mango doh-client[806]: 127.0.0.1:43338 - - [22/Aug/2023:20:00:32 +0200] "archlinux.org. IN A"
Aug 22 20:00:32 mango doh-client[806]: 2023/08/22 20:00:32 choose upstream: upstream type: IETF, upstream url: https://adguard_id.d.adguard-dns.com/dns-query
Aug 22 20:00:33 mango systemd[1]: Stopping DNS-over-HTTPS Client...
Aug 22 20:00:33 mango systemd[1]: doh-client.service: Deactivated successfully.
Aug 22 20:00:33 mango systemd[1]: Stopped DNS-over-HTTPS Client.
Aug 22 20:00:33 mango systemd[1]: Started DNS-over-HTTPS Client.
Aug 22 20:00:33 mango doh-client[973]: 2023/08/22 20:00:33 random mode start
Aug 22 20:00:33 mango systemd[1]: Stopping DNS-over-HTTPS Client...
Aug 22 20:00:33 mango systemd[1]: doh-client.service: Deactivated successfully.
Aug 22 20:00:33 mango systemd[1]: Stopped DNS-over-HTTPS Client.
Aug 22 20:00:33 mango systemd[1]: doh-client.service: Start request repeated too quickly.
Aug 22 20:00:33 mango systemd[1]: doh-client.service: Failed with result 'start-limit-hit'.
Aug 22 20:00:33 mango systemd[1]: Failed to start DNS-over-HTTPS Client.
Aug 22 20:43:43 mango systemd[1]: Started DNS-over-HTTPS Client.
ndelta0 commented 1 year ago

Here's running it in command line and browsing example.com:

$ sudo /usr/bin/doh-client -conf /etc/dns-over-https/doh-client.conf -verbose 
2023/08/22 21:34:40 random mode start
127.0.0.1:42150 - - [22/Aug/2023:21:34:45 +0200] "ping.archlinux.org. IN AAAA"
127.0.0.1:42150 - - [22/Aug/2023:21:34:45 +0200] "ping.archlinux.org. IN A"
2023/08/22 21:34:45 choose upstream: upstream type: IETF, upstream url: https://adguard_id.d.adguard-dns.com/dns-query
2023/08/22 21:34:45 choose upstream: upstream type: IETF, upstream url: https://adguard_id.d.adguard-dns.com/dns-query
127.0.0.1:34359 - - [22/Aug/2023:21:34:45 +0200] "ping.archlinux.org. IN A"
2023/08/22 21:34:45 choose upstream: upstream type: IETF, upstream url: https://adguard_id.d.adguard-dns.com/dns-query
127.0.0.1:34359 - - [22/Aug/2023:21:34:45 +0200] "ping.archlinux.org. IN AAAA"
2023/08/22 21:34:45 choose upstream: upstream type: IETF, upstream url: https://adguard_id.d.adguard-dns.com/dns-query
127.0.0.1:35798 - - [22/Aug/2023:21:34:46 +0200] "_dnslink.example.com. IN TXT"
2023/08/22 21:34:46 choose upstream: upstream type: IETF, upstream url: https://adguard_id.d.adguard-dns.com/dns-query
127.0.0.1:52656 - - [22/Aug/2023:21:34:46 +0200] "example.com. IN TXT"
2023/08/22 21:34:46 choose upstream: upstream type: IETF, upstream url: https://adguard_id.d.adguard-dns.com/dns-query
127.0.0.1:53264 - - [22/Aug/2023:21:34:46 +0200] "example.com. IN A"
2023/08/22 21:34:46 choose upstream: upstream type: IETF, upstream url: https://adguard_id.d.adguard-dns.com/dns-query
127.0.0.1:53264 - - [22/Aug/2023:21:34:46 +0200] "example.com. IN AAAA"
2023/08/22 21:34:46 choose upstream: upstream type: IETF, upstream url: https://adguard_id.d.adguard-dns.com/dns-query

As I said, there's nothing happening that shouldn't be.

brainwo commented 1 year ago

I have it starts successfully by changing the Type= from simple to idle and also After= to multi-user.target. Seems like there are something wrong with the order.

You can safely edit the .service file using:

$ sudo systemctl edit doh-client.service

The override content would be:

[Unit]
After=multi-user.target

[Service]
Type=idle
ndelta0 commented 1 year ago

I have it starts successfully by changing the Type= from simple to idle and also After= to multi-user.target.

I can confirm this does in fact work.

m13253 commented 1 year ago

Due to a technical limitation of Golang’s standard library, doh-client and doh-server is unable to detect network change, for example, loss of Wi-Fi signal or switching between wireless and wired network. Therefore, we rely on systemd to determine whether the network is online. The service will not start until the network is ready, and will be automatically restarted whenever there is a network change.

For most Linux distributions, systemd is configured to collaborate with either NetworkManager or whatever network management framework preinstalled. However, there are cases that your system contains more than one such framework, or the configuration is incorrect. In such situations, our service will be unable to start, or automatically restarts itself.

I’m glad your modification solves your issue. However, this fix doesn’t apply to every user. We expect systemd’s network detection to function properly, as it is necessary for those users on Wi-Fi or mobile network.

brainwo commented 1 year ago

Do you think we should suggest this fix to the Arch maintainer? Or should it better to just leave as a note in Arch Wiki?

m13253 commented 1 year ago

Do you think we should suggest this fix to the Arch maintainer?

Or should it better to just leave as a note in Arch Wiki?

I see! No wonder the online detection is broken. ArchLinux doesn’t come with a default network management framework, thus systemd comes without online detection pre-configured. If you are on Wi-Fi, I suggest making sure systemd’s online detection can function properly. I believe your system have already installed some network management framework such as NetworkManager to help you manage Wi-Fi passwords. Or, if you are on wired network, simply modifying the .service file to disable online detection would be the easiest solution. Installing NetworkManager for a non-mobile machine might be against the K.I.S.S. principle, and we don’t want it.

No matter which method you take, I believe I can’t make the choice for you. So I suggest we leave a note on ArchWiki.

For other distributions, their distributors usually have made the decision for them, so they all got NetworkManager or equivalent frameworks installed, even for non-mobile machines. So I believe this issue is mainly related to DIY distributions such as ArchLinux, Gentoo, NixOS, etc. putting it on ArchWiki is great.

brainwo commented 1 year ago

Thanks for your suggestion and advice, I appreciate it.

I don't have an Arch Wiki account, if anybody interested on this: you can help me adding the note to Arch Wiki.

Also it might be better if we change the title to "Service doesn't start properly in Arch Linux"? To make it easier for search engines.

ndelta0 commented 1 year ago

I don't have an Arch Wiki account, if anybody interested on this: you can help me adding the note to Arch Wiki.

I don't have an account there, so anyone who sees this and can do that feel free to do so.

Also it might be better if we change the title to "Service doesn't start properly in Arch Linux"? To make it easier for search engines.

Done

Nambers commented 11 months ago

Hi there, I found this problem on my Arch today and glad you guys already figure out the solution of it! Also, I left a note on ArchWiki https://wiki.archlinux.org/title/DNS-over-HTTPS

m13253 commented 11 months ago

Hi there, I found this problem on my Arch today and glad you guys already figure out the solution of it! Also, I left a note on ArchWiki https://wiki.archlinux.org/title/DNS-over-HTTPS

Thank you so much! ❤️