Closed pmcarrion closed 1 year ago
@pmcarrion Thanks for reporting!
It seems that ctrld tries to get a new ctrld.toml file every time the computer restarts or wakes up.
What's wakes up
do you mean? On restarting the computer, it's expected behavior.
It also seems that ctrld is unable to resolve its own URL and get the DoH service running.
Could you please give us the log when this happens?
Hi @cuonglm!
What's wakes up do you mean? On restarting the computer, it's expected behavior.
The computer goes to sleep after some time without activity or when the lid is closed. When activity resumes or the lid is opened, the computer wakes up.
Could you please give us the log when this happens?
Sure, what command should I use to get a log while using the service?
@pmcarrion The log is located under /var/log
directory.
In the system.log
file?
In the
system.log
file?
Nope, it's ctrld.log
.
There's no file by that name.
There's no file by that name.
Please paste the result of ls -l /var/log
, thank you.
ls -l /var/log total 72272 drwxr-xr-x 2 root wheel 64 Feb 9 04:39 CoreDuet drwxr-x--- 10 root admin 320 Mar 21 18:41 DiagnosticMessages -rw-r--r-- 1 root wheel 5299 Feb 16 07:07 PanGPUninstall.log -rwxr--r-- 1 root wheel 64800 Mar 10 14:11 acroUpdaterTools.log -rw-r--r-- 1 root wheel 0 Oct 24 22:16 alf.log -rw-rw-rw-@ 1 root wheel 1047293 Mar 21 21:14 anydesk.trace drwxr-xr-x 2 root wheel 64 Feb 9 04:39 apache2 -rw-r--r-- 1 root wheel 0 Oct 24 22:31 appfirewall.log drwxr-xr-x 18 root wheel 576 Mar 21 00:41 asl drwxr-xr-x 2 root wheel 64 Oct 24 22:32 com.apple.wifivelocity drwxr-xr-x 5 root wheel 160 Mar 21 21:21 com.apple.xpc.launchd drwxr-xr-x 3 root wheel 96 Feb 13 15:07 cups -rw-r--r-- 1 root wheel 880809 Mar 21 18:35 daily.out drwxr-xr-x 2 root wheel 64 Feb 9 04:39 dm -rw-r--r-- 1 root wheel 639733 Mar 21 17:14 fsck_apfs.log -rw-r--r-- 1 root wheel 51786 Mar 21 17:14 fsck_apfs_error.log -rw-r--r-- 1 root wheel 893268 Mar 21 20:25 fsck_hfs.log -rw-r--r-- 1 root wheel 0 Dec 26 23:00 hotfolder.log -rw-r--r--@ 1 root admin 29497601 Mar 21 21:26 install.log -rw-r--r-- 1 root admin 1840648 Jan 10 17:34 install.log.0.gz -rw-r--r-- 1 root wheel 0 Oct 31 18:01 insync.out.log drwxr-xr-x 2 _mdnsresponder wheel 64 Feb 9 04:39 mDNSResponder -rw-r--r-- 1 root wheel 1787 Mar 16 23:32 monthly.out -rw-r--r-- 1 root wheel 0 Oct 24 22:16 nix-daemon.log -rw-r--r-- 1 root wheel 270336 Dec 18 19:09 pf.db -rw-r--r-- 1 root wheel 32768 Dec 26 17:48 pf.db-shm -rw-r--r-- 1 root wheel 0 Dec 18 19:09 pf.db-wal drwxr-xr-x@ 18 root admin 576 Mar 21 00:41 powermanagement drwxr-xr-x 2 root wheel 64 Feb 9 04:39 ppp drwxr-xr-x 3 root wheel 96 Feb 15 21:08 rooksd -rw-r--r-- 1 root wheel 35947 Mar 21 15:33 shutdown_monitor.log -rw-r--r-- 1 root wheel 0 Oct 24 22:16 stubby.log -rw-r--r-- 1 root wheel 39 Dec 27 22:03 synology-active-backup-agent.log -rw-r-----@ 1 root admin 486007 Mar 21 21:40 system.log -rw-r----- 1 root admin 9315 Mar 21 00:00 system.log.0.gz -rw-r----- 1 root admin 6454 Mar 20 00:00 system.log.1.gz -rw-r----- 1 root admin 2266 Mar 19 00:10 system.log.2.gz -rw-r----- 1 root admin 13535 Mar 18 00:11 system.log.3.gz -rw-r----- 1 root admin 6449 Mar 17 00:00 system.log.4.gz -rw-r----- 1 root admin 6260 Mar 16 00:00 system.log.5.gz -rw-r----- 1 root admin 5183 Mar 15 00:01 system.log.6.gz drwxr-xr-x 2 _uucp wheel 64 Feb 9 04:39 uucp -rw-r--r-- 1 root wheel 690 Mar 20 18:49 weekly.out -rw-r----- 1 root admin 15566 Mar 21 21:13 wifi.log -rw-r----- 1 root admin 1609 Mar 21 00:41 wifi.log.0.bz2 -rw-r----- 1 root admin 2426 Mar 20 00:35 wifi.log.1.bz2 -rw-r----- 1 root admin 3751 Mar 11 00:36 wifi.log.10.bz2 -rw-r----- 1 root admin 2301 Mar 19 00:34 wifi.log.2.bz2 -rw-r----- 1 root admin 2136 Mar 18 00:44 wifi.log.3.bz2 -rw-r----- 1 root admin 3454 Mar 17 00:30 wifi.log.4.bz2 -rw-r----- 1 root admin 5357 Mar 16 00:30 wifi.log.5.bz2 -rw-r----- 1 root admin 8555 Mar 15 00:31 wifi.log.6.bz2 -rw-r----- 1 root admin 7458 Mar 14 00:38 wifi.log.7.bz2 -rw-r----- 1 root admin 9077 Mar 13 00:30 wifi.log.8.bz2 -rw-r----- 1 root admin 7305 Mar 12 00:30 wifi.log.9.bz2
Did you set log file in your config /Applications/CLI/ctrld/ctrld.toml
?
Nope. I haven't modified the file because it gets overwritten every time the computer restarts or wakes up.
ctrld.plist
Could you please check content of this file ctrld.plist
, to see where log files are located.
<key>StandardOutPath</key>
<string>/usr/local/var/log/ctrld.out.log</string>
<key>StandardErrorPath</key>
<string>/usr/local/var/log/ctrld.err.log</string>
The log folder is inaccessible.
I can't even open it with sudo cd /usr/local/var/log/
@pmcarrion Thanks, it's probably a bug in the service library that we used.
In the meantime, would you mind start ctrld with --log
flag, so we can see the log when ctrld running.
@cuonglm To which email do I send the log?
@pmcarrion hello@controld.com
@yegors @cuonglm I just sent the files.
@pmcarrion Thanks, we received the files and will investigate the problem.
@pmcarrion How did you "restart" ctrld?
@cuonglm
After restarting the computer, ctrld launched as expected, but didn't resolve any domains.
I used sudo ./ctrld service restart
, but DNS resolving didn't work either.
So, I used sudo ./ctrld service uninstall
and then used sudo ./ctrld start --cd $DID --config /Applications/CLI/ctrld/ctrld.toml --log /Applications/CLI/ctrld/ctrld.log
DNS resolving started working after that.
This issue happens every time the computer is restarted and I have to reinstall the service each time.
@pmcarrion It seems to me Ventura 13.2.1 has issue with Wi-Fi. I can see the problem happen with two cases:
In both case, that seems to be problem that os resolver hangs.
I could not reproduce problem when putting the computer to sleep, then waking up. ctrld
still works normally.
I also have this issue when using Ethernet only (Wi-Fi turned off). My house has a 4-AP Wi-Fi network with the same SSID. In-house Wi-Fi roaming doesn't seem affected. The ctrld DNS issue happens mostly after restarting the computer on every network interface.
I haven't noticed any sleep-related issues today.
@pmcarrion If you edit ctrld.plist
file, to remove /usr/local
prefix like this:
<key>StandardOutPath</key>
<string>/var/log/ctrld.out.log</string>
<key>StandardErrorPath</key>
<string>/var/log/ctrld.err.log</string>
Then you can see in the log:
cuonglm@Cuongs-MacBook-Pro ctrld % sudo tail -f /var/log/ctrld.out.log
loading config file from: /Users/cuonglm/ctrld.toml
starting ctrld...
Mar 23 10:34:47.371 INF fetching Controld D configuration from API: p2 mode=cd
Mar 23 10:34:47.743 INF generating ctrld config from Controld-D configuration mode=cd
Mar 23 10:34:47.745 INF writing config file to: /Users/cuonglm/ctrld.toml mode=cd
Mar 23 10:34:47.747 DBG Resolving "freedns.controld.com" using bootstrap DNS ["76.76.2.0:53" "127.0.0.1:53"]
Mar 23 10:34:47.883 DBG Bootstrap IPs: [2606:1a40::11 76.76.2.11]
Mar 23 10:34:47.883 INF Setting bootstrap IP for upstream.0 bootstrap_ip=76.76.2.11
Mar 23 10:34:47.883 DBG sending doh request to: 76.76.2.11:443
Mar 23 10:34:48.101 INF Starting DNS server on listener.0: 127.0.0.1:53
Mar 23 10:34:48.783 DBG [4eb166] 127.0.0.1:58459 -> listener.0: 127.0.0.1:53: received query: A gspe1-ssl.ls.apple.com
Mar 23 10:34:48.783 DBG [fb2aab] 127.0.0.1:49882 -> listener.0: 127.0.0.1:53: received query: HTTPS gs-loc.apple.com
Mar 23 10:34:48.784 INF [4eb166] no policy, no network, no rule -> [upstream.0]
Mar 23 10:34:48.784 INF [fb2aab] no policy, no network, no rule -> [upstream.0]
Mar 23 10:34:48.784 DBG [4eb166] sending query to upstream.0:
Mar 23 10:34:48.783 DBG [a14194] 127.0.0.1:59770 -> listener.0: 127.0.0.1:53: received query: HTTPS gspe1-ssl.ls.apple.com
Mar 23 10:34:48.785 INF [a14194] no policy, no network, no rule -> [upstream.0]
Mar 23 10:34:48.785 DBG [a14194] sending query to upstream.0:
Mar 23 10:34:48.783 DBG [2dce42] 127.0.0.1:60964 -> listener.0: 127.0.0.1:53: received query: AAAA gs-loc.apple.com
Mar 23 10:34:48.786 INF [2dce42] no policy, no network, no rule -> [upstream.0]
ctrld
has a mechanism to continue starting up only if network is up. The net checker use a backoff with max 1 minute. I guess that where you think ctrld
hangs.
Would you mind making changes to ctrld.plist
like I said above, then restarting to see what's in the log?
I modified the log path and was able to see its contents now. I just sent you the service logs by email.
I modified the log path and was able to see its contents now. I just sent you the service logs by email.
Thanks 🙏
But are you able to see ctrld
start working after hanging for a while?
Nope, I had to manually restart the service using sudo ./ctrld service restart
I know it's confusing!
You can see this in the ctrld.out.log
file:
After restarting the computer - DNS resolving doesn't work:
starting ctrld...
[90mMar 22 22:51:19.451[0m [32mINF[0m fetching Controld D configuration from API: $DID [36mmode=[0mcd
[90mMar 22 22:51:29.549[0m [31mWRN[0m could not fetch resolver config [36merror=[0m[31m"client.Do: Post \"https://api.controld.com/utility?platform=ctrld\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"[0m [36mmode=[0mcd
I waited for 6 minutes, then restarted the service.
After restarting the service - DNS resolving works again:
[90mMar 22 22:57:42.546[0m [32mINF[0m Service stopped
loading config file from: /Applications/CLI/ctrld/ctrld.toml
starting ctrld...
[90mMar 22 22:57:46.818[0m [32mINF[0m fetching Controld D configuration from API: $DID [36mmode=[0mcd
[90mMar 22 22:57:56.824[0m [31mWRN[0m could not fetch resolver config [36merror=[0m[31m"client.Do: Post \"https://api.controld.com/utility?platform=ctrld\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"[0m [36mmode=[0mcd
[90mMar 22 22:57:57.186[0m [32mINF[0m Setting bootstrap IP for upstream.0 [36mbootstrap_ip=[0m76.76.2.22
[90mMar 22 22:57:58.892[0m [32mINF[0m Starting DNS server on listener.0: 127.0.0.1:53
[90mMar 22 22:57:58.910[0m [32mINF[0m [1a8473] no policy, no network, no rule -> [upstream.0]
@pmcarrion While ctrld
stuck, are you able to ping 8.8.8.8
?
While ctrld stuck, are you able to ping 8.8.8.8
I'll have to restart and try that. BRB.
I restarted the computer. No DNS resolving.
I did a ping test to 8.8.8.8:
➜ ~ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=115 time=41.474 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=115 time=37.979 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=115 time=37.208 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=115 time=37.077 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=115 time=37.502 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=115 time=37.360 ms
64 bytes from 8.8.8.8: icmp_seq=6 ttl=115 time=37.196 ms
64 bytes from 8.8.8.8: icmp_seq=7 ttl=115 time=37.286 ms
64 bytes from 8.8.8.8: icmp_seq=8 ttl=115 time=37.355 ms
64 bytes from 8.8.8.8: icmp_seq=9 ttl=115 time=37.449 ms
^C
--- 8.8.8.8 ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 37.077/37.789/41.474/1.250 ms
Restarted the service using sudo ./ctrld service restart
DNS started resolving again.
I restarted the computer. No DNS resolving.
I did a ping test to 8.8.8.8:
➜ ~ ping 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: icmp_seq=0 ttl=115 time=41.474 ms 64 bytes from 8.8.8.8: icmp_seq=1 ttl=115 time=37.979 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=115 time=37.208 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=115 time=37.077 ms 64 bytes from 8.8.8.8: icmp_seq=4 ttl=115 time=37.502 ms 64 bytes from 8.8.8.8: icmp_seq=5 ttl=115 time=37.360 ms 64 bytes from 8.8.8.8: icmp_seq=6 ttl=115 time=37.196 ms 64 bytes from 8.8.8.8: icmp_seq=7 ttl=115 time=37.286 ms 64 bytes from 8.8.8.8: icmp_seq=8 ttl=115 time=37.355 ms 64 bytes from 8.8.8.8: icmp_seq=9 ttl=115 time=37.449 ms ^C --- 8.8.8.8 ping statistics --- 10 packets transmitted, 10 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 37.077/37.789/41.474/1.250 ms
Restarted the service using
sudo ./ctrld service restart
DNS started resolving again.
Thanks!
Looking at the log, is ctrld stuck at "starting ctrld ..." line?
Yes, it looks like it's stuck there after restarting the computer.
[90mMar 22 22:49:47.007[0m [32mINF[0m Service stopped
loading config file from: /Applications/CLI/ctrld/ctrld.toml
starting ctrld...
[90mMar 22 22:51:19.451[0m [32mINF[0m fetching Controld D configuration from API: $DID [36mmode=[0mcd
[90mMar 22 22:51:29.549[0m [31mWRN[0m could not fetch resolver config [36merror=[0m[31m"client.Do: Post \"https://api.controld.com/utility?platform=ctrld\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"[0m [36mmode=[0mcd
[90mMar 22 22:57:42.546[0m [32mINF[0m Service stopped
loading config file from: /Applications/CLI/ctrld/ctrld.toml
starting ctrld...
[90mMar 22 22:57:46.818[0m [32mINF[0m fetching Controld D configuration from API: $DID [36mmode=[0mcd
[90mMar 22 22:57:56.824[0m [31mWRN[0m could not fetch resolver config [36merror=[0m[31m"client.Do: Post \"https://api.controld.com/utility?platform=ctrld\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"[0m [36mmode=[0mcd
[90mMar 22 22:57:57.186[0m [32mINF[0m Setting bootstrap IP for upstream.0 [36mbootstrap_ip=[0m76.76.2.22
[90mMar 22 22:57:58.892[0m [32mINF[0m Starting DNS server on listener.0: 127.0.0.1:53
[90mMar 22 22:57:58.910[0m [32mINF[0m [1a8473] no policy, no network, no rule -> [upstream.0]
I just noticed something else... the logs show a different timezone (1 hour earlier).
@pmcarrion forgot to ask, do you have ipv6?
Yes, I have IPv6.
Same or similar problem here but with Windows 11, ctrld service won't start if there is no internet connection on startup, I have to start the service manually upon connecting to the internet.
Same or similar problem here but with Windows 11, ctrld service won't start if there is no internet connection on startup, I have to start the service manually upon connecting to the internet.
Hi @domy86, thanks for reporting.
What's exactly the state of service, it's starting but hangs, or in failed state?
Hi @pmcarrion, would you mind changing the following content in ctrld.plist
, then restart computer to see if it works:
<key>KeepAlive</key>
<false/>
<key>RunAtLoad</key>
<false/>
<key>Disabled</key>
<false/>
<key>WatchPaths</key>
<array>
<string>/private/var/run/resolv.conf</string>
</array>
Hi @cuonglm,
I changed the ctrld.plist
file, but it made things worse.
I rebooted the computer and there was no DNS resolving as expected.
Furthermore, I restarted the service using sudo ./ctrld service restart
, and it just wouldn't start at all.
I also tried using sudo ./ctrld service start
but it complained that there was no ctrld.toml
file in the home folder.
So, I had to use the command sudo ./ctrld start --config /Applications/CLI/ctrld/ctrld.toml
(without --cd
as suggested by @yegors) to get DNS again, but it overwrote the ctrld.plist
file and there's no way to access the logs from /usr/local
.
@pmcarrion Please try following step:
sudo ./ctrld start --cd <uid> -vv
tail -f /var/log/ctrld.log
@cuonglm
Changed the ctrld.plist
file and restarted the computer. Still no DNS resolving.
/var/log/ctrld.log
doesn't exist, so I changed it to /var/log/ctrld.out.log
Last login: Fri Mar 24 08:45:04 on console
➜ ~ tail -f /var/log/ctrld.log
tail: /var/log/ctrld.log: No such file or directory
➜ ~ tail -f /var/log/ctrld.out.log
Mar 24 06:43:26.076 INF [2e1b0e] no policy, no network, no rule -> [upstream.0]
Mar 24 06:43:26.080 INF [04e6a4] no policy, no network, no rule -> [upstream.0]
Mar 24 06:43:26.451 INF [ff26b1] no policy, no network, no rule -> [upstream.0]
Mar 24 06:43:32.193 INF [ba9833] no policy, no network, no rule -> [upstream.0]
Mar 24 06:43:32.503 INF [e3b463] no policy, no network, no rule -> [upstream.0]
Mar 24 06:43:32.789 INF [681981] no policy, no network, no rule -> [upstream.0]
Mar 24 06:43:32.790 INF Service stopped
loading config file from: /Applications/CLI/ctrld/ctrld.toml
starting ctrld...
Mar 24 06:54:21.470 INF Service stopped
Last login: Fri Mar 24 08:46:25 on ttys000
➜ ~ cd /Applications/CLI/ctrld
➜ ctrld sudo ./ctrld service restart
Password:
Service restarted
➜ ctrld sudo ./ctrld service status
Service is stopped
➜ ctrld sudo ./ctrld service start
loading config file from: /Users/pmcarrion/ctrld.toml
Mar 24 08:57:04.405 INF Service started
➜ ctrld
The ctrld.plist file reverted to its original form after using sudo ./ctrld service start
and lost the changes made before restarting.
@pmcarrion Do you have other service that messing the network configuration?
If you can get a video to demonstrate this, it will be easier for us to troubleshooting. FYI, here's how I got so far.
When service is loaded by launchd, its networking is delay somehow. Even when network status is connected, checking internet connection wont succeed. I tried various approach:
All cases work properly if I tried in terminal when wifi connected (network up).
The change to ctrld.plist makes launchd only start ctrld
when there's a change to /private/var/run/resolv.conf
, which means a network is up and configured. I tested in my M1, with same version Ventura 13.2.1, and ctrld
can start properly, though there's still a small delay, which I see also happens without ctrld.
Could you please retry another time (with video is great):
/private/var/run/resolv.conf
and see whether its content was changed.Same or similar problem here but with Windows 11, ctrld service won't start if there is no internet connection on startup, I have to start the service manually upon connecting to the internet.
Hi @domy86, thanks for reporting.
What's exactly the state of service, it's starting but hangs, or in failed state?
I see that it tries to connect to 76.76.2.22 and then timeouts...
A timeout was reached (45000 milliseconds) while waiting for the ctrld service to connect.
EDIT: I have configured the service to restart on failure, not helped.
@domy86 so something blocks you from connecting to the upstream:
$ dig +short dns.controld.com
76.76.2.22
So it explains why ctrld does not work.
Are you sure you don't have Windscribe or another VPN with an always on firewall?
@domy86 so something blocks you from connecting to the upstream:
$ dig +short dns.controld.com 76.76.2.22
So it explains why ctrld does not work.
No, there was no connection, I don't expect that ctrld works during that period. :D
For example, I go for a coffee in some bar, turn on my notebook to check or transfer some files to contractors' or clients' USB drive, come back home, my notebook connects to the network but internet is not working because ctrld service is not started.
I hope you understand now. :)
EDIT: I want to say that ctrld service should start regardless of network connection state.
Same or similar problem here but with Windows 11, ctrld service won't start if there is no internet connection on startup
I mean this is expected behavior. ctrld
could not work if there's no internet connection. It's probably Windows service manager give up starting ctrld
after too many failure :thinking:
@cuonglm
Aside from my application firewall, no. I don't think Little Snitch is the issue here.
I'll make a video and send it by email. Please note that I use Ethernet, not Wi-Fi.
I just sent the video links by email.
Issue has been reproduced. Thanks for the help. We'll get that fixed.
@domy86 It's very similar to my issue.
ctrld is running on startup, but doesn't resolve any domains.
In your case, it's enough to restart the service.
In my case, restarting the service just stops ctrld, it doesn't restart it. I have to reinstall it to get DNS resolving.
OS: macOS Ventura 13.2.1 (22D68) Computer: MacBook Pro with M1 Max chip (Apple Silicon) ctrld version: 1.1.3 darwin_arm64
DNS resolving fails when restarting or awaking the computer.
The command
sudo ./ctrld start --cd $DID --config /Applications/CLI/ctrld/ctrld.toml
creates a file calledctrld.plist
in/Library/LaunchDaemons
. This file sets a daemon with the following command:/Applications/CLI/ctrld/ctrld run --cd $DID --config /Applications/CLI/ctrld/ctrld.toml --iface=auto --homedir=/Users/$USER
It seems that ctrld tries to get a new
ctrld.toml
file every time the computer restarts or wakes up. This is completely unnecessary as thectrld.toml
file never changes and only needs to be fetched once when setting up the service.It also seems that ctrld is unable to resolve its own URL and get the DoH service running.
This issue causes my computer to freeze until I remove the DNS server (127.0.0.1) or uninstall the ctrld service.