search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.27k stars 910 forks source link

Creating instances causes excessive logging from dnsmasq #13220

Open simondeziel opened 3 months ago

simondeziel commented 3 months ago

Creating an instance causes dnsmasq to log one line for each of the existing/known instances in every projects. This can snowball as you create more instances over time. Here's how to reproduce it:

journalctl -f -n0 --grep dnsmasq.hosts & for i in $(seq -w 10); do lxc init --quiet ubuntu-minimal-daily:22.04 "c${i}"; done; kill %1

Which causes the following to be logged:

[1] 1953
Mar 27 01:36:21 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c01.eth0
Mar 27 01:36:36 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c01.eth0
Mar 27 01:36:36 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c02.eth0
Mar 27 01:36:51 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c01.eth0
Mar 27 01:36:51 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c02.eth0
Mar 27 01:36:51 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c03.eth0
Mar 27 01:37:06 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c01.eth0
Mar 27 01:37:06 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c02.eth0
Mar 27 01:37:06 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c03.eth0
Mar 27 01:37:06 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c04.eth0
Mar 27 01:37:20 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c01.eth0
Mar 27 01:37:20 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c02.eth0
Mar 27 01:37:20 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c03.eth0
Mar 27 01:37:20 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c04.eth0
Mar 27 01:37:20 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c05.eth0
Mar 27 01:37:35 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c01.eth0
Mar 27 01:37:35 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c02.eth0
Mar 27 01:37:35 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c03.eth0
Mar 27 01:37:35 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c04.eth0
Mar 27 01:37:35 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c05.eth0
Mar 27 01:37:35 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c06.eth0
Mar 27 01:37:53 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c01.eth0
Mar 27 01:37:53 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c02.eth0
Mar 27 01:37:53 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c03.eth0
Mar 27 01:37:53 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c04.eth0
Mar 27 01:37:53 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c05.eth0
Mar 27 01:37:53 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c06.eth0
Mar 27 01:37:53 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c07.eth0
Mar 27 01:38:08 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c01.eth0
Mar 27 01:38:08 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c02.eth0
Mar 27 01:38:08 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c03.eth0
Mar 27 01:38:08 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c04.eth0
Mar 27 01:38:08 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c05.eth0
Mar 27 01:38:08 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c06.eth0
Mar 27 01:38:08 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c07.eth0
Mar 27 01:38:08 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c08.eth0
Mar 27 01:38:23 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c01.eth0
Mar 27 01:38:23 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c02.eth0
Mar 27 01:38:23 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c03.eth0
Mar 27 01:38:23 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c04.eth0
Mar 27 01:38:23 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c05.eth0
Mar 27 01:38:23 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c06.eth0
Mar 27 01:38:23 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c07.eth0
Mar 27 01:38:23 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c08.eth0
Mar 27 01:38:23 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c09.eth0
Mar 27 01:38:40 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c01.eth0
Mar 27 01:38:40 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c02.eth0
Mar 27 01:38:40 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c03.eth0
Mar 27 01:38:40 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c04.eth0
Mar 27 01:38:40 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c05.eth0
Mar 27 01:38:40 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c06.eth0
Mar 27 01:38:40 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c07.eth0
Mar 27 01:38:40 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c08.eth0
Mar 27 01:38:40 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c09.eth0
Mar 27 01:38:40 v1 dnsmasq-dhcp[1484]: read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c10.eth0

journald stores more than just the log message which means more data needs to be saved to disk. While a flawed approximation, the JSON exported logs for that repeated messages alone weights ~100KiB:

$ journalctl -o json --grep dnsmasq.hosts > /tmp/dnsmasq-log.json
$ ll -h /tmp/dnsmasq-log.json 
-rw-r--r-- 1 root root 97K Mar 27 01:40 /tmp/dnsmasq-log.json
hamistao commented 3 months ago

If there is no problem I will assign this to myself

hamistao commented 1 month ago

That happens because the way we update host entries on dnsmasq is by sending a SIGHUP signal to the dnsmasq process. That makes it restart and read all the entries again from dnsmasq.hosts. The only viable alternative to that would be to use the flag --hostsdir instead of --dhcp-hostsfile, that way dnsmasq automatically updates created and modified entries inside the specified directory (but apparentely not removed ones, so it isn't a perfect solution).

hamistao commented 1 month ago

I took some time to test the usage of --hostsdir and it seems like it solves the problem even when deleting instances/removing entries. Even so, it seems too risky to use that flag without further investigation.