search

coreos / fleet

fleet ties together systemd and etcd into a distributed init system
Apache License 2.0
2.42k stars 303 forks source link

functional: close DBUS connections after running systemd commands #1706

Closed dongsupark closed 7 years ago

dongsupark commented 7 years ago

As described in https://github.com/coreos/fleet/pull/1704, functional test has a bug hitting an upper limit of ~40 tests in total. That's actually an issue of DBUS connections remaining opened even after nspawn containers got successfully terminated. That's why the number of unix sockets grows up to 256.

$ netstat -nap | grep /var/run/dbus/system_bus_socket | wc -l
256

From that moment on, functional test hangs mysteriously. Sometimes users could see errors like "The maximum number of active connections for UID 0 has been reached."

Its reason is that every DBUS connection was never closed. The more tests we add, the more stale DBUS connections we have. This bug has existed since the beginning.

Fix it by adding conn.Close() after running systemd commands.

dongsupark commented 7 years ago

As it's clearly a bug, and it doesn't affect main code, I'll simply merge. BTW fleetd doesn't explicitly close DBUS connections either. Though the remaining connections are already cleaned up when fleetd shuts down. That's what I see in my tests. So I suppose, we don't necessarily need to clean them up in fleetd.

jonboulle commented 7 years ago

nice catch!