search

bus1 / dbus-broker

Linux D-Bus Message Broker
https://github.com/bus1/dbus-broker/wiki
Apache License 2.0
677 stars 79 forks source link

dbus-broker uses an extremely high amount of memory #346

Closed BrainDamage closed 6 months ago

BrainDamage commented 7 months ago 
OS: arch linux

ps aux | grep dbus-broker
dbus        1096  0.1  0.0  14240  5504 ?        Ss   mar02   7:07 /usr/bin/dbus-broker-launch --scope system --audit
dbus        1097  0.6 11.3 1843456 1839232 ?     S    mar02  25:13 dbus-broker --log 4 --controller 9 --machine-id *** --max-bytes 536870912 --max-fds 4096 --max-matches 131072 --audit
cloud       9588  0.0  0.0   8860  2280 ?        S    mar02   0:00 /usr/bin/dbus-broker-launch --config-file=/usr/share/defaults/at-spi2/accessibility.conf --scope user
cloud       9589  0.0  0.0   4352  2220 ?        S    mar02   0:02 dbus-broker --log 4 --controller 9 --machine-id *** --max-bytes 100000000000000 --max-fds 6400000 --max-matches 5000000000

dbus-broker --v                                                                                                                                                                                        8s
dbus-broker 35

the system instance of dbus in my system is allocating 1.8GB worth of memory, which sounds a bit excessive to me; especially since the uptime of my system is less than 3 days.

I've tried a quick search for pre-existing issues and I found this old ticket, but it seems inconclusive and has been closed.

I've dumped the output of: sudo dbus-send --system --dest=org.freedesktop.DBus --type=method_call --print-reply /org/freedesktop/DBus org.freedesktop.DBus.Debug.Stats.GetStats > dbus-broker-stats.txt as was suggested in the previous ticket, and it's attached here; however I am unsure how to parse it.

dvdhrm commented 7 months ago

Your statistics-dump shows a connection with ID 1.82 (PID: 8838, UID: 1000) to have a huge amount of messages queued, which it does not process:

dict entry(
   string "OutgoingBytes"
   uint32 535562975
)

This is 500MiB of messages queued for the mentioned connection. It is also very close to exceeding the quota for UID 1000 (190605 of 536870912 bytes remaining before the quota will be exceeded).

This means there is a program on your machine running, which asked to receive D-Bus messages, but does not dispatch those messages. Hence, they will pile up until a quota is exceeded.

I assume the PID/connection-ID changes on each reboot. So if you see this happening again, use the same method to acquire the statistics-dump and search for OutgoingBytes until you find an entry with a high value. Shortly above it, the dump will show the connection-id, PID, UID, etc. (you can use busctl to show all connection IDs currently in use).

I also wrote a short explanation on how to find such D-Bus processes: https://dvdhrm.github.io/2021/04/14/locating-dbus-resource-leaks/

Let me know if you need more help!

BrainDamage commented 6 months ago

Thanks for the help, I've managed to find the root cause.

I had a systemd user service with auto-restart enabled, the service kept crashing and the stacktraces being dumped to coredumpctl every few s. Eventually the queue kept increasing to the point of allocating the mentioned ram amount. I aso learned a lesson next time to skim the global journal first instead of filtering on a per-unit output.