cockpit-project / cockpit

Cockpit is a web-based graphical interface for servers.
http://www.cockpit-project.org/
GNU Lesser General Public License v2.1
11.01k stars 1.1k forks source link

cockpit-bridge mutilple processes memory consumptions over 1Gb #13689

Closed nodje closed 1 year ago

nodje commented 4 years ago

Cockpit version: 212 OS: Debian/Fedora

cockpit-bridge processes very frequently add up to a very high RAM consumption of over 1Gb.

In any case, i regularly get 3 cockpit-bridge processes on different monitored machine, and it takes the best of their available RAM. I have to just killall cockpit-bridge regularly to reclaim it.

They must grow over time, and it feels like a memory leak really, as when I killall them and restart a connection from the cockpit-ws host the RAM consumption is very low.

FYI cockpit-pcp is enabled on the monitored machine.

HTH

aeastw commented 4 years ago

Cockpit version: 212 OS: Debian

I also see this issue, as described above, cockpit-bridge memory usage seems to grow over time. Requires kill and restart connection to free.

martinpitt commented 4 years ago

This does sound serious indeed, but I'm afraid we need some more information for this to become actionable and reproducible. Please describe your setup more specifically, i. e. how many machines are involved (local or ssh login over dashboard?), which pages you have open on each of them, how fast you see the memory growing ("over time" == minutes, hours, days?), do you get regular journal messages from cockpit, etc.

nodje commented 4 years ago

ok, my setup is pretty basic and straightforward I believe, so hopefully it'll help identifying what is happening.

I have cockpit-ws setup on one machine, fedora, and it monitors 4 other machines (all debian). They are all part of a realm, provided by a freeipa instance (not monitored).

I connect to the FE provided by the fedora cockpit-ws from 2 machines, and basically use it to check resource consumption, sometimes to apply security updates. So cockpit-pcp is enabled on all 4 debians.

Now the memory leaks seem to happen within ~12hours. Hard to be more precise than this atm.

I did killall cockpit-bridge on a postgres machine yesterday when I created this issue, and it's now using up ~1Gb of RAM again. I did only one unique login to cockpit-ws and accessed this postgres machine to disaply its resource consumption.

Here's the screenshot of an htop showing the culprit. These belong to the ssh user used to connect from cockpit-ws to "monitored"

Screenshot 2020-03-10 at 11 18 15

Note that the cockpit-bridge --privileged is owned by root and doesn't consume as much memory. cockpit-pcp processes also are quite reasonable in ram consumption, never had to restart them.

This happens on different machines at different time. Not systematic.

nodje commented 4 years ago

Also, as an aside, it happens that multiple cockpit-bridge parent processes are created: Screenshot 2020-03-10 at 14 24 11 killall in this case is also the simplest way to clean up

nodje commented 4 years ago

Same thing again today, using even more memory.

Screenshot 2020-03-11 at 09 58 37

So def. happening within a few hours. I'd be hard-pressed to find a better way to monitor this. Any idea welcome.

martinpitt commented 3 years ago

Does this still happen with the current version? Can you please disable PCP or uninstall cockpit-pcp (whichever is less intrusive), and check if it happens without that as well?

martinpitt commented 3 years ago

If this still happens, would it be possible to get ssh access (freshly created user is enough, doesn't need sudo) on your system to debug this?

marusak commented 1 year ago

No reply in ~2 years, closing