LXC container saturation

[ketomagasaki]

Hello,

OS: Ubuntu 22.04.3 Galera Manager: 1.8.1

I have a new issue; I noticed this morning that the Galera Manager GUI was no longer responding. Upon checking the container, the RAM and SWAP were full. Therefore, I stopped the container and increased the RAM to 2048. However, after starting, it gradually fills up the RAM and crashes after 10 minutes.

I also notice that the disk is getting filled.

"Image before modifying the RAM"

"Image after modifying the RAM"

[ketomagasaki]

I assume it's the logs that are filling up the storage space and eventually causing the container to crash.

What command or technique can I use to purge the logs? I searched for a method in the documentation but couldn't find one. With this information, I could establish a log purging routine.

[fa11er]

I also have a problem with the disk being full. How can I get this under control? How long is the log kept? How can I influence this?

[ayurchen]

Hi, unfortunately at the moment these things can't be controlled. Galera Manager collects one month of data (metrics and logs) so that it can be displayed and analyzed even when a node is lost, so it depends on how chatty your nodes are. Please check if you have general or slow logs enabled on the nodes. Having said that, we have not encountered memory usage anywhere near that. My influxd RES at the moment sits at 150Mb and its been running for a few days. We need a way to reproduce the issue. So any usage "tips" are welcome.

[ketomagasaki]

Hi,

I shut down the Galera Manager container a few days ago to prevent it from filling its disk. Currently, with 3 nodes, Influx DB is taking up 6.4 GB out of the container's 7.64 GB:

Since I added storage space to 10 Go, there have been no RAM problems.

I haven't found how to "slow down" log provisioning. In /etc/default/influxdb2:

INFLUXD_CONFIG_PATH=/etc/influxdb/config.toml
INFLUXD_HTTP_BIND_ADDRESS=127.0.0.1:8086
INFLUXD_FLUX_LOG_ENABLED=false
INFLUXD_LOG_LEVEL=error
INFLUXD_METRICS_DISABLED=true
INFLUXD_REPORTING_DISABLED=true
INFLUXD_UI_DISABLED=true

In the meantime, while looking for a solution and to avoid losing this container, I keep it turned off and only switch it on when I need to make modifications to the cluster.

[ayurchen]

I haven't found how to "slow down" log provisioning. In /etc/default/influxdb2:

INFLUXD_CONFIG_PATH=/etc/influxdb/config.toml
INFLUXD_HTTP_BIND_ADDRESS=127.0.0.1:8086
INFLUXD_FLUX_LOG_ENABLED=false
INFLUXD_LOG_LEVEL=error
INFLUXD_METRICS_DISABLED=true
INFLUXD_REPORTING_DISABLED=true
INFLUXD_UI_DISABLED=true

It is not about InfuxDB, it is about how much your mysql servers log. E.g. if you have

'mysql_native_password' is deprecated and will be removed in a future release.

reported for every client connection, you may have a lot of logs.

[ketomagasaki]

Okay,

However, do you have any idea what I should do? Because I can't understand what type of log is causing the saturation. Is there a way to find out? Or even a configuration file?