search

passbolt / charts-passbolt

Helm charts to run Passbolt on Kubernetes. No strings attached charts to run the open source password manager for teams!
https://passbolt.com
GNU Affero General Public License v3.0
40 stars 27 forks source link

Troubleshooting: Full disk due to stale MySQL bin log files and healthcheck logging #72

Closed Munsio closed 3 months ago

Munsio commented 6 months ago

Hi,

I don't know if this is the correct place for adding a troubleshooting section but as I am using the Helm chart maybe others will also stumble upon this problem.

What happened: My passbolt instance refused working due to a full mysql disk. Upon further inspection I found that the current configuration of the bitnami MariaDB image is missing a value for rotating the binary log files. According to https://mariadb.com/kb/en/setting-up-replication/ the binary logging is needed when using replication but it seems that the default value for binlog_expire_logs_seconds is zero. So it happened that half of my disk size was full with old bin log files wasting space.

Solution: Modifying the configmap, setting binlog_expire_logs_seconds and restarting the pod did solve the problem.

Second problem: With the current configuration every 10s kubernetes runs the health check and all those health check calls are getting logged inside the action_log table (which is now 4GB in size thanks to this). According to https://community.passbolt.com/t/as-an-admin-i-want-to-be-able-to-purge-the-logs-periodically/6013 there is no automatic cleanup process currently available in passbolt to solve this problem. On the other hand it would be greeat to have an option to ignore those healthchecks to appear inside the action_logs anyway as they are, imho, not necessary for audit reasons like the rest may be.

Edit: Clarified a few links + using the correct documentation its mariadb not mysql.

Munsio commented 6 months ago

I also raised the first part of this one in the bitnami charts repo: https://github.com/bitnami/charts/issues/21886

dlen commented 6 months ago

Hey @Munsio thanks for the explanation. About the action logs, if you are running the latest version a GET to the status should be ignored. If that is not the case it could be a bug either on this chart (making the wrong request) or in the API. Here is the blacklist that is included by default: https://github.com/passbolt/passbolt_api/blob/master/plugins/PassboltCe/Log/config/config.php

dlen commented 3 months ago

Bitnami issue has been addressed and action logs logging should be fixed as per my last comment so I'm closing this one.