search

mailcow / mailcow-dockerized

mailcow: dockerized - 🐮 + 🐋 = 💕
https://mailcow.email
GNU General Public License v3.0
8.33k stars 1.13k forks source link

create_cold_standby Could not create MariaDB backup on source #5799

Open jbunner opened 4 months ago

jbunner commented 4 months ago

Contribution guidelines

I've found a bug and checked that ...

Description

When running create_cold_standby script, fails with the error when creating MariaDB backup:
[ERR] - Could not create MariaDB backup on source

The MariaDB container itself reports:
mariabackup: Can't create file '/backup/ib_logfile0' (errno: 13 "Permission denied")

It appears to be a permissions issue within the MariaDB Docker imaging using user/group 999:999 and not having permissions to the /opt/mailcowdockerized/_tmp_mariabackup

Was able to modify the helper-scripts/_cold-standby.sh and add the last two lines of code:
    # Make sure a previous backup does not exist
    rm -rf "${SCRIPT_DIR}/../_tmp_mariabackup/"
    mkdir -m 777 "${SCRIPT_DIR}/../_tmp_mariabackup/"
    chown 999:999 "${SCRIPT_DIR}/../_tmp_mariabackup/"

Logs:

DEBUG: Using native docker compose on remote

Found compose project name mailcowdockerized for ######
Found SQL mariadb:10.5

Preparing remote...
Synchronizing mailcow base directory...
Running redis-cli save... OK
Creating remote mountpoint /var/lib/docker/volumes/mailcowdockerized_clamd-db-vol-1/_data for mailcowdockerized_clamd-db-vol-1...
Synchronizing mailcowdockerized_clamd-db-vol-1 from local /var/lib/docker/volumes/mailcowdockerized_clamd-db-vol-1/_data...
        737,280   0%    3.58MB/s    0:00:00 (xfr#1, to-chk=0/6487)   
Completed
Creating remote mountpoint /var/lib/docker/volumes/mailcowdockerized_crypt-vol-1/_data for mailcowdockerized_crypt-vol-1...
Synchronizing mailcowdockerized_crypt-vol-1 from local /var/lib/docker/volumes/mailcowdockerized_crypt-vol-1/_data...
              0   0%    0.00kB/s    0:00:00 (xfr#0, to-chk=0/3)
Completed
Creating remote mountpoint /var/lib/docker/volumes/mailcowdockerized_mysql-socket-vol-1/_data for mailcowdockerized_mysql-socket-vol-1...
Synchronizing mailcowdockerized_mysql-socket-vol-1 from local /var/lib/docker/volumes/mailcowdockerized_mysql-socket-vol-1/_data...
              0   0%    0.00kB/s    0:00:00 (xfr#0, to-chk=0/3)
Completed
Creating remote mountpoint /var/lib/docker/volumes/mailcowdockerized_mysql-vol-1/_data for mailcowdockerized_mysql-vol-1...
Creating consistent backup of MariaDB volume...
[00] 2024-03-20 14:08:52 Connecting to server host: mysql, user: root, password: set, port: not set, socket: /run/mysqld/mysqld.sock
[00] 2024-03-20 14:08:52 Using server version 10.5.24-MariaDB-1:10.5.24+maria~ubu2004
mariabackup based on MariaDB server 10.5.24-MariaDB debian-linux-gnu (x86_64)
[00] 2024-03-20 14:08:52 uses posix_fadvise().
[00] 2024-03-20 14:08:52 cd to /var/lib/mysql/
[00] 2024-03-20 14:08:52 open files limit requested 0, set to 1048576
[00] 2024-03-20 14:08:52 mariabackup: using the following InnoDB configuration:
[00] 2024-03-20 14:08:52 innodb_data_home_dir = 
[00] 2024-03-20 14:08:52 innodb_data_file_path = ibdata1:12M:autoextend
[00] 2024-03-20 14:08:52 innodb_log_group_home_dir = ./
[00] 2024-03-20 14:08:52 InnoDB: Using Linux native AIO
2024-03-20 14:08:52 0 [Note] InnoDB: Number of pools: 1
mariabackup: Can't create file '/backup/ib_logfile0' (errno: 13 "Permission denied")
[00] 2024-03-20 14:08:52 Error: failed to open the target stream for 'ib_logfile0'.
[ERR] - Could not create MariaDB backup on source

Steps to reproduce:

Running /opt/mailcowdockerized/create_cold_standby.sh as root

Which branch are you using?

master

Which architecture are you using?

x86

Operating System:

Rocky Linux 8.9

Server/VM specifications:

12G, 4 cores

Is Apparmor, SELinux or similar active?

SELinux enabled

Virtualization technology:

KVM

Docker version:

25.0.5

docker-compose version or docker compose version:

v2.25.0

mailcow version:

2024-02

Reverse proxy:

Nginx

Logs of git diff:

_cold-standby.sh

    # Make sure a previous backup does not exist
    rm -rf "${SCRIPT_DIR}/../_tmp_mariabackup/"
+    mkdir -m 777 "${SCRIPT_DIR}/../_tmp_mariabackup/"
+    chown 999:999 "${SCRIPT_DIR}/../_tmp_mariabackup/"

Logs of iptables -L -vn:

n/a

Logs of ip6tables -L -vn:

n/a

Logs of iptables -L -vn -t nat:

n/a

Logs of ip6tables -L -vn -t nat:

n/a

DNS check:

n/a
gomiunik commented 2 months ago

Hi, everyone. I've had the same issue. For me the problem was having two mailcowdockerized... networks (e.g. if your compose project name is mailcowdockerized): `docker network ls -qf name=mailcowdockerized` it returned two: mailcowdockerized_default mailcowdockerized_mailcow-network

After inspecting in docker-compose.yml which network was in use and also confirming it with docker network inspect mailcowdockerized_default (returning no attached host) i simply removed it with

docker network rm mailcowdockerized_default

and the script started working again. Hope this helps

EricThi commented 2 months ago

Thank you @gomiunik !

Adapt for me : 

docker network ls -qf name=mailcowdockerized_
ffc3f84adbb0
97222769bb79

and ;

docker network inspect mailcowdockerized_default
[
    {
        "Name": "mailcowdockerized_default",
        "Id": "ffc3f84adbb047e965b777a3aaa8bbae5355556e68945c20ad5a668204afc761",
        "Created": "2024-02-20T14:57:56.67510618+01:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "x.x.x.x/24",
                    "Gateway": "x.x.x.x"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "mailcowdockerized",
            "com.docker.compose.version": "2.24.5"
        }
    }
]

and on docker network inspect mailcowdockerized_mailcow-network => containers config founded !

docker network rm mailcowdockerized_default => relaucnh script and work very well

:)

milkmaker commented 2 days ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.