colinmollenhour / mariadb-galera-swarm

MariaDb Galera Cluster container based on official mariadb image which can auto-bootstrap and recover cluster state.
https://hub.docker.com/r/colinmollenhour/mariadb-galera-swarm
Apache License 2.0
219 stars 103 forks source link

waiting for server to become available... #110

Closed MisterDeeds closed 2 years ago

MisterDeeds commented 2 years ago

Hello together

I am new to the topic with Docker Swarm and Galera. Unfortunately I don't quite understand what the problem is. Maybe you can help me further. I have three hosts running Docker Swarm and Portainer. Other containers on them run without problems. I used the following example (https://github.com/colinmollenhour/mariadb-galera-swarm/tree/master/examples/swarm) to initialize the cluster. Here is my docker-compose.yml

version: '3.4'

services:
  seed:
    image: colinmollenhour/mariadb-galera-swarm
    environment:
      - XTRABACKUP_PASSWORD=/run/secrets/mysql.xtrabackup_password
      - MYSQL_USER=root
      - MYSQL_PASSWORD=/run/secrets/mysql.db_password
      - MYSQL_DATABASE=database
      - MYSQL_ROOT_PASSWORD=/run/secrets/mysql.root_password
      - NODE_ADDRESS=^10.0.*.*
    networks:
      - galera_network
    command: seed
    volumes:
      - mysql-data:/var/lib/mysql
    secrets:
      - mysql.xtrabackup_password
      - mysql.db_password
      - mysql.root_password
  node:
    image: colinmollenhour/mariadb-galera-swarm
    environment:
      - XTRABACKUP_PASSWORD=/run/secrets/mysql.xtrabackup_password
      - NODE_ADDRESS=^10.0.*.*
      - HEALTHY_WHILE_BOOTING=1
    networks:
      - galera_network
    command: node tasks.seed,tasks.node
    volumes:
      - mysql-data:/var/lib/mysql
    deploy:
      replicas: 0
    secrets:
      - mysql.xtrabackup_password

volumes:
  mysql-data:
    name: '{{.Service.Name}}-{{.Task.Slot}}-data'
    driver: local

networks:
  galera_network:
    driver: overlay

secrets:
  mysql.xtrabackup_password:
    external: true
  mysql.db_password:
    external: true
  mysql.root_password:
    external: true

It looks like the containers are started on the 3 hosts. Unbenannt

Here are the log files of the containers:

SrvContHost01

/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
2022/06/10 17:07:56 socat[245] E connect(5, AF=2 10.0.9.52:3309, 16): Connection timed out
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
2022/06/10 17:10:17 socat[411] E connect(5, AF=2 10.0.9.50:3309, 16): No route to host
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
2022/06/10 17:12:28 socat[416] E connect(5, AF=2 10.0.9.52:3309, 16): Connection timed out
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
2022/06/10 17:14:49 socat[579] E connect(5, AF=2 10.0.9.50:3309, 16): No route to host
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
2022/06/10 17:17:01 socat[584] E connect(5, AF=2 10.0.9.52:3309, 16): Connection timed out
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
2022/06/10 17:19:22 socat[750] E connect(5, AF=2 10.0.9.50:3309, 16): No route to host
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
2022/06/10 17:21:33 socat[752] E connect(5, AF=2 10.0.9.52:3309, 16): Connection timed out
/usr/local/bin/run-upgrades.sh: waiting for server to become available...

SrvContHost02

/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
2022/06/10 17:08:56 socat[243] E connect(5, AF=2 10.0.9.55:3309, 16): Connection timed out
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
2022/06/10 17:11:07 socat[319] E connect(5, AF=2 10.0.9.52:3309, 16): Connection timed out
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
2022/06/10 17:17:44 socat[552] E connect(5, AF=2 10.0.9.52:3309, 16): Connection timed out
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
2022/06/10 17:19:55 socat[621] E connect(5, AF=2 10.0.9.55:3309, 16): Connection timed out
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
/usr/local/bin/run-upgrades.sh: waiting for server to become available...

SrvContHost03

removed '/var/lib/mysql/ibtmp1'
removed '/var/lib/mysql/aria_log.00000001'
removed '/var/lib/mysql/database/db.opt'
removed directory '/var/lib/mysql/database'
removed '/var/lib/mysql/run-upgrades.flag'
removed '/var/lib/mysql/performance_schema/db.opt'
removed directory '/var/lib/mysql/performance_schema'
removed '/var/lib/mysql/aria_log_control'
removed '/var/lib/mysql/xtrabackup_info'
WSREP_SST: [INFO] Waiting for SST streaming to complete! (20220610 17:00:18.180)
2022-06-10 17:00:19 139770785163008 [Note] WSREP: (cd2e6c85, 'tcp://0.0.0.0:4567') turning message relay requesting off
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
   joiner: => Rate:[0.00 B/s] Avg:[0.00 B/s] Elapsed:0:00:10  Bytes: 0.00 B 
/usr/local/bin/run-upgrades.sh: waiting for server to become available...
   joiner: => Rate:[1.47MiB/s] Avg:[1.47MiB/s] Elapsed:0:00:12  Bytes: 18.1MiB 
2022-06-10 17:00:30 139770776770304 [Note] WSREP: 1.0 (3777a420f238): State transfer to 0.0 (79dc26a8345d) complete.
2022-06-10 17:00:30 139770776770304 [Note] WSREP: Member 1.0 (3777a420f238) synced with group.
WSREP_SST: [INFO] Preparing the backup at /var/lib/mysql//.sst (20220610 17:00:30.581)
WSREP_SST: [INFO] Evaluating /usr//bin/mariabackup --innobackupex   $INNOEXTRA --apply-log $rebuildcmd ${DATA} &> /var/lib/mysql//mariabackup.prepare.log (20220610 17:00:30.588)
WSREP_SST: [INFO] Moving the backup to /var/lib/mysql/ (20220610 17:00:31.191)
WSREP_SST: [INFO] Evaluating /usr//bin/mariabackup --innobackupex       --move-back --force-non-empty-directories ${DATA} &> /var/lib/mysql//mariabackup.move.log (20220610 17:00:31.195)
WSREP_SST: [INFO] Move successful, removing /var/lib/mysql//.sst (20220610 17:00:31.226)
WSREP_SST: [INFO] Galera co-ords from recovery: 9b1dad0a-e8de-11ec-89d9-a72dd5e286a0:0 (20220610 17:00:31.236)
WSREP_SST: [INFO] Total time on joiner: 0 seconds (20220610 17:00:31.243)
WSREP_SST: [INFO] Removing the sst_in_progress file (20220610 17:00:31.247)
2022-06-10 17:00:31 139773167150400 [Note] WSREP: SST complete, seqno: 0
2022-06-10 17:00:31 139773167150400 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2022-06-10 17:00:31 139773167150400 [Note] InnoDB: Uses event mutexes
2022-06-10 17:00:31 139773167150400 [Note] InnoDB: Compressed tables use zlib 1.2.11
2022-06-10 17:00:31 139773167150400 [Note] InnoDB: Using Linux native AIO
2022-06-10 17:00:31 139773167150400 [Note] InnoDB: Number of pools: 1
2022-06-10 17:00:31 139773167150400 [Note] InnoDB: Using SSE2 crc32 instructions
2022-06-10 17:00:31 139773167150400 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
2022-06-10 17:00:31 139773167150400 [Note] InnoDB: Completed initialization of buffer pool
2022-06-10 17:00:31 139769830000384 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2022-06-10 17:00:31 139773167150400 [Note] InnoDB: Setting log file ./ib_logfile101 size to 50331648 bytes
2022-06-10 17:00:31 139773167150400 [Note] InnoDB: Setting log file ./ib_logfile1 size to 50331648 bytes
2022-06-10 17:00:31 139773167150400 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2022-06-10 17:00:31 139773167150400 [Note] InnoDB: New log files created, LSN=1621194
2022-06-10 17:00:31 139773167150400 [Note] InnoDB: Highest supported file format is Barracuda.
2022-06-10 17:00:31 139773167150400 [Note] InnoDB: 128 out of 128 rollback segments are active.
2022-06-10 17:00:31 139773167150400 [Note] InnoDB: Creating shared tablespace for temporary tables
2022-06-10 17:00:31 139773167150400 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2022-06-10 17:00:31 139773167150400 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2022-06-10 17:00:31 139773167150400 [Note] InnoDB: 5.7.25 started; log sequence number 1621516
2022-06-10 17:00:31 139769669461760 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2022-06-10 17:00:31 139769669461760 [Note] InnoDB: Buffer pool(s) load completed at 220610 17:00:31
2022-06-10 17:00:31 139773167150400 [Note] Plugin 'FEEDBACK' is disabled.
2022-06-10 17:00:31 139773167150400 [Note] Recovering after a crash using tc.log
2022-06-10 17:00:31 139773167150400 [Note] Starting crash recovery...
2022-06-10 17:00:31 139773167150400 [Note] Crash recovery finished.
2022-06-10 17:00:31 139773167150400 [Note] Server socket created on IP: '0.0.0.0'.
2022-06-10 17:00:31 139773167150400 [Warning] 'user' entry 'root@c736207c9344' ignored in --skip-name-resolve mode.
2022-06-10 17:00:31 139773167150400 [Warning] 'proxies_priv' entry '@% root@c736207c9344' ignored in --skip-name-resolve mode.
2022-06-10 17:00:31 139773167150400 [Note] WSREP: Signalling provider to continue.
2022-06-10 17:00:31 139773167150400 [Note] WSREP: SST received: 9b1dad0a-e8de-11ec-89d9-a72dd5e286a0:0
2022-06-10 17:00:31 139770776770304 [Note] WSREP: 0.0 (79dc26a8345d): State transfer from 1.0 (3777a420f238) complete.
2022-06-10 17:00:31 139770776770304 [Note] WSREP: Shifting JOINER -> JOINED (TO: 0)
2022-06-10 17:00:31 139773167150400 [Note] Reading of all Master_info entries succeded
2022-06-10 17:00:31 139773167150400 [Note] Added new Master_info '' to hash table
2022-06-10 17:00:31 139773167150400 [Note] mysqld: ready for connections.
Version: '10.2.23-MariaDB-1:10.2.23+maria~bionic'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2022-06-10 17:00:31 139770776770304 [Note] WSREP: Member 0.0 (79dc26a8345d) synced with group.
2022-06-10 17:00:31 139770776770304 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 0)
2022-06-10 17:00:31 139773001697024 [Note] WSREP: Synchronized with group, ready for connections
2022-06-10 17:00:31 139773001697024 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2022-06-10 17:02:05 139770785163008 [Note] WSREP: (cd2e6c85, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.0.9.50:4567 
2022-06-10 17:02:06 139770785163008 [Note] WSREP: forgetting f92192ae (tcp://10.0.9.50:4567)
2022-06-10 17:02:06 139770785163008 [Note] WSREP: (cd2e6c85, 'tcp://0.0.0.0:4567') turning message relay requesting off
2022-06-10 17:02:06 139770785163008 [Note] WSREP: Node cd2e6c85 state prim
2022-06-10 17:02:06 139770785163008 [Note] WSREP: view(view_id(PRIM,cd2e6c85,12) memb {
    cd2e6c85,0
} joined {
} left {
} partitioned {
    f92192ae,0
})
2022-06-10 17:02:06 139770785163008 [Note] WSREP: save pc into disk
2022-06-10 17:02:06 139770785163008 [Note] WSREP: forgetting f92192ae (tcp://10.0.9.50:4567)
2022-06-10 17:02:06 139770776770304 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
2022-06-10 17:02:06 139770776770304 [Note] WSREP: STATE_EXCHANGE: sent state UUID: 0e888a95-e8df-11ec-80fa-0f2c40263987
2022-06-10 17:02:06 139770776770304 [Note] WSREP: STATE EXCHANGE: sent state msg: 0e888a95-e8df-11ec-80fa-0f2c40263987
2022-06-10 17:02:06 139770776770304 [Note] WSREP: STATE EXCHANGE: got state msg: 0e888a95-e8df-11ec-80fa-0f2c40263987 from 0 (79dc26a8345d)
2022-06-10 17:02:06 139770776770304 [Note] WSREP: Quorum results:
    version    = 4,
    component  = PRIMARY,
    conf_id    = 2,
    members    = 1/1 (joined/total),
    act_id     = 0,
    last_appl. = 0,
    protocols  = 0/9/3 (gcs/repl/appl),
    group UUID = 9b1dad0a-e8de-11ec-89d9-a72dd5e286a0
2022-06-10 17:02:06 139770776770304 [Note] WSREP: Flow-control interval: [16, 16]
2022-06-10 17:02:06 139770776770304 [Note] WSREP: Trying to continue unpaused monitor
2022-06-10 17:02:06 139773001697024 [Note] WSREP: New cluster view: global state: 9b1dad0a-e8de-11ec-89d9-a72dd5e286a0:0, view# 3: Primary, number of nodes: 1, my index: 0, protocol version 3
2022-06-10 17:02:06 139773001697024 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2022-06-10 17:02:06 139773001697024 [Note] WSREP: REPL Protocols: 9 (4, 2)
2022-06-10 17:02:06 139773001697024 [Note] WSREP: Assign initial position for certification: 0, protocol version: 4
2022-06-10 17:02:06 139773035267840 [Note] WSREP: Service thread queue flushed.
2022-06-10 17:02:11 139770785163008 [Note] WSREP:  cleaning up f92192ae (tcp://10.0.9.50:4567)

Unfortunately, I do not understand the meaning of NODE_ADDRESS. Furthermore it is not clear to me whether the ports e.g. 3306 must be released or not?

Sorry for my ignorance and already many thanks for the help

Many greetings

KKWaxy commented 2 years ago

Hello, The cluster still bootstraping. You need to wait until the cluster bootstraped.

MisterDeeds commented 2 years ago

Hello and thanks for the answer. Unfortunately, the cluster still has the same status even after a few days.

Node 1 1

Node 2 2

Node 3 3

I have also enabled the following ports ports

My docker-compose.yml looks like this

version: '3.4'

services:
  seed:
    image: colinmollenhour/mariadb-galera-swarm
    environment:
      - XTRABACKUP_PASSWORD=/run/secrets/mysql.xtrabackup_password
      - MYSQL_USER=root
      - MYSQL_PASSWORD=/run/secrets/mysql.db_password
      - MYSQL_DATABASE=database
      - MYSQL_ROOT_PASSWORD=/run/secrets/mysql.root_password
      - NODE_ADDRESS=^10.0.*.*
    networks:
      - galera_network
    command: seed
    volumes:
      - mysql-data:/var/lib/mysql
    secrets:
      - mysql.xtrabackup_password
      - mysql.db_password
      - mysql.root_password
  node:
    image: colinmollenhour/mariadb-galera-swarm
    environment:
      - XTRABACKUP_PASSWORD=/run/secrets/mysql.xtrabackup_passwordi
      - NODE_ADDRESS=^10.0.*.*
      - HEALTHY_WHILE_BOOTING=1
    networks:
      - galera_network
    command: node tasks.seed,tasks.node
    volumes:
      - mysql-data:/var/lib/mysql
    deploy:
      replicas: 0
    secrets:
      - mysql.xtrabackup_password
    ports:
      - 3306:3306
      - 4444:4444
      - 4567:4567/tcp
      - 4567:4567/udp
      - 4568:4568
      - 8080:8080
      - 8081:8081

volumes:
  mysql-data:
    name: '{{.Service.Name}}-{{.Task.Slot}}-data'
    driver: local

networks:
  galera_network:
    driver: overlay

secrets:
  mysql.xtrabackup_password:
    external: true
  mysql.db_password:
    external: true
  mysql.root_password:
    external: true

I am not quite clear what the NODE_ADDRESS means. Does it have to correspond to the node on which the container is running?

MisterDeeds commented 2 years ago

After some time this message appears in all containers:

===|mysqld.sh|===: Could not communicate with at least 1 other nodes and no nodes are up... Giving up!
===|mysqld.sh|===: Refusing to start since something is seriously wrong..
===|mysqld.sh|===: Touch /var/lib/mysql/new-cluster to force a node to start a new cluster.
===|mysqld.sh|===: 
===|mysqld.sh|===:       VvVvVv         
===|mysqld.sh|===:       |-  -|    //   
===|mysqld.sh|===:  <----|O  O|---<<<   
===|mysqld.sh|===:       |  D |    \\ 
===|mysqld.sh|===:       | () |         
===|mysqld.sh|===:        \__/         
===|mysqld.sh|===: 
MariaDB exited with return code (0)
# GALERA saved state

version: 2.1
uuid:    00000000-0000-0000-0000-000000000000
seqno:   -1
safe_to_bootstrap: 1
colinmollenhour commented 2 years ago

The NODE_ADDRESS env variable corresponds to the Galera --wsrep-node-address variable - the nodes must be able to communicate with each other on this IP address. I don't know anything about Portainer, but is this address one that nodes can reach each-other on? You can exec into the container and try ip addr and getent hosts $(hostname) to see what the different interface addresses are and then try tools like ping and telnet to see if a connection can be established between nodes. Once you find an interface that works you can set NODE_ADDRESS to either the interface name or a pattern that matches the IP and you should be good to go.