search

percona / everest

Percona Everest is a cloud-native database platform to deploy and manage enterprise-grade PostgreSQL, MongoDB and MySQL database clusters.
https://docs.percona.com/everest/index.html
Apache License 2.0
137 stars 11 forks source link

MySQL database not starting #578

Open muhammadhur2 opened 1 month ago

muhammadhur2 commented 1 month ago

For some reason I am unable to start MySQL database in Percona Everest. I am however able to start other databases i.e. MongoDB and PostgreSQL.

I have tried with 1 node and 3 nodes and neither works.

These are all the pods in the namespace.

image

I have added all the logs file in this repo: Log Files

oksana-grishchenko commented 1 month ago

Hi @muhammadhur2! Thank you for reporting an issue.

To understand the problem better and investigate it, it would be great to have also:

muhammadhur2 commented 1 month ago

Hello @oksana-grishchenko I have updated the above log files with the files required. I have, however, hidden the ips used in the yaml file.

This is a new cluster, however, as I had deleted the other one. I still have the same issue though.

hors commented 1 month ago

Hi @muhammadhur2, as I can see you have some communication problems between haproxy and pxc pods. It can be problems with password or network problems:

    Warning  Unhealthy  39m (x18 over 52m)    kubelet            Liveness probe failed: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 2
  Warning  BackOff    18m (x9 over 20m)     kubelet            Back-off restarting failed container haproxy in pod mysql-lz4-haproxy-0_everest(70ae1edb-37b9-40a7-b7d4-4bcd836951d6)
  Warning  Unhealthy  14m (x71 over 51m)    kubelet            Readiness probe failed: ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1:33062' (111)
  Warning  Unhealthy  4m4s (x464 over 53m)  kubelet            Readiness probe failed: ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 2

As you can see https://github.com/percona/percona-docker/blob/main/haproxy/dockerdir/usr/local/bin/liveness-check.sh liveness probe gets password from secret (which is mounted) and use it for connections.

It can be some network problems. Also, it will be nice to see log from pxc-monit container (haproxy). One more question: which platform are you using (EKS, GKE)?