Failed to get cluster primary addresses - Classic protocol error: MySQL Error 2013 (HY000): Lost connection to MySQL server at \'waiting for initial communication packet\', system error: 110 #482
MySQL units can be stuck with the following status in the initial deployment.
$ juju status -m openstack cinder-mysql
Model Controller Cloud/Region Version SLA Timestamp
openstack mysunbeam-controller mysunbeam-k8s/localhost 3.5.3 unsupported 02:05:50Z
SAAS Status Store URL
microceph active local admin/openstack-machines.microceph
App Version Status Scale Charm Channel Rev Address Exposed Message
cinder-mysql 8.0.37-0ubuntu0.22.04.3 maintenance 3 mysql-k8s 8.0/edge 174 10.152.183.91 no joining the cluster
Unit Workload Agent Address Ports Message
cinder-mysql/0* active idle 10.1.58.84 Primary
cinder-mysql/1 maintenance idle 10.1.48.85 joining the cluster
cinder-mysql/2 waiting idle 10.1.51.16 waiting to get cluster primary from peers
unit-cinder-mysql-2: 02:05:57 WARNING unit.cinder-mysql/2.juju-log Failed to check if cluster metadata exists from_instance='cinder-mysql-2.cinder-mysql-endpoints.openstack.svc.cluster.local'
unit-cinder-mysql-2: 02:06:38 WARNING unit.cinder-mysql/2.juju-log Failed to check if cluster metadata exists from_instance='cinder-mysql-0.cinder-mysql-endpoints.openstack.svc.cluster.local'
unit-cinder-mysql-2: 02:07:19 WARNING unit.cinder-mysql/2.juju-log Failed to get cluster primary addresses
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-cinder-mysql-2/charm/src/mysql_k8s_helpers.py", line 616, in _run_mysqlsh_script
stdout, _ = process.wait_output()
File "/var/lib/juju/agents/unit-cinder-mysql-2/charm/venv/ops/pebble.py", line 1771, in wait_output
raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
ops.pebble.ExecError: non-zero exit code 1 executing ['/usr/bin/mysqlsh', '--no-wizard', '--python', '--verbose=1', '-f', '/tmp/script.py', ';', 'rm', '/tmp/script.py'], stdout='', stderr='Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nverbose: 2024-08-15T02:06:39Z: Loading startup files...\nverbose: 2024-08-15T02:06:39Z: Loading plugins...\nverbose: 2024-08-15T02:06:39Z: Connecting to MySQL at: clusteradmin@cinder-mysql-0.cinder-mysql-endpoints.openstack.svc.cluster.local\nTraceback (most recent call last):\n File "<string>", line 1, in <module>\nValueError: Shell.connect_to_primary: X protocol error: MySQL server has gone away\nClassic protocol error: MySQL Error 2013 (HY000): Lost connection to MySQL server at \'waiting for initial communication packet\', system error: 110\n\n'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-cinder-mysql-2/charm/lib/charms/mysql/v0/mysql.py", line 2117, in get_cluster_primary_address
output = self._run_mysqlsh_script("\n".join(get_cluster_primary_commands))
File "/var/lib/juju/agents/unit-cinder-mysql-2/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 724, in wrapped_function
return callable(*args, **kwargs) # type: ignore
File "/var/lib/juju/agents/unit-cinder-mysql-2/charm/src/mysql_k8s_helpers.py", line 619, in _run_mysqlsh_script
raise MySQLClientError
charms.mysql.v0.mysql.MySQLClientError
github-actions[bot]
commented
3 months ago
Steps to reproduce
Expected behavior
All MySQL units get green.
Actual behavior
MySQL units can be stuck with the following status in the initial deployment.
Versions
Operating system: jammy
Juju CLI: 3.5.3-genericlinux-amd64
Juju agent: 3.5.3
Charm revision: 8.0/edge 174
microk8s: MicroK8s v1.28.12 revision 7038
Log output
Juju debug log: mysql_charm.log
Additional context