search

canonical / mysql-k8s-operator

A Charmed Operator for running MySQL on Kubernetes
https://charmhub.io/mysql-k8s
Apache License 2.0
8 stars 15 forks source link

charms.mysql.v0.mysql.MySQLServiceNotRunningError #447

Closed cjohnston1158 closed 1 week ago

cjohnston1158 commented 1 week ago

Steps to reproduce

Deploying sunbeam using the multi-node maas instructions.

Expected behavior

No error

Actual behavior

2024-06-27T22:36:05.375Z [pebble] Check "readiness" failure 22 (threshold 3): received non-20x status code 418
2024-06-27T22:36:14.079Z [container-agent] 2024-06-27 22:36:14 ERROR juju-log Uncaught exception while in charm code:
2024-06-27T22:36:14.079Z [container-agent] Traceback (most recent call last):
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/./src/charm.py", line 839, in <module>
2024-06-27T22:36:14.079Z [container-agent]     main(MySQLOperatorCharm)
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/ops/main.py", line 548, in main
2024-06-27T22:36:14.079Z [container-agent]     manager.run()
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/ops/main.py", line 527, in run
2024-06-27T22:36:14.079Z [container-agent]     self._emit()
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/ops/main.py", line 516, in _emit
2024-06-27T22:36:14.079Z [container-agent]     _emit_charm_event(self.charm, self.dispatcher.event_name)
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/ops/main.py", line 147, in _emit_charm_event
2024-06-27T22:36:14.079Z [container-agent]     event_to_emit.emit(*args, **kwargs)
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/ops/framework.py", line 348, in emit
2024-06-27T22:36:14.079Z [container-agent]     framework._emit(event)
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/ops/framework.py", line 860, in _emit
2024-06-27T22:36:14.079Z [container-agent]     self._reemit(event_path)
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/ops/framework.py", line 950, in _reemit
2024-06-27T22:36:14.079Z [container-agent]     custom_handler(event)
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 544, in wrapped_function
2024-06-27T22:36:14.079Z [container-agent]     return callable(*args, **kwargs)  # type: ignore
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/./src/charm.py", line 648, in _on_mysql_pebble_ready
2024-06-27T22:36:14.079Z [container-agent]     self._configure_instance(container)
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 544, in wrapped_function
2024-06-27T22:36:14.079Z [container-agent]     return callable(*args, **kwargs)  # type: ignore
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/./src/charm.py", line 577, in _configure_instance
2024-06-27T22:36:14.079Z [container-agent]     self._mysql.wait_until_mysql_connection(check_port=False)
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/lib/charms/tempo_k8s/v1/charm_tracing.py", line 544, in wrapped_function
2024-06-27T22:36:14.079Z [container-agent]     return callable(*args, **kwargs)  # type: ignore
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/tenacity/__init__.py", line 330, in wrapped_f
2024-06-27T22:36:14.079Z [container-agent]     return self(f, *args, **kw)
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/tenacity/__init__.py", line 467, in __call__
2024-06-27T22:36:14.079Z [container-agent]     do = self.iter(retry_state=retry_state)
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/tenacity/__init__.py", line 368, in iter
2024-06-27T22:36:14.079Z [container-agent]     result = action(retry_state)
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/tenacity/__init__.py", line 410, in exc_check
2024-06-27T22:36:14.079Z [container-agent]     raise retry_exc.reraise()
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/tenacity/__init__.py", line 183, in reraise
2024-06-27T22:36:14.079Z [container-agent]     raise self.last_attempt.result()
2024-06-27T22:36:14.079Z [container-agent]   File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2024-06-27T22:36:14.079Z [container-agent]     return self.__get_result()
2024-06-27T22:36:14.079Z [container-agent]   File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2024-06-27T22:36:14.079Z [container-agent]     raise self._exception
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/venv/tenacity/__init__.py", line 470, in __call__
2024-06-27T22:36:14.079Z [container-agent]     result = fn(*args, **kwargs)
2024-06-27T22:36:14.079Z [container-agent]   File "/var/lib/juju/agents/unit-cinder-mysql-0/charm/src/mysql_k8s_helpers.py", line 228, in wait_until_mysql_connection
2024-06-27T22:36:14.079Z [container-agent]     raise MySQLServiceNotRunningError
2024-06-27T22:36:14.079Z [container-agent] charms.mysql.v0.mysql.MySQLServiceNotRunningError

Versions

juju status -m openstack cinder-mysql
Model      Controller            Cloud/Region                Version  SLA          Timestamp
openstack  sunbeam01-controller  sunbeam-microk8s/localhost  3.4.3    unsupported  00:40:14Z

SAAS       Status  Store  URL
microceph  active  local  admin/openstack-machines.microceph

App           Version                  Status   Scale  Charm      Channel        Rev  Address         Exposed  Message
cinder-mysql  8.0.36-0ubuntu0.22.04.1  waiting      3  mysql-k8s  8.0/candidate  153  10.152.183.119  no       installing agent

Unit             Workload     Agent      Address       Ports  Message
cinder-mysql/0*  maintenance  executing  10.1.164.214         Initialising mysqld
cinder-mysql/1   waiting      idle       10.1.89.135          waiting to get cluster primary from peers
cinder-mysql/2   waiting      idle       10.1.57.22           waiting to get cluster primary from peers
openstack  2024.1              540    2024.1/edge    canonical✓  -
microceph  18.2.0+snapf263b74280  1047   reef/edge           canonical✓  held
microk8s   v1.28.10               6829   1.28-strict/stable  canonical✓  -

cinder-mysql-0-charm.log cinder-mysql-0-describe.txt cinder-mysql-0-mysql.log

github-actions[bot] commented 1 week ago

https://warthogs.atlassian.net/browse/DPE-4800

paulomach commented 1 week ago

@cjohnston1158 Thanks for the report - I can't reproduce it locally still. But i think that's a transient issue and the solution is two fold:

While I am at it, if you see it again, please capture the /var/log/mysql/error.log inside the mysql container for the pod. I just wan't to be sure the daemon is starting correctly and it's a matter of time.

cjohnston1158 commented 1 week ago 
-> Start:
2024-06-28T19:01:38.801074Z mysqld_safe Logging to '/var/log/mysql/error.log'.

--> ready:
2024-06-28T19:02:13.139958Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.36-0ubuntu0.22.04.1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).

Currently we are waiting 30s to have socket connection. From the logs it's clear that it's taking longer (~35s) which is triggering the initial hook failure. Fix is on the way