search

canonical / mysql-k8s-operator

A Charmed Operator for running MySQL on Kubernetes
https://charmhub.io/mysql-k8s
Apache License 2.0
8 stars 16 forks source link

Mysql-k8s breaks after scale down 0 #409

Open pedrofragola opened 7 months ago

pedrofragola commented 7 months ago

[Issue description] Using the juju and charms versions below:

juju status Model Controller Cloud/Region Version SLA Timestamp test my-k8s-localhost my-k8s/localhost 3.2.4 unsupported 18:07:02-03:00

App Version Status Scale Charm Channel Rev Address Exposed Message mysql-k8s waiting 1 mysql-k8s 8.0/stable 127 10.152.183.28 no installing agent mysql-test-app 0.0.2 waiting 1 mysql-test-app latest/edge 36 10.152.183.94 no installing agent

Unit Workload Agent Address Ports Message mysql-k8s/0 unknown executing 10.1.192.108
mysql-test-app/0 waiting idle 10.1.192.107

After scaling down the mysql-k8s to 0 and scaling up to 1 the mysql-k8s is broken on juju side, the k8s side shows the pod running:

NAME READY STATUS RESTARTS AGE modeloperator-847d5d9b4c-qxdxr 1/1 Running 0 9m26s mysql-k8s-0 2/2 Running 0 5m44s mysql-test-app-0 1/1 Running 0 8m37s

I was able to simulate this issue after having some app with the database relation like the app mysql-test-app-0.

DEBUG LOG: juju deploy mysql-k8s --trust --channel 8.0/stable

unit-mysql-k8s-0: 18:07:15 ERROR unit.mysql-k8s/0.juju-log database:4: Timed out waiting for k8s service to be ready Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/src/relations/mysql_provider.py", line 124, in _on_database_requested self.charm.k8s_helpers.wait_service_ready((primary_endpoint, 3306)) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/tenacity/init.py", line 289, in wrapped_f return self(f, *args, kw) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/tenacity/init.py", line 379, in call do = self.iter(retry_state=retry_state) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/tenacity/init.py", line 325, in iter raise retry_exc.reraise() File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/tenacity/init.py", line 158, in reraise raise self.last_attempt.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result return self.get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in get_result raise self._exception File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/tenacity/init.py", line 382, in call result = fn(*args, *kwargs) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/src/k8s_helpers.py", line 201, in wait_service_ready raise TimeoutError TimeoutError unit-mysql-k8s-0: 18:07:15 ERROR unit.mysql-k8s/0.juju-log database:4: Uncaught exception while in charm code: Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/./src/charm.py", line 770, in main(MySQLOperatorCharm) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/main.py", line 456, in main _emit_charm_event(charm, dispatcher.event_name) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/main.py", line 144, in _emit_charm_event event_to_emit.emit(args, kwargs) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/framework.py", line 351, in emit framework._emit(event) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/framework.py", line 853, in _emit self._reemit(event_path) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/framework.py", line 943, in _reemit custom_handler(event) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 1928, in _on_relation_changed_event getattr(self.on, "database_requested").emit( File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/framework.py", line 351, in emit framework._emit(event) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/framework.py", line 853, in _emit self._reemit(event_path) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/framework.py", line 943, in _reemit custom_handler(event) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/src/relations/mysql_provider.py", line 124, in _on_database_requested self.charm.k8s_helpers.wait_service_ready((primary_endpoint, 3306)) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/tenacity/init.py", line 289, in wrapped_f return self(f, *args, *kw) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/tenacity/init.py", line 379, in call do = self.iter(retry_state=retry_state) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/tenacity/init.py", line 325, in iter raise retry_exc.reraise() File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/tenacity/init.py", line 158, in reraise raise self.last_attempt.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result return self.get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in get_result raise self._exception File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/tenacity/init.py", line 382, in call result = fn(args, **kwargs) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/src/k8s_helpers.py", line 201, in wait_service_ready raise TimeoutError TimeoutError unit-mysql-k8s-0: 18:07:15 ERROR juju.worker.uniter.operation hook "database-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1 unit-mysql-k8s-0: 18:07:15 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook unit-mysql-k8s-0: 18:07:25 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook unit-mysql-k8s-0: 18:07:26 ERROR unit.mysql-k8s/0.juju-log database:4: Failed to get cluster status for cluster-7f7c9a3504e4f58a371acc55cca28b49 unit-mysql-k8s-0: 18:07:26 ERROR unit.mysql-k8s/0.juju-log database:4: Failed to get cluster endpoints Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/src/mysql_k8s_helpers.py", line 836, in update_endpoints rw_endpoints, ro_endpoints, offline = self.get_cluster_endpoints(get_ips=False) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/lib/charms/mysql/v0/mysql.py", line 1469, in get_cluster_endpoints raise MySQLGetClusterEndpointsError("Failed to get endpoints from cluster status") charms.mysql.v0.mysql.MySQLGetClusterEndpointsError: Failed to get endpoints from cluster status unit-mysql-k8s-0: 18:07:26 WARNING unit.mysql-k8s/0.juju-log database:4: Kubernetes service already exists

So we need a way to fix the juju status.

github-actions[bot] commented 7 months ago

https://warthogs.atlassian.net/browse/DPE-4200

swasenius commented 5 months ago

I assume there is no update on this? Just lost a database once again in kubernetes environment.

paulomach commented 5 months ago

@swasenius There's some details in discussion, but support for scale up from zero will come on the linked PR.

sombrafam commented 4 months ago

I assume there is no update on this? Just lost a database once again in kubernetes environment.

Can you give details on why you needed to scale it down?