search

canonical / mysql-k8s-operator

A Charmed Operator for running MySQL on Kubernetes
https://charmhub.io/mysql-k8s
Apache License 2.0
8 stars 15 forks source link

one out of 3 unit stuck in "Waiting for leader election" #466

Open jeffreychang911 opened 1 month ago

jeffreychang911 commented 1 month ago

Steps to reproduce

  1. SolQA deployed 3 mysql-k8s on top of charmed k8s. also added self-signed-certificates and data-integrator.

Expected behavior

All mysql-k8s instance should settle in few minutes.

Actual behavior

The unit stuck for an hour until test timeout.

Versions

Operating system:

Juju CLI: 3.5.2

Juju agent: 3.5.2

Charm revision: mysql-k8s rev 153 from latest/stable

Charmed Kubernetes 1.30

Log output

No error found, just post log from that unit. Juju debug log: unit-mysql-k8s-1: 2024-07-22 00:06:49 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:06:51 INFO juju.worker.uniter.operation ran "upgrade-relation-joined" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:06:53 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:06:55 INFO juju.worker.uniter.operation ran "certificates-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:06:57 INFO juju.worker.uniter.operation ran "database-peers-relation-joined" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:06:59 INFO juju.worker.uniter.operation ran "upgrade-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:07:01 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:07:03 INFO juju.worker.uniter.operation ran "certificates-relation-joined" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:07:05 INFO juju.worker.uniter.operation ran "upgrade-relation-joined" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:07:06 INFO juju.worker.uniter.operation ran "upgrade-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:07:08 INFO juju.worker.uniter.operation ran "certificates-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:07:10 INFO juju.worker.uniter.operation ran "restart-relation-joined" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:07:12 INFO juju.worker.uniter.operation ran "database-relation-joined" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:07:13 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:07:15 INFO juju.worker.uniter.operation ran "restart-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:07:17 INFO juju.worker.uniter.operation ran "database-peers-relation-joined" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:07:19 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:07:21 INFO juju.worker.uniter.operation ran "restart-relation-joined" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:07:23 INFO juju.worker.uniter.operation ran "restart-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:07:25 INFO juju.worker.uniter.operation ran "upgrade-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:07:27 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:07:29 INFO juju.worker.uniter.operation ran "upgrade-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:08:01 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:08:30 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:08:32 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:08:34 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:08:58 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:09:08 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:09:11 INFO juju.worker.uniter.operation ran "certificates-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:09:18 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:09:20 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:09:23 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:09:25 INFO juju.worker.uniter.operation ran "certificates-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:09:34 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:09:38 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:11:46 INFO unit.mysql-k8s/1.juju-log Unit state is waiting unit-mysql-k8s-1: 2024-07-22 00:11:46 INFO unit.mysql-k8s/1.juju-log Unit state is waiting unit-mysql-k8s-1: 2024-07-22 00:11:47 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch) unit-mysql-k8s-1: 2024-07-22 00:16:30 INFO unit.mysql-k8s/1.juju-log Unit state is waiting unit-mysql-k8s-1: 2024-07-22 00:16:30 INFO unit.mysql-k8s/1.juju-log Unit state is waiting unit-mysql-k8s-1: 2024-07-22 00:16:31 INFO juju.worker.uniter.operation ran "update-status" hook (via hook dispatching script: dispatch)

Additional context

This is from SolQA test run - https://solutions.qa.canonical.com/testruns/5b7ddf03-6f20-436a-bde6-a4ef8a1c7b7b crashdump - https://oil-jenkins.canonical.com/artifacts/5b7ddf03-6f20-436a-bde6-a4ef8a1c7b7b/generated/generated/mysql-k8s/crashdump-2024-07-22-01.06.00.tar.gz pods logs available in https://oil-jenkins.canonical.com/artifacts/5b7ddf03-6f20-436a-bde6-a4ef8a1c7b7b/generated/generated/kubernetes-openstack/juju-crashdump-kubernetes-openstack-2024-07-22-01.08.14.tar.gz Will attach more occurrence if we see this again.

github-actions[bot] commented 1 month ago

https://warthogs.atlassian.net/browse/DPE-4910