search

canonical / mysql-k8s-operator

A Charmed Operator for running MySQL on Kubernetes
https://charmhub.io/mysql-k8s
Apache License 2.0
8 stars 15 forks source link

"Lost connection to MySQL server" while deploying to an AKS cluster #394

Closed orfeas-k closed 2 months ago

orfeas-k commented 6 months ago

I deployed mysql-k8s from 8.0/stable (rev 113) to AKS (as part of CKF) and stumbled on the following errors (both katib-db and kfp-db use mysql-k8s): logs. Here are also the full logs of the CI run ci-logs.txt

Steps to reproduce

This is intermittent but deploying CKF to an AKS cluster is what is needed in order to reproduce. You can trigger the Create an AKS cluster and deploy CKF workflow that CKF team has (it will be merged in the following days).

Expected behavior

MySQL server is healthy and functional

Actual behavior

MySQL server isn't responding

Versions

Operating system: Ubuntu 22.04.4 LTS Juju CLI: 3.1.7-genericlinux-amd64 Juju agent: 3.1.7 Charm revision: 113 AKS: 1.26

Log output

Juju debug log: juju-debug-log-replay.txt

github-actions[bot] commented 6 months ago

https://warthogs.atlassian.net/browse/DPE-3826

orfeas-k commented 6 months ago

MySQL failed again in the CI when deploying to AKS with the following logs in this run.

This time it was deployed from 8.0/edge and it uses revision 136. This is something that I 'm getting let's say 1 out of 5 times or so when deploying to AKS. (you can see that one MySQL instance is healthy in the same deployment)

Here are the complete CI logs for future reference logs_22556141130.zip.

paulomach commented 5 months ago

For record, I could not reproduce it after multiple tries on AKS, using same setup as the test. I believe this bug is related to juju/pebble lp#2052517, fixed on 3.4.1+

paulomach commented 2 months ago

Hi @orfeas-k this was released/fixed on PR #458 and it's in edge.