search

canonical / wordpress-k8s-operator

wordpress-k8s-operator - charm repository.
Apache License 2.0
13 stars 7 forks source link

Cannot re-relate with mysql-k8s (8.0/stable) #133

Open taurus-forever opened 1 year ago

taurus-forever commented 1 year ago

Bug Description

Hi,

I am experiencing an issues on initial relation between mysql-k8s (revision 99, from 8.0/stable):

Issue 1: Initial bootstrap stuck IF relation requested immediately:

juju deploy mysql-k8s --trust
juju deploy wordpress-k8s
juju relate mysql-k8s wordpress-k8s

Actual results:

Unit               Workload  Agent  Address     Ports  Message
mysql-k8s/0*       active    idle   10.1.84.97         Primary
mysql-test-app/0*  waiting   idle   10.1.84.84

Error log is empty and I see properly populated databag by mysql-k8s:

  - relation-id: 3
    endpoint: database
    related-endpoint: database
    application-data:
      data: '{"database": "wordpress-k8s"}'
      database: wordpress-k8s
      endpoints: mysql-k8s-primary.test7.svc.cluster.local:3306
      password: Cwck7x8MuBpZPPWZh1fB6icW
      read-only-endpoints: mysql-k8s-replicas.test7.svc.cluster.local:3306
      username: relation-3
      version: 8.0.34-0ubuntu0.22.04.1

NOTE: the is no such issue1 IF relation between MySQL and Wordpress established after the full MySQL deployment:

juju deploy mysql-k8s --trust
juju wait-for application mysql-k8s
juju deploy wordpress-k8s
juju relate mysql-k8s wordpress-k8s

Any ideas what can be missing here?

Issue 2: It is not possible to remove relation if it stuck on issue1 described above:

juju remove-relation mysql-k8s wordpress-k8s

juju debug-log --tail
ubuntu@taurus-dev:~$ unit-mysql-k8s-0: 14:28:52 DEBUG unit.mysql-k8s/0.juju-log database:3: ops 2.6.0 up and running.
unit-mysql-k8s-0: 14:28:52 DEBUG unit.mysql-k8s/0.juju-log database:3: Emitting Juju event database_relation_departed.
unit-mysql-k8s-0: 14:28:52 DEBUG unit.mysql-k8s/0.juju-log database:3: Retrieved secret root-password for app
unit-mysql-k8s-0: 14:28:52 DEBUG unit.mysql-k8s/0.juju-log database:3: Retrieved secret server-config-password for app
unit-mysql-k8s-0: 14:28:52 DEBUG unit.mysql-k8s/0.juju-log database:3: Retrieved secret cluster-admin-password for app
unit-mysql-k8s-0: 14:28:52 DEBUG unit.mysql-k8s/0.juju-log database:3: Retrieved secret monitoring-password for app
unit-mysql-k8s-0: 14:28:52 DEBUG unit.mysql-k8s/0.juju-log database:3: Retrieved secret backups-password for app
unit-mysql-k8s-0: 14:28:53 INFO juju.worker.uniter.operation ran "database-relation-departed" hook (via hook dispatching script: dispatch)
unit-mysql-k8s-0: 14:28:53 DEBUG unit.mysql-k8s/0.juju-log database:3: ops 2.6.0 up and running.
unit-mysql-k8s-0: 14:28:53 DEBUG unit.mysql-k8s/0.juju-log database:3: Emitting Juju event database_relation_broken.
unit-mysql-k8s-0: 14:28:53 INFO unit.mysql-k8s/0.juju-log database:3: Kubernetes service mysql-k8s-primary deleted
unit-mysql-k8s-0: 14:28:53 INFO unit.mysql-k8s/0.juju-log database:3: Kubernetes service mysql-k8s-replicas deleted
unit-mysql-k8s-0: 14:28:53 DEBUG unit.mysql-k8s/0.juju-log database:3: Retrieved secret root-password for app
unit-mysql-k8s-0: 14:28:53 DEBUG unit.mysql-k8s/0.juju-log database:3: Retrieved secret server-config-password for app
unit-mysql-k8s-0: 14:28:53 DEBUG unit.mysql-k8s/0.juju-log database:3: Retrieved secret cluster-admin-password for app
unit-mysql-k8s-0: 14:28:53 DEBUG unit.mysql-k8s/0.juju-log database:3: Retrieved secret monitoring-password for app
unit-mysql-k8s-0: 14:28:53 DEBUG unit.mysql-k8s/0.juju-log database:3: Retrieved secret backups-password for app
unit-mysql-k8s-0: 14:28:53 INFO unit.mysql-k8s/0.juju-log database:3: Removed user(s) for relation 3
unit-mysql-k8s-0: 14:28:54 INFO juju.worker.uniter.operation ran "database-relation-broken" hook (via hook dispatching script: dispatch)

And relation stuck here in dying state:

> juju relate mysql-k8s wordpress-k8s
ERROR cannot add relation "wordpress-k8s:database mysql-k8s:database"
relation wordpress-k8s:database mysql-k8s:database is dying, but not yet removed (already exists): 

Use 'juju status --relations' to view the current relations.

Maybe it worth to update data-platform-libs? Good ideas are welcome!

To Reproduce

Issue 1: juju deploy mysql-k8s --trust juju deploy wordpress-k8s juju relate mysql-k8s wordpress-k8s

Issue 2: juju remove-relation mysql-k8s wordpress-k8s juju relate mysql-k8s wordpress-k8s

Environment

juju 3.1.6-f6a66aa 24582 3.1/edge canonical✓ - microk8s v1.28.0 5802 1.28-strict/stable canonical✓ -

ubuntu@taurus-dev:~$ juju status --relations
Model  Controller  Cloud/Region        Version  SLA          Timestamp
test8  k8s6        microk8s/localhost  3.1.6    unsupported  14:32:35+02:00

App            Version                  Status   Scale  Charm          Channel     Rev  Address         Exposed  Message
mysql-k8s      8.0.34-0ubuntu0.22.04.1  active       1  mysql-k8s      8.0/stable   99  10.152.183.132  no       
wordpress-k8s  5.9.3                    waiting      1  wordpress-k8s  stable       42  10.152.183.111  no       installing agent

Unit              Workload     Agent  Address      Ports  Message
mysql-k8s/0*      active       idle   10.1.84.123         Primary
wordpress-k8s/0*  maintenance  idle   10.1.84.108         Starting apache-prometheus-exporter container

Integration provider             Requirer                         Interface          Type     Message
mysql-k8s:database               wordpress-k8s:database           mysql_client       regular  
mysql-k8s:database-peers         mysql-k8s:database-peers         mysql_peers        peer     
mysql-k8s:upgrade                mysql-k8s:upgrade                upgrade            peer     
wordpress-k8s:wordpress-replica  wordpress-k8s:wordpress-replica  wordpress-replica  peer     
ubuntu@taurus-dev:~$

Relevant log output

None

Additional context

No response

mthaddon commented 1 year ago

I haven't been able to reproduce this. I just ran:

juju deploy mysql-k8s --trust
juju deploy wordpress-k8s
juju relate mysql-k8s wordpress-k8s

And it worked fine for me. Could you include the output of microk8s kubectl get pods -n $name-of-your-model? I suspect the wordpress pod was still downloading the images since it was in "waiting" status.

taurus-forever commented 1 year ago

With the latest wordpress-k8s revision 50, it stucks on installing charm software for:

juju deploy mysql-k8s --trust
juju deploy wordpress-k8s
juju relate mysql-k8s wordpress-k8s

juju status:

Model       Controller  Cloud/Region        Version  SLA          Timestamp
wordpress2  microk8s    microk8s/localhost  3.1.6    unsupported  02:01:06+02:00

App            Version                  Status   Scale  Charm          Channel     Rev  Address         Exposed  Message
mysql-k8s      8.0.34-0ubuntu0.22.04.1  active       1  mysql-k8s      8.0/stable   99  10.152.183.172  no       
wordpress-k8s  5.9.3                    waiting      1  wordpress-k8s  stable       50  10.152.183.120  no       installing agent

Unit              Workload     Agent      Address     Ports  Message
mysql-k8s/0*      active       idle       10.1.12.1          Primary
wordpress-k8s/0*  maintenance  executing  10.1.12.50         installing charm software

The pods list (one is missing):

microk8s kubectl get pods -n wordpress2
NAME                             READY   STATUS    RESTARTS   AGE
modeloperator-7dbc85757c-qqt8g   1/1     Running   0          3m19s
mysql-k8s-0                      2/2     Running   0          3m10s
wordpress-k8s-0                  2/3     Running   0          3m

but from here all resources are locally available and issue in a helthcheck, IMHO:

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  2m50s                 default-scheduler  Successfully assigned wordpress2/wordpress-k8s-0 to juju31
  Normal   Pulled     2m50s                 kubelet            Container image "jujusolutions/jujud-operator:3.1.6" already present on machine
  Normal   Created    2m50s                 kubelet            Created container charm-init
  Normal   Started    2m50s                 kubelet            Started container charm-init
  Normal   Pulled     2m50s                 kubelet            Container image "jujusolutions/charm-base:ubuntu-22.04" already present on machine
  Normal   Created    2m50s                 kubelet            Created container charm
  Normal   Started    2m49s                 kubelet            Started container charm
  Normal   Pulled     2m49s                 kubelet            Container image "registry.jujucharms.com/charm/600jll14kzaovquhnzyajwwvvcp8a6s1a2tgb/apache-prometheus-exporter-image@sha256:149dd4ba47f374919b630d4316b19f0822b7ef90eac6ca6f5f539f13f5579655" already present on machine
  Normal   Created    2m49s                 kubelet            Created container apache-prometheus-exporter
  Normal   Started    2m49s                 kubelet            Started container apache-prometheus-exporter
  Normal   Pulled     2m49s                 kubelet            Container image "registry.jujucharms.com/charm/600jll14kzaovquhnzyajwwvvcp8a6s1a2tgb/wordpress-image@sha256:71f96308ce89ddb3715feb5e1e66aa078c5cee75649f52df6621e53ff8320f0d" already present on machine
  Normal   Created    2m49s                 kubelet            Created container wordpress
  Normal   Started    2m49s                 kubelet            Started container wordpress
  Warning  Unhealthy  80s (x13 over 2m15s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 502

The full output: https://pastebin.canonical.com/p/4RZ7dw9pbd/ (see the line 77). How to trace from here?