search

canonical / mysql-router-k8s-operator

Mysql router operator charm for kubernetes
https://charmhub.io/mysql-router-k8s
Apache License 2.0
2 stars 7 forks source link

Bootstrap fails when relating to a different cluster #67

Closed carlcsaposs-canonical closed 1 year ago

carlcsaposs-canonical commented 1 year ago

Steps to reproduce

  1. juju deploy mysql-k8s --channel 8.0/edge --trust
  2. juju deploy mysql-router-k8s --channel 8.0/edge
  3. juju relate mysql-k8s mysql-router-k8s
  4. Wait for active/idle on mysql-k8s and blocked/idle on mysql-router-k8s
  5. juju remove-relation mysql-k8s mysql-router-k8s
  6. juju deploy mysql-k8s mysql-2nd-k8s --channel 8.0/edge --trust
  7. juju relate mysql-2nd-k8s mysql-router-k8s
  8. Wait for mysql-2nd-k8s to start

Expected behavior

Active/idle on mysql-k8s and blocked/idle on mysql-router-k8s

Actual behavior

mysql-router-k8s error hook failed: "backend-database-relation-changed" for mysql-2nd-k8s:database

Versions

Operating system: Ubuntu 22.04.2 LTS

Juju CLI: 2.9.43-ubuntu-amd64

Juju agent: 2.9.43 (installed following these instructions: https://chat.canonical.com/canonical/pl/g6swyah9zty15kx1w81e31u3jy)

mysql-k8s charm revision: 80 mysql-router-k8s charm revision: 31

microk8s: MicroK8s v1.26.4 revision 5219

Log output

unit-mysql-router-k8s-0: 19:24:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Operator Framework 2.2.0 up and running.
unit-mysql-router-k8s-0: 19:24:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Emitting Juju event backend_database_relation_changed.
unit-mysql-router-k8s-0: 19:24:28 INFO unit.mysql-router-k8s/0.juju-log backend-database:4: database created at 2023-05-18 19:24:28.351091
unit-mysql-router-k8s-0: 19:24:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Emitting custom event <DatabaseCreatedEvent via MySQLRouterOperatorCharm/DatabaseRequires[backend-database]/on/database_created[48]>.
unit-mysql-router-k8s-0: 19:24:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: State of reconcile self.unit.is_leader()=True, isinstance(self.workload, workload.AuthenticatedWorkload)=True, self.database_requires.relation and self.database_requires.relation.is_breaking(event)=False, self.workload.container_ready=True, isinstance(event, ops.UpgradeCharmEvent)=False
unit-mysql-router-k8s-0: 19:24:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Reconciling users event=<DatabaseCreatedEvent via MySQLRouterOperatorCharm/DatabaseRequires[backend-database]/on/database_created[48]>, router_endpoint='mysql-router-k8s.foo46.svc.cluster.local'
unit-mysql-router-k8s-0: 19:24:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: State of reconcile users requested_users=[], created_users=[]
unit-mysql-router-k8s-0: 19:24:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Reconciled users event=<DatabaseCreatedEvent via MySQLRouterOperatorCharm/DatabaseRequires[backend-database]/on/database_created[48]>, router_endpoint='mysql-router-k8s.foo46.svc.cluster.local'
unit-mysql-router-k8s-0: 19:24:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Enabling MySQL Router service
unit-mysql-router-k8s-0: 19:24:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Bootstrapping router tls=False, self._database_requires_relation.host='mysql-2nd-k8s-primary.foo46.svc.cluster.local', self._database_requires_relation.port='3306'
unit-mysql-router-k8s-0: 19:24:28 ERROR unit.mysql-router-k8s/0.juju-log backend-database:4: Failed to bootstrap router
stderr:
Error: The given Router instance is already configured for a cluster named 'cluster_3e97a9b353fcfb511f0181d85ae3dd44'.
If you'd like to replace it, please use the --force configuration option.

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/src/workload.py", line 175, in _bootstrap_router
    process.wait_output()
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/pebble.py", line 1252, in wait_output
    raise ExecError(self._command, exit_code, out_value, err_value)
ops.pebble.ExecError: non-zero exit code 1 executing ['mysqlrouter', '--bootstrap', 'relation-4:ZJfWC6Jq12uldwx8Px0HcBMz@mysql-2nd-k8s-primary.foo46.svc.cluster.local:3306', '--strict', '--user', 'mysql', '--conf-set-option', 'http_server.bind_address=127.0.0.1', '--conf-use-gr-notifications'], stdout='', stderr="Error: The given Router instance is already configured for a cluster named 'cluster_3e97a9b353fcfb511f0181d85ae3dd44'.\nIf you'd like to replace it, please use the --force configuration option.\n"
unit-mysql-router-k8s-0: 19:24:28 ERROR unit.mysql-router-k8s/0.juju-log backend-database:4: Uncaught exception while in charm code:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/./src/charm.py", line 250, in <module>
    ops.main.main(MySQLRouterOperatorCharm)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/main.py", line 441, in main
    _emit_charm_event(charm, dispatcher.event_name)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/main.py", line 149, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 354, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 830, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 919, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 946, in _on_relation_changed_event
    self.on.database_created.emit(event.relation, app=event.app, unit=event.unit)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 354, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 830, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 919, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/./src/charm.py", line 224, in reconcile_database_relations
    self.workload.enable(tls=self.tls.certificate_saved, unit_name=self.unit.name)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/src/workload.py", line 208, in enable
    self._bootstrap_router(tls=tls)
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/src/workload.py", line 175, in _bootstrap_router
    process.wait_output()
  File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/pebble.py", line 1252, in wait_output
    raise ExecError(self._command, exit_code, out_value, err_value)
ops.pebble.ExecError: non-zero exit code 1 executing ['mysqlrouter', '--bootstrap', 'relation-4:ZJfWC6Jq12uldwx8Px0HcBMz@mysql-2nd-k8s-primary.foo46.svc.cluster.local:3306', '--strict', '--user', 'mysql', '--conf-set-option', 'http_server.bind_address=127.0.0.1', '--conf-use-gr-notifications'], stdout='', stderr="Error: The given Router instance is already configured for a cluster named 'cluster_3e97a9b353fcfb511f0181d85ae3dd44'.\nIf you'd like to replace it, please use the --force configuration option.\n"
unit-mysql-router-k8s-0: 19:24:28 ERROR juju.worker.uniter.operation hook "backend-database-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1

Additional context

carlcsaposs-canonical commented 1 year ago
juju debug-log --replay ```shell controller-0: 19:20:28 INFO juju.worker.apicaller [673835] "controller-0" successfully connected to "localhost:17070" controller-0: 19:20:28 INFO juju.worker.logforwarder config change - log forwarding not enabled controller-0: 19:20:28 INFO juju.worker.logger logger worker started controller-0: 19:20:28 INFO juju.worker.pruner.action status history config: max age: 336h0m0s, max collection size 5120M for foo46 (67383503-8c9a-481e-883e-5b45733c431e) controller-0: 19:20:28 INFO juju.worker.pruner.statushistory status history config: max age: 336h0m0s, max collection size 5120M for foo46 (67383503-8c9a-481e-883e-5b45733c431e) model-67383503-8c9a-481e-883e-5b45733c431e: 19:20:31 INFO juju.worker.caasupgrader abort check blocked until version event received model-67383503-8c9a-481e-883e-5b45733c431e: 19:20:31 INFO juju.worker.caasupgrader unblocking abort check model-67383503-8c9a-481e-883e-5b45733c431e: 19:20:32 INFO juju.worker.muxhttpserver starting http server on [::]:17071 model-67383503-8c9a-481e-883e-5b45733c431e: 19:20:32 INFO juju.worker.caasadmission ensuring model k8s webhook configurations controller-0: 19:20:44 INFO juju.worker.caasapplicationprovisioner.runner start "mysql-k8s" unit-mysql-k8s-0: 19:20:56 INFO juju.cmd running containerAgent [2.9.43 6a8b1c207bbe3d563121dfe129df90748a2a3499 gc go1.20.4] unit-mysql-k8s-0: 19:20:56 INFO juju.cmd.containeragent.unit start "unit" unit-mysql-k8s-0: 19:20:56 INFO juju.worker.upgradesteps upgrade steps for 2.9.43 have already been run. unit-mysql-k8s-0: 19:20:56 INFO juju.worker.probehttpserver starting http server on [::]:65301 controller-0: 19:20:56 INFO juju.worker.caasapplicationprovisioner.runner start "mysql-router-k8s" unit-mysql-k8s-0: 19:20:56 INFO juju.api cannot resolve "controller-service.controller-microk8s.svc.cluster.local": lookup controller-service.controller-microk8s.svc.cluster.local: operation was canceled unit-mysql-k8s-0: 19:20:56 INFO juju.api connection established to "wss://10.152.183.71:17070/model/67383503-8c9a-481e-883e-5b45733c431e/api" unit-mysql-k8s-0: 19:20:56 INFO juju.worker.apicaller [673835] "unit-mysql-k8s-0" successfully connected to "10.152.183.71:17070" unit-mysql-k8s-0: 19:20:56 INFO juju.worker.migrationminion migration phase is now: NONE unit-mysql-k8s-0: 19:20:56 INFO juju.worker.logger logger worker started unit-mysql-k8s-0: 19:20:56 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: "" unit-mysql-k8s-0: 19:20:56 INFO juju.worker.caasupgrader abort check blocked until version event received unit-mysql-k8s-0: 19:20:56 INFO juju.worker.caasupgrader unblocking abort check unit-mysql-k8s-0: 19:20:56 INFO juju.worker.leadership mysql-k8s/0 promoted to leadership of mysql-k8s unit-mysql-k8s-0: 19:20:56 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-mysql-k8s-0 unit-mysql-k8s-0: 19:20:56 INFO juju.worker.uniter unit "mysql-k8s/0" started unit-mysql-k8s-0: 19:20:56 INFO juju.worker.uniter resuming charm install unit-mysql-k8s-0: 19:20:56 INFO juju.worker.uniter.charm downloading ch:amd64/jammy/mysql-k8s-80 from API server unit-mysql-k8s-0: 19:20:56 INFO juju.downloader downloading from ch:amd64/jammy/mysql-k8s-80 unit-mysql-k8s-0: 19:20:56 INFO juju.downloader download complete ("ch:amd64/jammy/mysql-k8s-80") unit-mysql-k8s-0: 19:20:56 INFO juju.downloader download verified ("ch:amd64/jammy/mysql-k8s-80") controller-0: 19:21:06 INFO juju.worker.caasapplicationprovisioner.runner stopped "mysql-router-k8s", err: Operation cannot be fulfilled on pods "mysql-router-k8s-0": the object has been modified; please apply your changes to the latest version and try again controller-0: 19:21:06 ERROR juju.worker.caasapplicationprovisioner.runner exited "mysql-router-k8s": Operation cannot be fulfilled on pods "mysql-router-k8s-0": the object has been modified; please apply your changes to the latest version and try again controller-0: 19:21:06 INFO juju.worker.caasapplicationprovisioner.runner restarting "mysql-router-k8s" in 3s unit-mysql-router-k8s-0: 19:21:06 INFO juju.cmd running containerAgent [2.9.43 6a8b1c207bbe3d563121dfe129df90748a2a3499 gc go1.20.4] unit-mysql-router-k8s-0: 19:21:06 INFO juju.cmd.containeragent.unit start "unit" unit-mysql-router-k8s-0: 19:21:06 INFO juju.worker.upgradesteps upgrade steps for 2.9.43 have already been run. unit-mysql-router-k8s-0: 19:21:06 INFO juju.worker.probehttpserver starting http server on [::]:65301 unit-mysql-router-k8s-0: 19:21:06 INFO juju.api cannot resolve "controller-service.controller-microk8s.svc.cluster.local": lookup controller-service.controller-microk8s.svc.cluster.local: operation was canceled unit-mysql-router-k8s-0: 19:21:06 INFO juju.api connection established to "wss://10.152.183.71:17070/model/67383503-8c9a-481e-883e-5b45733c431e/api" unit-mysql-router-k8s-0: 19:21:06 INFO juju.worker.apicaller [673835] "unit-mysql-router-k8s-0" successfully connected to "10.152.183.71:17070" unit-mysql-router-k8s-0: 19:21:06 INFO juju.worker.migrationminion migration phase is now: NONE unit-mysql-router-k8s-0: 19:21:06 INFO juju.worker.logger logger worker started unit-mysql-router-k8s-0: 19:21:06 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: "" unit-mysql-router-k8s-0: 19:21:06 INFO juju.worker.caasupgrader abort check blocked until version event received unit-mysql-router-k8s-0: 19:21:06 INFO juju.worker.caasupgrader unblocking abort check unit-mysql-router-k8s-0: 19:21:06 INFO juju.worker.leadership mysql-router-k8s/0 promoted to leadership of mysql-router-k8s unit-mysql-router-k8s-0: 19:21:06 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-mysql-router-k8s-0 unit-mysql-router-k8s-0: 19:21:06 INFO juju.worker.uniter unit "mysql-router-k8s/0" started unit-mysql-router-k8s-0: 19:21:06 INFO juju.worker.uniter resuming charm install unit-mysql-router-k8s-0: 19:21:06 INFO juju.worker.uniter.charm downloading ch:amd64/jammy/mysql-router-k8s-31 from API server unit-mysql-router-k8s-0: 19:21:06 INFO juju.downloader downloading from ch:amd64/jammy/mysql-router-k8s-31 unit-mysql-router-k8s-0: 19:21:06 INFO juju.downloader download complete ("ch:amd64/jammy/mysql-router-k8s-31") unit-mysql-router-k8s-0: 19:21:07 INFO juju.downloader download verified ("ch:amd64/jammy/mysql-router-k8s-31") controller-0: 19:21:09 INFO juju.worker.caasapplicationprovisioner.runner start "mysql-router-k8s" unit-mysql-k8s-0: 19:21:13 INFO juju.worker.uniter hooks are retried true unit-mysql-k8s-0: 19:21:13 INFO juju.worker.uniter found queued "install" hook unit-mysql-k8s-0: 19:21:15 DEBUG unit.mysql-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 19:21:15 INFO unit.mysql-k8s/0.juju-log Running legacy hooks/install. unit-mysql-k8s-0: 19:21:15 DEBUG unit.mysql-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 19:21:15 DEBUG unit.mysql-k8s/0.juju-log Charm called itself via hooks/install. unit-mysql-k8s-0: 19:21:15 DEBUG unit.mysql-k8s/0.juju-log Legacy hooks/install exited with status 0. unit-mysql-k8s-0: 19:21:15 DEBUG unit.mysql-k8s/0.juju-log Using local storage: not a Kubernetes podspec charm unit-mysql-k8s-0: 19:21:16 DEBUG unit.mysql-k8s/0.juju-log Initializing SQLite local storage: /var/lib/juju/agents/unit-mysql-k8s-0/charm/.unit-state.db. unit-mysql-k8s-0: 19:21:16 DEBUG unit.mysql-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 19:21:16 DEBUG unit.mysql-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 19:21:16 DEBUG unit.mysql-k8s/0.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:16 DEBUG unit.mysql-k8s/0.juju-log Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:16 DEBUG unit.mysql-k8s/0.juju-log Emitting Juju event install. unit-mysql-k8s-0: 19:21:16 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 19:21:17 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 19:21:17 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 19:21:17 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 19:21:17 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:17 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:17 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Emitting Juju event database_peers_relation_created. unit-mysql-k8s-0: 19:21:17 INFO juju.worker.uniter.operation ran "database-peers-relation-created" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 19:21:18 DEBUG unit.mysql-k8s/0.juju-log database:2: Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 19:21:18 DEBUG unit.mysql-k8s/0.juju-log database:2: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 19:21:18 DEBUG unit.mysql-k8s/0.juju-log database:2: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 19:21:18 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:18 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:18 DEBUG unit.mysql-k8s/0.juju-log database:2: Emitting Juju event database_relation_created. unit-mysql-k8s-0: 19:21:18 INFO juju.worker.uniter.operation ran "database-relation-created" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 19:21:18 INFO juju.worker.uniter hooks are retried true unit-mysql-k8s-0: 19:21:18 INFO juju.worker.uniter found queued "leader-elected" hook unit-mysql-router-k8s-0: 19:21:18 INFO juju.worker.uniter found queued "install" hook unit-mysql-k8s-0: 19:21:19 DEBUG unit.mysql-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 19:21:19 DEBUG unit.mysql-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 19:21:19 DEBUG unit.mysql-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 19:21:19 DEBUG unit.mysql-k8s/0.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:19 DEBUG unit.mysql-k8s/0.juju-log Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:19 DEBUG unit.mysql-k8s/0.juju-log Emitting Juju event leader_elected. unit-mysql-k8s-0: 19:21:19 DEBUG unit.mysql-k8s/0.juju-log Setting root-password unit-mysql-k8s-0: 19:21:19 DEBUG unit.mysql-k8s/0.juju-log Setting server-config-password unit-mysql-k8s-0: 19:21:19 DEBUG unit.mysql-k8s/0.juju-log Setting cluster-admin-password unit-mysql-k8s-0: 19:21:19 DEBUG unit.mysql-k8s/0.juju-log Setting monitoring-password unit-mysql-router-k8s-0: 19:21:19 DEBUG unit.mysql-router-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:21:19 INFO unit.mysql-router-k8s/0.juju-log Running legacy hooks/install. unit-mysql-k8s-0: 19:21:20 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log Charm called itself via hooks/install. unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log Legacy hooks/install exited with status 0. unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log Using local storage: not a Kubernetes podspec charm unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log Initializing SQLite local storage: /var/lib/juju/agents/unit-mysql-router-k8s-0/charm/.unit-state.db. unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log Emitting Juju event install. unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log Patching k8s service name='mysql-router-k8s', ro_port=6447, rw_port=6446 unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log connection.connect_tcp.started host='10.152.183.1' port=443 local_address=None timeout=None unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log connection.connect_tcp.complete return_value= unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log connection.start_tls.started ssl_context= server_hostname='10.152.183.1' timeout=None unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log connection.start_tls.complete return_value= unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.send_request_headers.started request= unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.send_request_headers.complete unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.send_request_body.started request= unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.send_request_body.complete unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.receive_response_headers.started request= unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'0c4cf098-f9bc-4c2f-90d9-e6c28f2f2d6d'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:21:20 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-router-k8s-0: 19:21:20 INFO unit.mysql-router-k8s/0.juju-log HTTP Request: GET https://10.152.183.1/api/v1/namespaces/foo46/pods/mysql-router-k8s-0 "HTTP/1.1 200 OK" unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.receive_response_body.started request= unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.receive_response_body.complete unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.response_closed.started unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.response_closed.complete unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.send_request_headers.started request= unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.send_request_headers.complete unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.send_request_body.started request= unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.send_request_body.complete unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.receive_response_headers.started request= unit-mysql-k8s-0: 19:21:20 DEBUG unit.mysql-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'5a0b13c9-5f71-4104-8c64-7c0d9e045e43'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:21:20 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-router-k8s-0: 19:21:20 INFO unit.mysql-router-k8s/0.juju-log HTTP Request: PATCH https://10.152.183.1/api/v1/namespaces/foo46/services/mysql-router-k8s?fieldManager=mysql-router-k8s "HTTP/1.1 200 OK" unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.receive_response_body.started request= unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.receive_response_body.complete unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.response_closed.started unit-mysql-k8s-0: 19:21:20 DEBUG unit.mysql-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log http11.response_closed.complete unit-mysql-k8s-0: 19:21:20 DEBUG unit.mysql-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-router-k8s-0: 19:21:20 DEBUG unit.mysql-router-k8s/0.juju-log Patched k8s service name='mysql-router-k8s', ro_port=6447, rw_port=6446 unit-mysql-k8s-0: 19:21:20 DEBUG unit.mysql-k8s/0.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:20 DEBUG unit.mysql-k8s/0.juju-log Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:20 DEBUG unit.mysql-k8s/0.juju-log Emitting Juju event mysql_pebble_ready. unit-mysql-k8s-0: 19:21:20 DEBUG unit.mysql-k8s/0.juju-log Leader not ready yet, waiting... unit-mysql-k8s-0: 19:21:20 DEBUG unit.mysql-k8s/0.juju-log Deferring . unit-mysql-router-k8s-0: 19:21:21 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 19:21:21 INFO juju.worker.uniter.operation ran "mysql-pebble-ready" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 19:21:21 DEBUG unit.mysql-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:21:21 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 19:21:21 DEBUG unit.mysql-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 19:21:21 DEBUG unit.mysql-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 19:21:21 DEBUG unit.mysql-k8s/0.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-router-k8s-0: 19:21:21 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Emitting Juju event backend_database_relation_created. unit-mysql-k8s-0: 19:21:21 DEBUG unit.mysql-k8s/0.juju-log Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:21 DEBUG unit.mysql-k8s/0.juju-log Re-emitting deferred event . unit-mysql-k8s-0: 19:21:21 DEBUG unit.mysql-k8s/0.juju-log Leader not ready yet, waiting... unit-mysql-k8s-0: 19:21:21 DEBUG unit.mysql-k8s/0.juju-log Deferring . unit-mysql-k8s-0: 19:21:21 DEBUG unit.mysql-k8s/0.juju-log Emitting Juju event database_storage_attached. unit-mysql-router-k8s-0: 19:21:22 INFO juju.worker.uniter.operation ran "backend-database-relation-created" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 19:21:22 INFO juju.worker.uniter.operation ran "database-storage-attached" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 19:21:22 DEBUG unit.mysql-router-k8s/0.juju-log mysql-router-peers:1: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:21:22 DEBUG unit.mysql-router-k8s/0.juju-log mysql-router-peers:1: Emitting Juju event mysql_router_peers_relation_created. unit-mysql-k8s-0: 19:21:22 DEBUG unit.mysql-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 19:21:22 DEBUG unit.mysql-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 19:21:22 DEBUG unit.mysql-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 19:21:22 DEBUG unit.mysql-k8s/0.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:22 DEBUG unit.mysql-k8s/0.juju-log Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:22 DEBUG unit.mysql-k8s/0.juju-log Re-emitting deferred event . unit-mysql-k8s-0: 19:21:22 DEBUG unit.mysql-k8s/0.juju-log Leader not ready yet, waiting... unit-mysql-k8s-0: 19:21:22 DEBUG unit.mysql-k8s/0.juju-log Deferring . unit-mysql-k8s-0: 19:21:22 DEBUG unit.mysql-k8s/0.juju-log Emitting Juju event config_changed. unit-mysql-router-k8s-0: 19:21:22 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-created" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 19:21:23 INFO juju.worker.uniter found queued "leader-elected" hook unit-mysql-k8s-0: 19:21:23 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 19:21:23 INFO juju.worker.uniter found queued "start" hook unit-mysql-router-k8s-0: 19:21:23 DEBUG unit.mysql-router-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:21:23 DEBUG unit.mysql-router-k8s/0.juju-log Emitting Juju event leader_elected. unit-mysql-router-k8s-0: 19:21:23 DEBUG unit.mysql-router-k8s/0.juju-log Set status to BlockedStatus('Missing relation: database') unit-mysql-k8s-0: 19:21:23 DEBUG unit.mysql-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 19:21:23 INFO unit.mysql-k8s/0.juju-log Running legacy hooks/start. unit-mysql-router-k8s-0: 19:21:24 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log Charm called itself via hooks/start. unit-mysql-router-k8s-0: 19:21:24 DEBUG unit.mysql-router-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:21:24 DEBUG unit.mysql-router-k8s/0.juju-log Emitting Juju event mysql_router_pebble_ready. unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log Legacy hooks/start exited with status 0. unit-mysql-router-k8s-0: 19:21:24 DEBUG unit.mysql-router-k8s/0.juju-log State of reconcile self.unit.is_leader()=True, isinstance(self.workload, workload.AuthenticatedWorkload)=False, self.database_requires.relation and self.database_requires.relation.is_breaking(event)=None, self.workload.container_ready=True, isinstance(event, ops.UpgradeCharmEvent)=False unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log Re-emitting deferred event . unit-mysql-router-k8s-0: 19:21:24 DEBUG unit.mysql-router-k8s/0.juju-log Set status to BlockedStatus('Missing relation: database') unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log connection.connect_tcp.started host='10.152.183.1' port=443 local_address=None timeout=None unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log connection.connect_tcp.complete return_value= unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log connection.start_tls.started ssl_context= server_hostname='10.152.183.1' timeout=None unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log connection.start_tls.complete return_value= unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log http11.send_request_headers.started request= unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log http11.send_request_headers.complete unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log http11.send_request_body.started request= unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log http11.send_request_body.complete unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log http11.receive_response_headers.started request= unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'7b4fb091-da80-4b2a-b993-997b57fee4d8'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:21:24 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-k8s-0: 19:21:24 INFO unit.mysql-k8s/0.juju-log HTTP Request: GET https://10.152.183.1/api/v1/namespaces/foo46/pods/mysql-k8s-0 "HTTP/1.1 200 OK" unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log http11.receive_response_body.started request= unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log http11.receive_response_body.complete unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log http11.response_closed.started unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log http11.response_closed.complete unit-mysql-k8s-0: 19:21:24 INFO unit.mysql-k8s/0.juju-log Retrieving the total memory of the server unit-mysql-k8s-0: 19:21:24 DEBUG unit.mysql-k8s/0.juju-log Initializing instance unit-mysql-router-k8s-0: 19:21:24 INFO juju.worker.uniter.operation ran "mysql-router-pebble-ready" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 19:21:25 DEBUG unit.mysql-router-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:21:25 DEBUG unit.mysql-router-k8s/0.juju-log Emitting Juju event config_changed. unit-mysql-router-k8s-0: 19:21:25 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 19:21:25 INFO juju.worker.uniter found queued "start" hook unit-mysql-router-k8s-0: 19:21:26 DEBUG unit.mysql-router-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:21:26 INFO unit.mysql-router-k8s/0.juju-log Running legacy hooks/start. unit-mysql-router-k8s-0: 19:21:26 DEBUG unit.mysql-router-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:21:26 DEBUG unit.mysql-router-k8s/0.juju-log Charm called itself via hooks/start. unit-mysql-router-k8s-0: 19:21:26 DEBUG unit.mysql-router-k8s/0.juju-log Legacy hooks/start exited with status 0. unit-mysql-router-k8s-0: 19:21:26 DEBUG unit.mysql-router-k8s/0.juju-log Emitting Juju event start. unit-mysql-router-k8s-0: 19:21:26 DEBUG unit.mysql-router-k8s/0.juju-log State of reconcile self.unit.is_leader()=True, isinstance(self.workload, workload.AuthenticatedWorkload)=False, self.database_requires.relation and self.database_requires.relation.is_breaking(event)=None, self.workload.container_ready=True, isinstance(event, ops.UpgradeCharmEvent)=False unit-mysql-router-k8s-0: 19:21:26 DEBUG unit.mysql-router-k8s/0.juju-log Set status to BlockedStatus('Missing relation: database') unit-mysql-router-k8s-0: 19:21:27 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 19:21:27 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:21:27 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Emitting Juju event backend_database_relation_joined. unit-mysql-router-k8s-0: 19:21:28 INFO juju.worker.uniter.operation ran "backend-database-relation-joined" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 19:21:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:21:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Emitting Juju event backend_database_relation_changed. unit-mysql-router-k8s-0: 19:21:29 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 19:21:33 DEBUG unit.mysql-k8s/0.juju-log Adding pebble layer unit-mysql-k8s-0: 19:21:33 DEBUG unit.mysql-k8s/0.juju-log Safe stopping mysqld safe unit-mysql-k8s-0: 19:21:34 DEBUG unit.mysql-k8s/0.juju-log Waiting for instance to be ready unit-mysql-k8s-0: 19:21:39 INFO unit.mysql-k8s/0.juju-log Configuring instance unit-mysql-k8s-0: 19:21:39 DEBUG unit.mysql-k8s/0.juju-log Configuring users unit-mysql-k8s-0: 19:21:39 DEBUG unit.mysql-k8s/0.juju-log Configuring instance for InnoDB on mysql-k8s-0.mysql-k8s-endpoints unit-mysql-k8s-0: 19:21:46 DEBUG unit.mysql-k8s/0.juju-log Getting InnoDB version unit-mysql-k8s-0: 19:21:46 INFO unit.mysql-k8s/0.juju-log Creating cluster on the leader unit unit-mysql-k8s-0: 19:21:46 DEBUG unit.mysql-k8s/0.juju-log Creating a MySQL InnoDB cluster on mysql-k8s-0.mysql-k8s-endpoints unit-mysql-k8s-0: 19:21:49 DEBUG unit.mysql-k8s/0.juju-log Emitting Juju event start. unit-mysql-k8s-0: 19:21:49 DEBUG unit.mysql-k8s/0.juju-log Alert rules path does not exist: src/prometheus_alert_rules unit-mysql-k8s-0: 19:21:50 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 19:21:50 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 19:21:50 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 19:21:50 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 19:21:50 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:50 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:50 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Emitting Juju event database_peers_relation_changed. unit-mysql-k8s-0: 19:21:50 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Updating pod labels unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: connection.connect_tcp.started host='10.152.183.1' port=443 local_address=None timeout=None unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: connection.connect_tcp.complete return_value= unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: connection.start_tls.started ssl_context= server_hostname='10.152.183.1' timeout=None unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: connection.start_tls.complete return_value= unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_headers.started request= unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_headers.complete unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_body.started request= unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_body.complete unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_headers.started request= unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'14901e05-791a-4e48-b5cb-d247919a9f9c'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:21:51 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-k8s-0: 19:21:51 INFO unit.mysql-k8s/0.juju-log database-peers:0: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/foo46/pods/mysql-k8s-0 "HTTP/1.1 200 OK" unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_body.started request= unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_body.complete unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.response_closed.started unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.response_closed.complete unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_headers.started request= unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_headers.complete unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_body.started request= unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_body.complete unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_headers.started request= unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'54a23da6-d37c-44ba-9cc0-5e53a543444d'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:21:51 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-k8s-0: 19:21:51 INFO unit.mysql-k8s/0.juju-log database-peers:0: HTTP Request: PATCH https://10.152.183.1/api/v1/namespaces/foo46/pods/mysql-k8s-0 "HTTP/1.1 200 OK" unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_body.started request= unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_body.complete unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.response_closed.started unit-mysql-k8s-0: 19:21:51 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.response_closed.complete unit-mysql-k8s-0: 19:21:51 INFO unit.mysql-k8s/0.juju-log database-peers:0: Kubernetes pod label primary created unit-mysql-k8s-0: 19:21:51 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Emitting Juju event database_peers_relation_changed. unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Updating pod labels unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: connection.connect_tcp.started host='10.152.183.1' port=443 local_address=None timeout=None unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: connection.connect_tcp.complete return_value= unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: connection.start_tls.started ssl_context= server_hostname='10.152.183.1' timeout=None unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: connection.start_tls.complete return_value= unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_headers.started request= unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_headers.complete unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_body.started request= unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_body.complete unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_headers.started request= unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'6b54fb3e-c8f7-4af9-b998-9c1027aed885'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:21:52 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-k8s-0: 19:21:52 INFO unit.mysql-k8s/0.juju-log database-peers:0: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/foo46/pods/mysql-k8s-0 "HTTP/1.1 200 OK" unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_body.started request= unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_body.complete unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.response_closed.started unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.response_closed.complete unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_headers.started request= unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_headers.complete unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_body.started request= unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_body.complete unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_headers.started request= unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'4a9cdbc4-f37a-4e5a-ab85-2f908ea033d9'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:21:52 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-k8s-0: 19:21:52 INFO unit.mysql-k8s/0.juju-log database-peers:0: HTTP Request: PATCH https://10.152.183.1/api/v1/namespaces/foo46/pods/mysql-k8s-0 "HTTP/1.1 200 OK" unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_body.started request= unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_body.complete unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.response_closed.started unit-mysql-k8s-0: 19:21:52 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.response_closed.complete unit-mysql-k8s-0: 19:21:52 INFO unit.mysql-k8s/0.juju-log database-peers:0: Kubernetes pod label primary created unit-mysql-k8s-0: 19:21:52 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 19:21:53 DEBUG unit.mysql-k8s/0.juju-log database:2: Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 19:21:53 DEBUG unit.mysql-k8s/0.juju-log database:2: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 19:21:53 DEBUG unit.mysql-k8s/0.juju-log database:2: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 19:21:53 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:53 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 19:21:53 DEBUG unit.mysql-k8s/0.juju-log database:2: Emitting Juju event database_relation_changed. unit-mysql-k8s-0: 19:21:53 DEBUG unit.mysql-k8s/0.juju-log database:2: Emitting custom event . unit-mysql-k8s-0: 19:21:53 DEBUG unit.mysql-k8s/0.juju-log database:2: Getting InnoDB version unit-mysql-k8s-0: 19:21:53 DEBUG unit.mysql-k8s/0.juju-log database:2: Updating pod labels unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: connection.connect_tcp.started host='10.152.183.1' port=443 local_address=None timeout=None unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: connection.connect_tcp.complete return_value= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: connection.start_tls.started ssl_context= server_hostname='10.152.183.1' timeout=None unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: connection.start_tls.complete return_value= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'b21471ec-7a98-4c65-b831-1f5db29d2d20'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:21:54 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-k8s-0: 19:21:54 INFO unit.mysql-k8s/0.juju-log database:2: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/foo46/pods/mysql-k8s-0 "HTTP/1.1 200 OK" unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.started unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'ebb898f1-f3a6-494b-a3f9-0866171190ac'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:21:54 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-k8s-0: 19:21:54 INFO unit.mysql-k8s/0.juju-log database:2: HTTP Request: PATCH https://10.152.183.1/api/v1/namespaces/foo46/pods/mysql-k8s-0 "HTTP/1.1 200 OK" unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.started unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.complete unit-mysql-k8s-0: 19:21:54 INFO unit.mysql-k8s/0.juju-log database:2: Kubernetes pod label primary created unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'852b69f2-0779-4555-af81-56180e1cbbb2'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:21:54 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-k8s-0: 19:21:54 INFO unit.mysql-k8s/0.juju-log database:2: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/foo46/pods/mysql-k8s-0 "HTTP/1.1 200 OK" unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.started unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 201, b'Created', [(b'Audit-Id', b'58dec4f4-f42a-4b45-861c-771d0591792f'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:21:54 GMT'), (b'Content-Length', b'1270')]) unit-mysql-k8s-0: 19:21:54 INFO unit.mysql-k8s/0.juju-log database:2: HTTP Request: POST https://10.152.183.1/api/v1/namespaces/foo46/services "HTTP/1.1 201 Created" unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.started unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.complete unit-mysql-k8s-0: 19:21:54 INFO unit.mysql-k8s/0.juju-log database:2: Kubernetes service mysql-k8s-primary created unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'ca15477e-59f9-4f1b-a35c-c4f0244de2fd'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:21:54 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-k8s-0: 19:21:54 INFO unit.mysql-k8s/0.juju-log database:2: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/foo46/pods/mysql-k8s-0 "HTTP/1.1 200 OK" unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.started unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 201, b'Created', [(b'Audit-Id', b'1df2885f-2142-4039-82fc-ddb1de4cf327'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:21:54 GMT'), (b'Content-Length', b'1270')]) unit-mysql-k8s-0: 19:21:54 INFO unit.mysql-k8s/0.juju-log database:2: HTTP Request: POST https://10.152.183.1/api/v1/namespaces/foo46/services "HTTP/1.1 201 Created" unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.started request= unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.complete unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.started unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.complete unit-mysql-k8s-0: 19:21:54 INFO unit.mysql-k8s/0.juju-log database:2: Kubernetes service mysql-k8s-replicas created unit-mysql-k8s-0: 19:21:54 DEBUG unit.mysql-k8s/0.juju-log database:2: Kubernetes service endpoint not ready yet unit-mysql-k8s-0: 19:21:55 DEBUG unit.mysql-k8s/0.juju-log database:2: Kubernetes service endpoint not ready yet unit-mysql-k8s-0: 19:21:56 DEBUG unit.mysql-k8s/0.juju-log database:2: Kubernetes service endpoint not ready yet unit-mysql-k8s-0: 19:21:57 DEBUG unit.mysql-k8s/0.juju-log database:2: Kubernetes service endpoint not ready yet unit-mysql-k8s-0: 19:21:58 DEBUG unit.mysql-k8s/0.juju-log database:2: Kubernetes service endpoint not ready yet unit-mysql-k8s-0: 19:21:59 DEBUG unit.mysql-k8s/0.juju-log database:2: Kubernetes service endpoint not ready yet unit-mysql-k8s-0: 19:22:00 DEBUG unit.mysql-k8s/0.juju-log database:2: Kubernetes service endpoint ready unit-mysql-k8s-0: 19:22:00 DEBUG unit.mysql-k8s/0.juju-log database:2: Getting cluster primary member's address from mysql-k8s-0.mysql-k8s-endpoints unit-mysql-k8s-0: 19:22:01 DEBUG unit.mysql-k8s/0.juju-log database:2: Getting cluster primary member's address from mysql-k8s-0.mysql-k8s-endpoints unit-mysql-k8s-0: 19:22:01 INFO unit.mysql-k8s/0.juju-log database:2: Created user for app mysql-router-k8s unit-mysql-k8s-0: 19:22:02 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 19:22:02 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:22:02 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Emitting Juju event backend_database_relation_changed. unit-mysql-k8s-0: 19:22:02 DEBUG unit.mysql-k8s/0.juju-log database:2: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:22:02 INFO unit.mysql-router-k8s/0.juju-log backend-database:2: database created at 2023-05-18 19:22:02.891435 unit-mysql-router-k8s-0: 19:22:02 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Emitting custom event . unit-mysql-router-k8s-0: 19:22:02 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: State of reconcile self.unit.is_leader()=True, isinstance(self.workload, workload.AuthenticatedWorkload)=True, self.database_requires.relation and self.database_requires.relation.is_breaking(event)=False, self.workload.container_ready=True, isinstance(event, ops.UpgradeCharmEvent)=False unit-mysql-k8s-0: 19:22:02 DEBUG unit.mysql-k8s/0.juju-log database:2: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-router-k8s-0: 19:22:02 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Reconciling users event=, router_endpoint='mysql-router-k8s.foo46.svc.cluster.local' unit-mysql-k8s-0: 19:22:02 DEBUG unit.mysql-k8s/0.juju-log database:2: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-router-k8s-0: 19:22:02 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: State of reconcile users requested_users=[], created_users=[] unit-mysql-router-k8s-0: 19:22:02 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Reconciled users event=, router_endpoint='mysql-router-k8s.foo46.svc.cluster.local' unit-mysql-k8s-0: 19:22:02 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 19:22:02 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-router-k8s-0: 19:22:02 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Enabling MySQL Router service unit-mysql-router-k8s-0: 19:22:02 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Bootstrapping router tls=False, self._database_requires_relation.host='mysql-k8s-primary.foo46.svc.cluster.local', self._database_requires_relation.port='3306' unit-mysql-k8s-0: 19:22:02 DEBUG unit.mysql-k8s/0.juju-log database:2: Emitting Juju event database_relation_joined. unit-mysql-k8s-0: 19:22:03 INFO juju.worker.uniter.operation ran "database-relation-joined" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 19:22:03 DEBUG unit.mysql-k8s/0.juju-log database:2: Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 19:22:03 DEBUG unit.mysql-k8s/0.juju-log database:2: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 19:22:03 DEBUG unit.mysql-k8s/0.juju-log database:2: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 19:22:03 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 19:22:03 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 19:22:03 DEBUG unit.mysql-k8s/0.juju-log database:2: Emitting Juju event database_relation_changed. unit-mysql-k8s-0: 19:22:04 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 19:22:04 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Bootstrapped router tls=False, self._database_requires_relation.host='mysql-k8s-primary.foo46.svc.cluster.local', self._database_requires_relation.port='3306' unit-mysql-router-k8s-0: 19:22:04 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Adding attributes='{"created_by_user": "relation-2", "router_id": "mysql-router-k8s-0.mysql-router-k8s-endpoints.foo46.svc.cluster.local::system", "created_by_juju_unit": "mysql-router-k8s/0"}' to username='mysql_router1_8lsquzd4uft8' unit-mysql-router-k8s-0: 19:22:05 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Added attributes='{"created_by_user": "relation-2", "router_id": "mysql-router-k8s-0.mysql-router-k8s-endpoints.foo46.svc.cluster.local::system", "created_by_juju_unit": "mysql-router-k8s/0"}' to username='mysql_router1_8lsquzd4uft8' unit-mysql-router-k8s-0: 19:22:05 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Enabled MySQL Router service unit-mysql-router-k8s-0: 19:22:05 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Waiting until MySQL Router is ready unit-mysql-router-k8s-0: 19:22:05 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: MySQL Router is ready unit-mysql-router-k8s-0: 19:22:05 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Set status to BlockedStatus('Missing relation: database') unit-mysql-router-k8s-0: 19:22:05 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 19:22:06 DEBUG unit.mysql-k8s/0.juju-log database:2: Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 19:22:06 DEBUG unit.mysql-k8s/0.juju-log database:2: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 19:22:06 DEBUG unit.mysql-k8s/0.juju-log database:2: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 19:22:06 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 19:22:06 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 19:22:06 DEBUG unit.mysql-k8s/0.juju-log database:2: Emitting Juju event database_relation_changed. unit-mysql-k8s-0: 19:22:06 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 19:23:12 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:23:12 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Emitting Juju event backend_database_relation_departed. unit-mysql-k8s-0: 19:23:12 DEBUG unit.mysql-k8s/0.juju-log database:2: Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 19:23:12 DEBUG unit.mysql-k8s/0.juju-log database:2: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 19:23:12 DEBUG unit.mysql-k8s/0.juju-log database:2: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 19:23:12 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 19:23:12 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 19:23:12 DEBUG unit.mysql-k8s/0.juju-log database:2: Emitting Juju event database_relation_departed. unit-mysql-router-k8s-0: 19:23:12 INFO juju.worker.uniter.operation ran "backend-database-relation-departed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 19:23:12 DEBUG unit.mysql-k8s/0.juju-log database:2: Getting cluster primary member's address from mysql-k8s-0.mysql-k8s-endpoints unit-mysql-router-k8s-0: 19:23:12 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:23:12 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Emitting Juju event backend_database_relation_broken. unit-mysql-router-k8s-0: 19:23:12 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: State of reconcile self.unit.is_leader()=True, isinstance(self.workload, workload.AuthenticatedWorkload)=True, self.database_requires.relation and self.database_requires.relation.is_breaking(event)=True, self.workload.container_ready=True, isinstance(event, ops.UpgradeCharmEvent)=False unit-mysql-router-k8s-0: 19:23:12 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Reconciling users event=, router_endpoint='mysql-router-k8s.foo46.svc.cluster.local' unit-mysql-router-k8s-0: 19:23:13 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: State of reconcile users requested_users=[], created_users=[] unit-mysql-router-k8s-0: 19:23:13 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Reconciled users event=, router_endpoint='mysql-router-k8s.foo46.svc.cluster.local' unit-mysql-router-k8s-0: 19:23:13 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Disabling MySQL Router service unit-mysql-k8s-0: 19:23:13 INFO unit.mysql-k8s/0.juju-log database:2: Deleted router user mysql_router1_8lsquzd4uft8 unit-mysql-k8s-0: 19:23:13 DEBUG unit.mysql-k8s/0.juju-log database:2: Getting cluster primary member's address from mysql-k8s-0.mysql-k8s-endpoints unit-mysql-k8s-0: 19:23:13 INFO unit.mysql-k8s/0.juju-log database:2: Removed router from metadata mysql-router-k8s-0.mysql-router-k8s-endpoints.foo46.svc.cluster.local::system unit-mysql-router-k8s-0: 19:23:14 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Disabled MySQL Router service unit-mysql-router-k8s-0: 19:23:14 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Set status to BlockedStatus('Missing relations: backend-database, database') unit-mysql-k8s-0: 19:23:14 INFO juju.worker.uniter.operation ran "database-relation-departed" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 19:23:14 INFO juju.worker.uniter.operation ran "backend-database-relation-broken" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: Emitting Juju event database_relation_broken. unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: connection.connect_tcp.started host='10.152.183.1' port=443 local_address=None timeout=None unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: connection.connect_tcp.complete return_value= unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: connection.start_tls.started ssl_context= server_hostname='10.152.183.1' timeout=None unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: connection.start_tls.complete return_value= unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.started request= unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.complete unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.started request= unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.complete unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.started request= unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'f21d1561-24c2-4964-9ed0-b00397c192a9'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:23:14 GMT'), (b'Content-Length', b'1270')]) unit-mysql-k8s-0: 19:23:14 INFO unit.mysql-k8s/0.juju-log database:2: HTTP Request: DELETE https://10.152.183.1/api/v1/namespaces/foo46/services/mysql-k8s-primary "HTTP/1.1 200 OK" unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.started request= unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.complete unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.started unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.complete unit-mysql-k8s-0: 19:23:14 INFO unit.mysql-k8s/0.juju-log database:2: Kubernetes service mysql-k8s-primary deleted unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.started request= unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.complete unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.started request= unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.complete unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.started request= unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'c34c6004-a1e9-47ed-956b-77eddfebb661'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:23:14 GMT'), (b'Content-Length', b'1270')]) unit-mysql-k8s-0: 19:23:14 INFO unit.mysql-k8s/0.juju-log database:2: HTTP Request: DELETE https://10.152.183.1/api/v1/namespaces/foo46/services/mysql-k8s-replicas "HTTP/1.1 200 OK" unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.started request= unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.complete unit-mysql-k8s-0: 19:23:14 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.started unit-mysql-k8s-0: 19:23:15 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.complete unit-mysql-k8s-0: 19:23:15 INFO unit.mysql-k8s/0.juju-log database:2: Kubernetes service mysql-k8s-replicas deleted unit-mysql-k8s-0: 19:23:15 DEBUG unit.mysql-k8s/0.juju-log database:2: Getting cluster primary member's address from mysql-k8s-0.mysql-k8s-endpoints unit-mysql-k8s-0: 19:23:15 ERROR unit.mysql-k8s/0.juju-log database:2: Failed to delete users for relation 2 Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/src/mysql_k8s_helpers.py", line 658, in _run_mysqlsh_script stdout, _ = process.wait_output() File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/pebble.py", line 1252, in wait_output raise ExecError(self._command, exit_code, out_value, err_value) ops.pebble.ExecError: non-zero exit code 1 executing ['/usr/bin/mysqlsh', '--no-wizard', '--python', '--verbose=1', '-f', '/tmp/script.py', ';', 'rm', '/tmp/script.py'], stdout='', stderr='Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nverbose: 2023-05-18T19:23:15Z: Loading startup files...\nverbose: 2023-05-18T19:23:15Z: Loading plugins...\nverbose: 2023-05-18T19:23:15Z: Connecting to MySQL at: serverconfig@mysql-k8s-0.mysql-k8s-endpoints.foo46.svc.cluster.local:3306\nverbose: 2023-05-18T19:23:15Z: Shell.connect: tid=92: CONNECTED: mysql-k8s-0.mysql-k8s-endpoints.foo46.svc.cluster.local:3306\nverbose: 2023-05-18T19:23:15Z: Now tracking \'sql_mode\' system variable.\nverbose: 2023-05-18T19:23:15Z: ClassicSession.run_sql: tid=92: MySQL Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near \'NULL\' at line 1, SQL: PREPARE stmt FROM @sql\nTraceback (most recent call last):\n File "", line 4, in \nmysqlsh.DBError: MySQL Error (1064): ClassicSession.run_sql: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right synta' [truncated] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/lib/charms/mysql/v0/mysql.py", line 626, in delete_users_for_relation self._run_mysqlsh_script("\n".join(drop_users_command)) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/src/mysql_k8s_helpers.py", line 661, in _run_mysqlsh_script raise MySQLClientError(e.stderr) charms.mysql.v0.mysql.MySQLClientError: Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory verbose: 2023-05-18T19:23:15Z: Loading startup files... verbose: 2023-05-18T19:23:15Z: Loading plugins... verbose: 2023-05-18T19:23:15Z: Connecting to MySQL at: serverconfig@mysql-k8s-0.mysql-k8s-endpoints.foo46.svc.cluster.local:3306 verbose: 2023-05-18T19:23:15Z: Shell.connect: tid=92: CONNECTED: mysql-k8s-0.mysql-k8s-endpoints.foo46.svc.cluster.local:3306 verbose: 2023-05-18T19:23:15Z: Now tracking 'sql_mode' system variable. verbose: 2023-05-18T19:23:15Z: ClassicSession.run_sql: tid=92: MySQL Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'NULL' at line 1, SQL: PREPARE stmt FROM @sql Traceback (most recent call last): File "", line 4, in mysqlsh.DBError: MySQL Error (1064): ClassicSession.run_sql: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'NULL' at line 1 unit-mysql-k8s-0: 19:23:15 ERROR unit.mysql-k8s/0.juju-log database:2: Failed to delete user(s) for relation 2 unit-mysql-k8s-0: 19:23:15 INFO juju.worker.uniter.operation ran "database-relation-broken" hook (via hook dispatching script: dispatch) controller-0: 19:23:22 INFO juju.worker.caasapplicationprovisioner.runner start "mysql-2nd-k8s" unit-mysql-router-k8s-0: 19:23:30 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:23:30 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Emitting Juju event backend_database_relation_created. unit-mysql-router-k8s-0: 19:23:30 INFO juju.worker.uniter.operation ran "backend-database-relation-created" hook (via hook dispatching script: dispatch) unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.cmd running containerAgent [2.9.43 6a8b1c207bbe3d563121dfe129df90748a2a3499 gc go1.20.4] unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.cmd.containeragent.unit start "unit" unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.worker.upgradesteps upgrade steps for 2.9.43 have already been run. unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.worker.probehttpserver starting http server on [::]:65301 unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.api cannot resolve "controller-service.controller-microk8s.svc.cluster.local": lookup controller-service.controller-microk8s.svc.cluster.local: operation was canceled unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.api connection established to "wss://10.152.183.71:17070/model/67383503-8c9a-481e-883e-5b45733c431e/api" unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.worker.apicaller [673835] "unit-mysql-2nd-k8s-0" successfully connected to "10.152.183.71:17070" unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.api connection established to "wss://controller-service.controller-microk8s.svc.cluster.local:17070/model/67383503-8c9a-481e-883e-5b45733c431e/api" unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.worker.apicaller [673835] "unit-mysql-2nd-k8s-0" successfully connected to "controller-service.controller-microk8s.svc.cluster.local:17070" unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.worker.migrationminion migration phase is now: NONE unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.worker.logger logger worker started unit-mysql-2nd-k8s-0: 19:23:33 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: "" unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.worker.caasupgrader abort check blocked until version event received unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.worker.caasupgrader unblocking abort check unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.worker.leadership mysql-2nd-k8s/0 promoted to leadership of mysql-2nd-k8s unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-mysql-2nd-k8s-0 unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.worker.uniter unit "mysql-2nd-k8s/0" started unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.worker.uniter resuming charm install unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.worker.uniter.charm downloading ch:amd64/jammy/mysql-k8s-80 from API server unit-mysql-2nd-k8s-0: 19:23:33 INFO juju.downloader downloading from ch:amd64/jammy/mysql-k8s-80 unit-mysql-2nd-k8s-0: 19:23:34 INFO juju.downloader download complete ("ch:amd64/jammy/mysql-k8s-80") unit-mysql-2nd-k8s-0: 19:23:34 INFO juju.downloader download verified ("ch:amd64/jammy/mysql-k8s-80") unit-mysql-2nd-k8s-0: 19:23:49 INFO juju.worker.uniter hooks are retried true unit-mysql-2nd-k8s-0: 19:23:49 INFO juju.worker.uniter found queued "install" hook unit-mysql-2nd-k8s-0: 19:23:50 DEBUG unit.mysql-2nd-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-2nd-k8s-0: 19:23:50 INFO unit.mysql-2nd-k8s/0.juju-log Running legacy hooks/install. unit-mysql-2nd-k8s-0: 19:23:51 DEBUG unit.mysql-2nd-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-2nd-k8s-0: 19:23:51 DEBUG unit.mysql-2nd-k8s/0.juju-log Charm called itself via hooks/install. unit-mysql-2nd-k8s-0: 19:23:51 DEBUG unit.mysql-2nd-k8s/0.juju-log Legacy hooks/install exited with status 0. unit-mysql-2nd-k8s-0: 19:23:51 DEBUG unit.mysql-2nd-k8s/0.juju-log Using local storage: not a Kubernetes podspec charm unit-mysql-2nd-k8s-0: 19:23:51 DEBUG unit.mysql-2nd-k8s/0.juju-log Initializing SQLite local storage: /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/.unit-state.db. unit-mysql-2nd-k8s-0: 19:23:51 DEBUG unit.mysql-2nd-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-2nd-k8s-0: 19:23:51 DEBUG unit.mysql-2nd-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-2nd-k8s-0: 19:23:51 DEBUG unit.mysql-2nd-k8s/0.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:23:51 DEBUG unit.mysql-2nd-k8s/0.juju-log Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:23:51 DEBUG unit.mysql-2nd-k8s/0.juju-log Emitting Juju event install. unit-mysql-2nd-k8s-0: 19:23:51 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 19:23:52 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Operator Framework 2.2.0 up and running. unit-mysql-2nd-k8s-0: 19:23:52 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:23:52 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Emitting Juju event backend_database_relation_joined. unit-mysql-2nd-k8s-0: 19:23:52 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-2nd-k8s-0: 19:23:52 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-2nd-k8s-0: 19:23:52 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:23:52 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:23:52 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Emitting Juju event database_relation_created. unit-mysql-router-k8s-0: 19:23:52 INFO juju.worker.uniter.operation ran "backend-database-relation-joined" hook (via hook dispatching script: dispatch) unit-mysql-2nd-k8s-0: 19:23:52 INFO juju.worker.uniter.operation ran "database-relation-created" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 19:23:53 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:23:53 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Emitting Juju event backend_database_relation_changed. unit-mysql-2nd-k8s-0: 19:23:53 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: Operator Framework 2.2.0 up and running. unit-mysql-2nd-k8s-0: 19:23:53 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-2nd-k8s-0: 19:23:53 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-2nd-k8s-0: 19:23:53 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:23:53 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:23:53 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: Emitting Juju event database_peers_relation_created. unit-mysql-router-k8s-0: 19:23:53 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-2nd-k8s-0: 19:23:53 INFO juju.worker.uniter.operation ran "database-peers-relation-created" hook (via hook dispatching script: dispatch) unit-mysql-2nd-k8s-0: 19:23:53 INFO juju.worker.uniter found queued "leader-elected" hook unit-mysql-2nd-k8s-0: 19:23:54 DEBUG unit.mysql-2nd-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-2nd-k8s-0: 19:23:54 DEBUG unit.mysql-2nd-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-2nd-k8s-0: 19:23:54 DEBUG unit.mysql-2nd-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-2nd-k8s-0: 19:23:54 DEBUG unit.mysql-2nd-k8s/0.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:23:54 DEBUG unit.mysql-2nd-k8s/0.juju-log Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:23:54 DEBUG unit.mysql-2nd-k8s/0.juju-log Emitting Juju event leader_elected. unit-mysql-2nd-k8s-0: 19:23:54 DEBUG unit.mysql-2nd-k8s/0.juju-log Setting root-password unit-mysql-2nd-k8s-0: 19:23:54 DEBUG unit.mysql-2nd-k8s/0.juju-log Setting server-config-password unit-mysql-2nd-k8s-0: 19:23:54 DEBUG unit.mysql-2nd-k8s/0.juju-log Setting cluster-admin-password unit-mysql-2nd-k8s-0: 19:23:54 DEBUG unit.mysql-2nd-k8s/0.juju-log Setting monitoring-password unit-mysql-2nd-k8s-0: 19:23:54 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch) unit-mysql-2nd-k8s-0: 19:23:55 DEBUG unit.mysql-2nd-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-2nd-k8s-0: 19:23:55 DEBUG unit.mysql-2nd-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-2nd-k8s-0: 19:23:55 DEBUG unit.mysql-2nd-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-2nd-k8s-0: 19:23:55 DEBUG unit.mysql-2nd-k8s/0.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:23:55 DEBUG unit.mysql-2nd-k8s/0.juju-log Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:23:55 DEBUG unit.mysql-2nd-k8s/0.juju-log Emitting Juju event mysql_pebble_ready. unit-mysql-2nd-k8s-0: 19:23:55 DEBUG unit.mysql-2nd-k8s/0.juju-log Leader not ready yet, waiting... unit-mysql-2nd-k8s-0: 19:23:55 DEBUG unit.mysql-2nd-k8s/0.juju-log Deferring . unit-mysql-2nd-k8s-0: 19:23:55 INFO juju.worker.uniter.operation ran "mysql-pebble-ready" hook (via hook dispatching script: dispatch) unit-mysql-2nd-k8s-0: 19:23:56 DEBUG unit.mysql-2nd-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-2nd-k8s-0: 19:23:56 DEBUG unit.mysql-2nd-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-2nd-k8s-0: 19:23:56 DEBUG unit.mysql-2nd-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-2nd-k8s-0: 19:23:56 DEBUG unit.mysql-2nd-k8s/0.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:23:56 DEBUG unit.mysql-2nd-k8s/0.juju-log Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:23:56 DEBUG unit.mysql-2nd-k8s/0.juju-log Re-emitting deferred event . unit-mysql-2nd-k8s-0: 19:23:56 DEBUG unit.mysql-2nd-k8s/0.juju-log Leader not ready yet, waiting... unit-mysql-2nd-k8s-0: 19:23:56 DEBUG unit.mysql-2nd-k8s/0.juju-log Deferring . unit-mysql-2nd-k8s-0: 19:23:56 DEBUG unit.mysql-2nd-k8s/0.juju-log Emitting Juju event database_storage_attached. unit-mysql-2nd-k8s-0: 19:23:56 INFO juju.worker.uniter.operation ran "database-storage-attached" hook (via hook dispatching script: dispatch) unit-mysql-2nd-k8s-0: 19:23:57 DEBUG unit.mysql-2nd-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-2nd-k8s-0: 19:23:57 DEBUG unit.mysql-2nd-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-2nd-k8s-0: 19:23:57 DEBUG unit.mysql-2nd-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-2nd-k8s-0: 19:23:57 DEBUG unit.mysql-2nd-k8s/0.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:23:57 DEBUG unit.mysql-2nd-k8s/0.juju-log Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:23:57 DEBUG unit.mysql-2nd-k8s/0.juju-log Re-emitting deferred event . unit-mysql-2nd-k8s-0: 19:23:57 DEBUG unit.mysql-2nd-k8s/0.juju-log Leader not ready yet, waiting... unit-mysql-2nd-k8s-0: 19:23:57 DEBUG unit.mysql-2nd-k8s/0.juju-log Deferring . unit-mysql-2nd-k8s-0: 19:23:57 DEBUG unit.mysql-2nd-k8s/0.juju-log Emitting Juju event config_changed. unit-mysql-2nd-k8s-0: 19:23:57 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch) unit-mysql-2nd-k8s-0: 19:23:57 INFO juju.worker.uniter found queued "start" hook unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-2nd-k8s-0: 19:23:58 INFO unit.mysql-2nd-k8s/0.juju-log Running legacy hooks/start. unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log Charm called itself via hooks/start. unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log Legacy hooks/start exited with status 0. unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log Re-emitting deferred event . unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log connection.connect_tcp.started host='10.152.183.1' port=443 local_address=None timeout=None unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log connection.connect_tcp.complete return_value= unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log connection.start_tls.started ssl_context= server_hostname='10.152.183.1' timeout=None unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log connection.start_tls.complete return_value= unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log http11.send_request_headers.started request= unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log http11.send_request_headers.complete unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log http11.send_request_body.started request= unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log http11.send_request_body.complete unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log http11.receive_response_headers.started request= unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'7c46cc8f-f566-42fb-bf82-b0b412e300a8'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:23:58 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-2nd-k8s-0: 19:23:58 INFO unit.mysql-2nd-k8s/0.juju-log HTTP Request: GET https://10.152.183.1/api/v1/namespaces/foo46/pods/mysql-2nd-k8s-0 "HTTP/1.1 200 OK" unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log http11.receive_response_body.started request= unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log http11.receive_response_body.complete unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log http11.response_closed.started unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log http11.response_closed.complete unit-mysql-2nd-k8s-0: 19:23:58 INFO unit.mysql-2nd-k8s/0.juju-log Retrieving the total memory of the server unit-mysql-2nd-k8s-0: 19:23:58 DEBUG unit.mysql-2nd-k8s/0.juju-log Initializing instance unit-mysql-2nd-k8s-0: 19:24:06 DEBUG unit.mysql-2nd-k8s/0.juju-log Adding pebble layer unit-mysql-2nd-k8s-0: 19:24:06 DEBUG unit.mysql-2nd-k8s/0.juju-log Safe stopping mysqld safe unit-mysql-2nd-k8s-0: 19:24:07 DEBUG unit.mysql-2nd-k8s/0.juju-log Waiting for instance to be ready unit-mysql-2nd-k8s-0: 19:24:12 INFO unit.mysql-2nd-k8s/0.juju-log Configuring instance unit-mysql-2nd-k8s-0: 19:24:12 DEBUG unit.mysql-2nd-k8s/0.juju-log Configuring users unit-mysql-2nd-k8s-0: 19:24:12 DEBUG unit.mysql-2nd-k8s/0.juju-log Configuring instance for InnoDB on mysql-2nd-k8s-0.mysql-2nd-k8s-endpoints unit-mysql-2nd-k8s-0: 19:24:19 DEBUG unit.mysql-2nd-k8s/0.juju-log Getting InnoDB version unit-mysql-2nd-k8s-0: 19:24:19 INFO unit.mysql-2nd-k8s/0.juju-log Creating cluster on the leader unit unit-mysql-2nd-k8s-0: 19:24:19 DEBUG unit.mysql-2nd-k8s/0.juju-log Creating a MySQL InnoDB cluster on mysql-2nd-k8s-0.mysql-2nd-k8s-endpoints unit-mysql-2nd-k8s-0: 19:24:22 DEBUG unit.mysql-2nd-k8s/0.juju-log Emitting Juju event start. unit-mysql-2nd-k8s-0: 19:24:22 DEBUG unit.mysql-2nd-k8s/0.juju-log Alert rules path does not exist: src/prometheus_alert_rules unit-mysql-2nd-k8s-0: 19:24:23 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch) unit-mysql-2nd-k8s-0: 19:24:23 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Operator Framework 2.2.0 up and running. unit-mysql-2nd-k8s-0: 19:24:23 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-2nd-k8s-0: 19:24:23 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-2nd-k8s-0: 19:24:23 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:24:23 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:24:23 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Emitting Juju event database_relation_changed. unit-mysql-2nd-k8s-0: 19:24:23 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Emitting custom event . unit-mysql-2nd-k8s-0: 19:24:23 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Getting InnoDB version unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Updating pod labels unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: connection.connect_tcp.started host='10.152.183.1' port=443 local_address=None timeout=None unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: connection.connect_tcp.complete return_value= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: connection.start_tls.started ssl_context= server_hostname='10.152.183.1' timeout=None unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: connection.start_tls.complete return_value= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_headers.started request= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_headers.complete unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_body.started request= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_body.complete unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_headers.started request= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'406bd211-c230-46b4-b7d6-13d1d9b23442'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:24:24 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-2nd-k8s-0: 19:24:24 INFO unit.mysql-2nd-k8s/0.juju-log database:4: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/foo46/pods/mysql-2nd-k8s-0 "HTTP/1.1 200 OK" unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_body.started request= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_body.complete unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.response_closed.started unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.response_closed.complete unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_headers.started request= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_headers.complete unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_body.started request= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_body.complete unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_headers.started request= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'ef72014c-2aa2-4068-ba0b-d201c22b17d8'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:24:24 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-2nd-k8s-0: 19:24:24 INFO unit.mysql-2nd-k8s/0.juju-log database:4: HTTP Request: PATCH https://10.152.183.1/api/v1/namespaces/foo46/pods/mysql-2nd-k8s-0 "HTTP/1.1 200 OK" unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_body.started request= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_body.complete unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.response_closed.started unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.response_closed.complete unit-mysql-2nd-k8s-0: 19:24:24 INFO unit.mysql-2nd-k8s/0.juju-log database:4: Kubernetes pod label primary created unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_headers.started request= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_headers.complete unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_body.started request= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_body.complete unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_headers.started request= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'9da8cb27-e858-4e61-a155-f7c0857a9696'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:24:24 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-2nd-k8s-0: 19:24:24 INFO unit.mysql-2nd-k8s/0.juju-log database:4: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/foo46/pods/mysql-2nd-k8s-0 "HTTP/1.1 200 OK" unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_body.started request= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_body.complete unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.response_closed.started unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.response_closed.complete unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_headers.started request= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_headers.complete unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_body.started request= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_body.complete unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_headers.started request= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 201, b'Created', [(b'Audit-Id', b'2d55620d-231a-439e-8315-9751af18e801'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:24:24 GMT'), (b'Content-Length', b'1282')]) unit-mysql-2nd-k8s-0: 19:24:24 INFO unit.mysql-2nd-k8s/0.juju-log database:4: HTTP Request: POST https://10.152.183.1/api/v1/namespaces/foo46/services "HTTP/1.1 201 Created" unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_body.started request= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_body.complete unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.response_closed.started unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.response_closed.complete unit-mysql-2nd-k8s-0: 19:24:24 INFO unit.mysql-2nd-k8s/0.juju-log database:4: Kubernetes service mysql-2nd-k8s-primary created unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_headers.started request= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_headers.complete unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_body.started request= unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_body.complete unit-mysql-2nd-k8s-0: 19:24:24 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_headers.started request= unit-mysql-2nd-k8s-0: 19:24:25 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'831a6442-a4f5-417a-8d78-17e8e6908055'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:24:24 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-2nd-k8s-0: 19:24:25 INFO unit.mysql-2nd-k8s/0.juju-log database:4: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/foo46/pods/mysql-2nd-k8s-0 "HTTP/1.1 200 OK" unit-mysql-2nd-k8s-0: 19:24:25 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_body.started request= unit-mysql-2nd-k8s-0: 19:24:25 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_body.complete unit-mysql-2nd-k8s-0: 19:24:25 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.response_closed.started unit-mysql-2nd-k8s-0: 19:24:25 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.response_closed.complete unit-mysql-2nd-k8s-0: 19:24:25 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_headers.started request= unit-mysql-2nd-k8s-0: 19:24:25 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_headers.complete unit-mysql-2nd-k8s-0: 19:24:25 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_body.started request= unit-mysql-2nd-k8s-0: 19:24:25 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.send_request_body.complete unit-mysql-2nd-k8s-0: 19:24:25 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_headers.started request= unit-mysql-2nd-k8s-0: 19:24:25 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 201, b'Created', [(b'Audit-Id', b'2c23d682-610b-4e77-a9e7-f9fd15d78d65'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:24:25 GMT'), (b'Content-Length', b'1284')]) unit-mysql-2nd-k8s-0: 19:24:25 INFO unit.mysql-2nd-k8s/0.juju-log database:4: HTTP Request: POST https://10.152.183.1/api/v1/namespaces/foo46/services "HTTP/1.1 201 Created" unit-mysql-2nd-k8s-0: 19:24:25 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_body.started request= unit-mysql-2nd-k8s-0: 19:24:25 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.receive_response_body.complete unit-mysql-2nd-k8s-0: 19:24:25 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.response_closed.started unit-mysql-2nd-k8s-0: 19:24:25 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: http11.response_closed.complete unit-mysql-2nd-k8s-0: 19:24:25 INFO unit.mysql-2nd-k8s/0.juju-log database:4: Kubernetes service mysql-2nd-k8s-replicas created unit-mysql-2nd-k8s-0: 19:24:25 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Kubernetes service endpoint not ready yet unit-mysql-2nd-k8s-0: 19:24:26 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Kubernetes service endpoint ready unit-mysql-2nd-k8s-0: 19:24:26 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Getting cluster primary member's address from mysql-2nd-k8s-0.mysql-2nd-k8s-endpoints unit-mysql-2nd-k8s-0: 19:24:26 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Getting cluster primary member's address from mysql-2nd-k8s-0.mysql-2nd-k8s-endpoints unit-mysql-2nd-k8s-0: 19:24:27 INFO unit.mysql-2nd-k8s/0.juju-log database:4: Created user for app mysql-router-k8s unit-mysql-2nd-k8s-0: 19:24:27 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 19:24:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:24:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Emitting Juju event backend_database_relation_changed. unit-mysql-router-k8s-0: 19:24:28 INFO unit.mysql-router-k8s/0.juju-log backend-database:4: database created at 2023-05-18 19:24:28.351091 unit-mysql-router-k8s-0: 19:24:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Emitting custom event . unit-mysql-router-k8s-0: 19:24:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: State of reconcile self.unit.is_leader()=True, isinstance(self.workload, workload.AuthenticatedWorkload)=True, self.database_requires.relation and self.database_requires.relation.is_breaking(event)=False, self.workload.container_ready=True, isinstance(event, ops.UpgradeCharmEvent)=False unit-mysql-router-k8s-0: 19:24:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Reconciling users event=, router_endpoint='mysql-router-k8s.foo46.svc.cluster.local' unit-mysql-router-k8s-0: 19:24:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: State of reconcile users requested_users=[], created_users=[] unit-mysql-router-k8s-0: 19:24:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Reconciled users event=, router_endpoint='mysql-router-k8s.foo46.svc.cluster.local' unit-mysql-router-k8s-0: 19:24:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Enabling MySQL Router service unit-mysql-router-k8s-0: 19:24:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Bootstrapping router tls=False, self._database_requires_relation.host='mysql-2nd-k8s-primary.foo46.svc.cluster.local', self._database_requires_relation.port='3306' unit-mysql-2nd-k8s-0: 19:24:28 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Operator Framework 2.2.0 up and running. unit-mysql-2nd-k8s-0: 19:24:28 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-2nd-k8s-0: 19:24:28 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-2nd-k8s-0: 19:24:28 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:24:28 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:24:28 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Emitting Juju event database_relation_joined. unit-mysql-router-k8s-0: 19:24:28 ERROR unit.mysql-router-k8s/0.juju-log backend-database:4: Failed to bootstrap router stderr: Error: The given Router instance is already configured for a cluster named 'cluster_3e97a9b353fcfb511f0181d85ae3dd44'. If you'd like to replace it, please use the --force configuration option. Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/src/workload.py", line 175, in _bootstrap_router process.wait_output() File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/pebble.py", line 1252, in wait_output raise ExecError(self._command, exit_code, out_value, err_value) ops.pebble.ExecError: non-zero exit code 1 executing ['mysqlrouter', '--bootstrap', 'relation-4:ZJfWC6Jq12uldwx8Px0HcBMz@mysql-2nd-k8s-primary.foo46.svc.cluster.local:3306', '--strict', '--user', 'mysql', '--conf-set-option', 'http_server.bind_address=127.0.0.1', '--conf-use-gr-notifications'], stdout='', stderr="Error: The given Router instance is already configured for a cluster named 'cluster_3e97a9b353fcfb511f0181d85ae3dd44'.\nIf you'd like to replace it, please use the --force configuration option.\n" unit-mysql-router-k8s-0: 19:24:28 ERROR unit.mysql-router-k8s/0.juju-log backend-database:4: Uncaught exception while in charm code: Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/./src/charm.py", line 250, in ops.main.main(MySQLRouterOperatorCharm) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/main.py", line 441, in main _emit_charm_event(charm, dispatcher.event_name) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/main.py", line 149, in _emit_charm_event event_to_emit.emit(*args, **kwargs) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 354, in emit framework._emit(event) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 830, in _emit self._reemit(event_path) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 919, in _reemit custom_handler(event) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 946, in _on_relation_changed_event self.on.database_created.emit(event.relation, app=event.app, unit=event.unit) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 354, in emit framework._emit(event) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 830, in _emit self._reemit(event_path) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 919, in _reemit custom_handler(event) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/./src/charm.py", line 224, in reconcile_database_relations self.workload.enable(tls=self.tls.certificate_saved, unit_name=self.unit.name) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/src/workload.py", line 208, in enable self._bootstrap_router(tls=tls) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/src/workload.py", line 175, in _bootstrap_router process.wait_output() File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/pebble.py", line 1252, in wait_output raise ExecError(self._command, exit_code, out_value, err_value) ops.pebble.ExecError: non-zero exit code 1 executing ['mysqlrouter', '--bootstrap', 'relation-4:ZJfWC6Jq12uldwx8Px0HcBMz@mysql-2nd-k8s-primary.foo46.svc.cluster.local:3306', '--strict', '--user', 'mysql', '--conf-set-option', 'http_server.bind_address=127.0.0.1', '--conf-use-gr-notifications'], stdout='', stderr="Error: The given Router instance is already configured for a cluster named 'cluster_3e97a9b353fcfb511f0181d85ae3dd44'.\nIf you'd like to replace it, please use the --force configuration option.\n" unit-mysql-2nd-k8s-0: 19:24:28 INFO juju.worker.uniter.operation ran "database-relation-joined" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 19:24:28 ERROR juju.worker.uniter.operation hook "backend-database-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1 unit-mysql-router-k8s-0: 19:24:28 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook unit-mysql-2nd-k8s-0: 19:24:29 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Operator Framework 2.2.0 up and running. unit-mysql-2nd-k8s-0: 19:24:29 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-2nd-k8s-0: 19:24:29 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-2nd-k8s-0: 19:24:29 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:24:29 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:24:29 DEBUG unit.mysql-2nd-k8s/0.juju-log database:4: Emitting Juju event database_relation_changed. unit-mysql-2nd-k8s-0: 19:24:29 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: Operator Framework 2.2.0 up and running. unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-2nd-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: Emitting Juju event database_peers_relation_changed. unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: Updating pod labels unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: connection.connect_tcp.started host='10.152.183.1' port=443 local_address=None timeout=None unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: connection.connect_tcp.complete return_value= unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: connection.start_tls.started ssl_context= server_hostname='10.152.183.1' timeout=None unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: connection.start_tls.complete return_value= unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.send_request_headers.started request= unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.send_request_headers.complete unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.send_request_body.started request= unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.send_request_body.complete unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.receive_response_headers.started request= unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'ec87dac8-9a67-4ccc-9918-adbb7a1e5f3a'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:24:30 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-2nd-k8s-0: 19:24:30 INFO unit.mysql-2nd-k8s/0.juju-log database-peers:3: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/foo46/pods/mysql-2nd-k8s-0 "HTTP/1.1 200 OK" unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.receive_response_body.started request= unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.receive_response_body.complete unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.response_closed.started unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.response_closed.complete unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.send_request_headers.started request= unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.send_request_headers.complete unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.send_request_body.started request= unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.send_request_body.complete unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.receive_response_headers.started request= unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'2f9f8f2e-bb9e-4329-952b-156062535829'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'98a7e6af-0217-4337-9a38-f72d38fda03f'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'ac0f11ef-c423-4407-a80f-b1a2487d4983'), (b'Date', b'Thu, 18 May 2023 19:24:30 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-2nd-k8s-0: 19:24:30 INFO unit.mysql-2nd-k8s/0.juju-log database-peers:3: HTTP Request: PATCH https://10.152.183.1/api/v1/namespaces/foo46/pods/mysql-2nd-k8s-0 "HTTP/1.1 200 OK" unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.receive_response_body.started request= unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.receive_response_body.complete unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.response_closed.started unit-mysql-2nd-k8s-0: 19:24:30 DEBUG unit.mysql-2nd-k8s/0.juju-log database-peers:3: http11.response_closed.complete unit-mysql-2nd-k8s-0: 19:24:30 INFO unit.mysql-2nd-k8s/0.juju-log database-peers:3: Kubernetes pod label primary created unit-mysql-2nd-k8s-0: 19:24:31 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 19:24:33 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook unit-mysql-router-k8s-0: 19:24:34 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:24:34 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Emitting Juju event backend_database_relation_changed. unit-mysql-router-k8s-0: 19:24:34 INFO unit.mysql-router-k8s/0.juju-log backend-database:4: database created at 2023-05-18 19:24:34.326408 unit-mysql-router-k8s-0: 19:24:34 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Emitting custom event . unit-mysql-router-k8s-0: 19:24:34 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: State of reconcile self.unit.is_leader()=True, isinstance(self.workload, workload.AuthenticatedWorkload)=True, self.database_requires.relation and self.database_requires.relation.is_breaking(event)=False, self.workload.container_ready=True, isinstance(event, ops.UpgradeCharmEvent)=False unit-mysql-router-k8s-0: 19:24:34 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Reconciling users event=, router_endpoint='mysql-router-k8s.foo46.svc.cluster.local' unit-mysql-router-k8s-0: 19:24:34 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: State of reconcile users requested_users=[], created_users=[] unit-mysql-router-k8s-0: 19:24:34 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Reconciled users event=, router_endpoint='mysql-router-k8s.foo46.svc.cluster.local' unit-mysql-router-k8s-0: 19:24:34 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Enabling MySQL Router service unit-mysql-router-k8s-0: 19:24:34 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Bootstrapping router tls=False, self._database_requires_relation.host='mysql-2nd-k8s-primary.foo46.svc.cluster.local', self._database_requires_relation.port='3306' unit-mysql-router-k8s-0: 19:24:34 ERROR unit.mysql-router-k8s/0.juju-log backend-database:4: Failed to bootstrap router stderr: Error: The given Router instance is already configured for a cluster named 'cluster_3e97a9b353fcfb511f0181d85ae3dd44'. If you'd like to replace it, please use the --force configuration option. Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/src/workload.py", line 175, in _bootstrap_router process.wait_output() File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/pebble.py", line 1252, in wait_output raise ExecError(self._command, exit_code, out_value, err_value) ops.pebble.ExecError: non-zero exit code 1 executing ['mysqlrouter', '--bootstrap', 'relation-4:ZJfWC6Jq12uldwx8Px0HcBMz@mysql-2nd-k8s-primary.foo46.svc.cluster.local:3306', '--strict', '--user', 'mysql', '--conf-set-option', 'http_server.bind_address=127.0.0.1', '--conf-use-gr-notifications'], stdout='', stderr="Error: The given Router instance is already configured for a cluster named 'cluster_3e97a9b353fcfb511f0181d85ae3dd44'.\nIf you'd like to replace it, please use the --force configuration option.\n" unit-mysql-router-k8s-0: 19:24:34 ERROR unit.mysql-router-k8s/0.juju-log backend-database:4: Uncaught exception while in charm code: Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/./src/charm.py", line 250, in ops.main.main(MySQLRouterOperatorCharm) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/main.py", line 441, in main _emit_charm_event(charm, dispatcher.event_name) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/main.py", line 149, in _emit_charm_event event_to_emit.emit(*args, **kwargs) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 354, in emit framework._emit(event) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 830, in _emit self._reemit(event_path) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 919, in _reemit custom_handler(event) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 946, in _on_relation_changed_event self.on.database_created.emit(event.relation, app=event.app, unit=event.unit) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 354, in emit framework._emit(event) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 830, in _emit self._reemit(event_path) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 919, in _reemit custom_handler(event) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/./src/charm.py", line 224, in reconcile_database_relations self.workload.enable(tls=self.tls.certificate_saved, unit_name=self.unit.name) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/src/workload.py", line 208, in enable self._bootstrap_router(tls=tls) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/src/workload.py", line 175, in _bootstrap_router process.wait_output() File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/pebble.py", line 1252, in wait_output raise ExecError(self._command, exit_code, out_value, err_value) ops.pebble.ExecError: non-zero exit code 1 executing ['mysqlrouter', '--bootstrap', 'relation-4:ZJfWC6Jq12uldwx8Px0HcBMz@mysql-2nd-k8s-primary.foo46.svc.cluster.local:3306', '--strict', '--user', 'mysql', '--conf-set-option', 'http_server.bind_address=127.0.0.1', '--conf-use-gr-notifications'], stdout='', stderr="Error: The given Router instance is already configured for a cluster named 'cluster_3e97a9b353fcfb511f0181d85ae3dd44'.\nIf you'd like to replace it, please use the --force configuration option.\n" unit-mysql-router-k8s-0: 19:24:34 ERROR juju.worker.uniter.operation hook "backend-database-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1 unit-mysql-router-k8s-0: 19:24:34 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook unit-mysql-router-k8s-0: 19:24:44 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook unit-mysql-router-k8s-0: 19:24:45 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 19:24:45 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Emitting Juju event backend_database_relation_changed. unit-mysql-router-k8s-0: 19:24:45 INFO unit.mysql-router-k8s/0.juju-log backend-database:4: database created at 2023-05-18 19:24:45.333858 unit-mysql-router-k8s-0: 19:24:45 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Emitting custom event . unit-mysql-router-k8s-0: 19:24:45 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: State of reconcile self.unit.is_leader()=True, isinstance(self.workload, workload.AuthenticatedWorkload)=True, self.database_requires.relation and self.database_requires.relation.is_breaking(event)=False, self.workload.container_ready=True, isinstance(event, ops.UpgradeCharmEvent)=False unit-mysql-router-k8s-0: 19:24:45 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Reconciling users event=, router_endpoint='mysql-router-k8s.foo46.svc.cluster.local' unit-mysql-router-k8s-0: 19:24:45 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: State of reconcile users requested_users=[], created_users=[] unit-mysql-router-k8s-0: 19:24:45 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Reconciled users event=, router_endpoint='mysql-router-k8s.foo46.svc.cluster.local' unit-mysql-router-k8s-0: 19:24:45 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Enabling MySQL Router service unit-mysql-router-k8s-0: 19:24:45 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:4: Bootstrapping router tls=False, self._database_requires_relation.host='mysql-2nd-k8s-primary.foo46.svc.cluster.local', self._database_requires_relation.port='3306' unit-mysql-router-k8s-0: 19:24:45 ERROR unit.mysql-router-k8s/0.juju-log backend-database:4: Failed to bootstrap router stderr: Error: The given Router instance is already configured for a cluster named 'cluster_3e97a9b353fcfb511f0181d85ae3dd44'. If you'd like to replace it, please use the --force configuration option. Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/src/workload.py", line 175, in _bootstrap_router process.wait_output() File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/pebble.py", line 1252, in wait_output raise ExecError(self._command, exit_code, out_value, err_value) ops.pebble.ExecError: non-zero exit code 1 executing ['mysqlrouter', '--bootstrap', 'relation-4:ZJfWC6Jq12uldwx8Px0HcBMz@mysql-2nd-k8s-primary.foo46.svc.cluster.local:3306', '--strict', '--user', 'mysql', '--conf-set-option', 'http_server.bind_address=127.0.0.1', '--conf-use-gr-notifications'], stdout='', stderr="Error: The given Router instance is already configured for a cluster named 'cluster_3e97a9b353fcfb511f0181d85ae3dd44'.\nIf you'd like to replace it, please use the --force configuration option.\n" unit-mysql-router-k8s-0: 19:24:45 ERROR unit.mysql-router-k8s/0.juju-log backend-database:4: Uncaught exception while in charm code: Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/./src/charm.py", line 250, in ops.main.main(MySQLRouterOperatorCharm) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/main.py", line 441, in main _emit_charm_event(charm, dispatcher.event_name) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/main.py", line 149, in _emit_charm_event event_to_emit.emit(*args, **kwargs) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 354, in emit framework._emit(event) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 830, in _emit self._reemit(event_path) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 919, in _reemit custom_handler(event) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/lib/charms/data_platform_libs/v0/data_interfaces.py", line 946, in _on_relation_changed_event self.on.database_created.emit(event.relation, app=event.app, unit=event.unit) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 354, in emit framework._emit(event) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 830, in _emit self._reemit(event_path) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/framework.py", line 919, in _reemit custom_handler(event) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/./src/charm.py", line 224, in reconcile_database_relations self.workload.enable(tls=self.tls.certificate_saved, unit_name=self.unit.name) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/src/workload.py", line 208, in enable self._bootstrap_router(tls=tls) File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/src/workload.py", line 175, in _bootstrap_router process.wait_output() File "/var/lib/juju/agents/unit-mysql-router-k8s-0/charm/venv/ops/pebble.py", line 1252, in wait_output raise ExecError(self._command, exit_code, out_value, err_value) ops.pebble.ExecError: non-zero exit code 1 executing ['mysqlrouter', '--bootstrap', 'relation-4:ZJfWC6Jq12uldwx8Px0HcBMz@mysql-2nd-k8s-primary.foo46.svc.cluster.local:3306', '--strict', '--user', 'mysql', '--conf-set-option', 'http_server.bind_address=127.0.0.1', '--conf-use-gr-notifications'], stdout='', stderr="Error: The given Router instance is already configured for a cluster named 'cluster_3e97a9b353fcfb511f0181d85ae3dd44'.\nIf you'd like to replace it, please use the --force configuration option.\n" unit-mysql-router-k8s-0: 19:24:45 ERROR juju.worker.uniter.operation hook "backend-database-relation-changed" (via hook dispatching script: dispatch) failed: exit status 1 unit-mysql-router-k8s-0: 19:24:45 INFO juju.worker.uniter awaiting error resolution for "relation-changed" hook ```
github-actions[bot] commented 1 year ago

https://warthogs.atlassian.net/browse/DPE-1983