search

canonical / mysql-k8s-operator

A Charmed Operator for running MySQL on Kubernetes
https://charmhub.io/mysql-k8s
Apache License 2.0
8 stars 15 forks source link

Invalid SQL syntax in delete_users_for_relation for MySQL Router relation #230

Closed carlcsaposs-canonical closed 1 year ago

carlcsaposs-canonical commented 1 year ago

Steps to reproduce

  1. juju deploy mysql-k8s --channel 8.0/edge --trust --series jammy
  2. juju deploy mysql-router-k8s --channel 8.0/edge --series jammy
  3. juju relate mysql-k8s mysql-router-k8s
  4. Wait for active/idle on mysql-k8s and blocked/idle on mysql-router-k8s
  5. juju remove-relation mysql-k8s mysql-router-k8s

Expected behavior

No errors in debug log

Actual behavior

Error in debug log

Versions

Operating system: Ubuntu 22.04.2 LTS

Juju CLI: 2.9.43-ubuntu-amd64

Juju agent: 2.9.29

Charm revision: 80

microk8s: MicroK8s v1.26.4 revision 5219

Log output

unit-mysql-k8s-0: 13:10:31 ERROR unit.mysql-k8s/0.juju-log database:2: Failed to delete users for relation 2
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/src/mysql_k8s_helpers.py", line 658, in _run_mysqlsh_script
    stdout, _ = process.wait_output()
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/pebble.py", line 1252, in wait_output
    raise ExecError(self._command, exit_code, out_value, err_value)
ops.pebble.ExecError: non-zero exit code 1 executing ['/usr/bin/mysqlsh', '--no-wizard', '--python', '--verbose=1', '-f', '/tmp/script.py', ';', 'rm', '/tmp/script.py'], stdout='', stderr='Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nverbose: 2023-05-19T13:10:31Z: Loading startup files...\nverbose: 2023-05-19T13:10:31Z: Loading plugins...\nverbose: 2023-05-19T13:10:31Z: Connecting to MySQL at: serverconfig@mysql-k8s-0.mysql-k8s-endpoints.foo3.svc.cluster.local:3306\nverbose: 2023-05-19T13:10:31Z: Shell.connect: tid=87: CONNECTED: mysql-k8s-0.mysql-k8s-endpoints.foo3.svc.cluster.local:3306\nverbose: 2023-05-19T13:10:31Z: Now tracking \'sql_mode\' system variable.\nverbose: 2023-05-19T13:10:31Z: ClassicSession.run_sql: tid=87: MySQL Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near \'NULL\' at line 1, SQL: PREPARE stmt FROM @sql\nTraceback (most recent call last):\n  File "<string>", line 4, in <module>\nmysqlsh.DBError: MySQL Error (1064): ClassicSession.run_sql: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax ' [truncated]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/lib/charms/mysql/v0/mysql.py", line 626, in delete_users_for_relation
    self._run_mysqlsh_script("\n".join(drop_users_command))
  File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/src/mysql_k8s_helpers.py", line 661, in _run_mysqlsh_script
    raise MySQLClientError(e.stderr)
charms.mysql.v0.mysql.MySQLClientError: Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory
verbose: 2023-05-19T13:10:31Z: Loading startup files...
verbose: 2023-05-19T13:10:31Z: Loading plugins...
verbose: 2023-05-19T13:10:31Z: Connecting to MySQL at: serverconfig@mysql-k8s-0.mysql-k8s-endpoints.foo3.svc.cluster.local:3306
verbose: 2023-05-19T13:10:31Z: Shell.connect: tid=87: CONNECTED: mysql-k8s-0.mysql-k8s-endpoints.foo3.svc.cluster.local:3306
verbose: 2023-05-19T13:10:31Z: Now tracking 'sql_mode' system variable.
verbose: 2023-05-19T13:10:31Z: ClassicSession.run_sql: tid=87: MySQL Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'NULL' at line 1, SQL: PREPARE stmt FROM @sql
Traceback (most recent call last):
  File "<string>", line 4, in <module>
mysqlsh.DBError: MySQL Error (1064): ClassicSession.run_sql: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'NULL' at line 1

unit-mysql-k8s-0: 13:10:31 ERROR unit.mysql-k8s/0.juju-log database:2: Failed to delete user(s) for relation 2
unit-mysql-k8s-0: 13:10:31 INFO juju.worker.uniter.operation ran "database-relation-broken" hook (via hook dispatching script: dispatch)

Additional context

carlcsaposs-canonical commented 1 year ago
juju debug-log --replay ```shell controller-0: 13:08:44 INFO juju.worker.apicaller [8c6aa0] "controller-0" successfully connected to "localhost:17070" controller-0: 13:08:44 INFO juju.worker.logforwarder config change - log forwarding not enabled controller-0: 13:08:44 INFO juju.worker.logger logger worker started controller-0: 13:08:44 INFO juju.worker.pruner.statushistory status history config: max age: 336h0m0s, max collection size 5120M for foo3 (8c6aa098-7056-47e5-8e3f-9f515f87a48e) controller-0: 13:08:44 INFO juju.worker.pruner.action status history config: max age: 336h0m0s, max collection size 5120M for foo3 (8c6aa098-7056-47e5-8e3f-9f515f87a48e) controller-0: 13:08:52 INFO juju.worker.caasapplicationprovisioner.runner start "mysql-k8s" controller-0: 13:09:00 INFO juju.worker.caasapplicationprovisioner.runner start "mysql-router-k8s" unit-mysql-k8s-0: 13:09:03 INFO juju.cmd running containerAgent [2.9.29 54b87ef5071691a2f089e26395908794321009a7 gc go1.17.9] unit-mysql-k8s-0: 13:09:03 INFO juju.cmd.containeragent.unit start "unit" unit-mysql-k8s-0: 13:09:03 INFO juju.worker.upgradesteps upgrade steps for 2.9.29 have already been run. unit-mysql-k8s-0: 13:09:03 INFO juju.worker.probehttpserver starting http server on [::]:3856 unit-mysql-k8s-0: 13:09:03 INFO juju.api cannot resolve "controller-service.controller-micro.svc.cluster.local": operation was canceled unit-mysql-k8s-0: 13:09:03 INFO juju.api connection established to "wss://10.152.183.191:17070/model/8c6aa098-7056-47e5-8e3f-9f515f87a48e/api" unit-mysql-k8s-0: 13:09:03 INFO juju.worker.apicaller [8c6aa0] "unit-mysql-k8s-0" successfully connected to "10.152.183.191:17070" unit-mysql-k8s-0: 13:09:04 INFO juju.api connection established to "wss://controller-service.controller-micro.svc.cluster.local:17070/model/8c6aa098-7056-47e5-8e3f-9f515f87a48e/api" unit-mysql-k8s-0: 13:09:04 INFO juju.worker.apicaller [8c6aa0] "unit-mysql-k8s-0" successfully connected to "controller-service.controller-micro.svc.cluster.local:17070" unit-mysql-k8s-0: 13:09:04 INFO juju.worker.migrationminion migration phase is now: NONE unit-mysql-k8s-0: 13:09:04 INFO juju.worker.logger logger worker started unit-mysql-k8s-0: 13:09:04 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: "" unit-mysql-k8s-0: 13:09:04 INFO juju.worker.caasupgrader abort check blocked until version event received unit-mysql-k8s-0: 13:09:04 INFO juju.worker.caasupgrader unblocking abort check unit-mysql-k8s-0: 13:09:04 INFO juju.worker.leadership mysql-k8s/0 promoted to leadership of mysql-k8s unit-mysql-k8s-0: 13:09:04 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-mysql-k8s-0 unit-mysql-k8s-0: 13:09:04 INFO juju.worker.uniter unit "mysql-k8s/0" started unit-mysql-k8s-0: 13:09:04 INFO juju.worker.uniter resuming charm install unit-mysql-k8s-0: 13:09:04 INFO juju.worker.uniter.charm downloading ch:amd64/jammy/mysql-k8s-80 from API server unit-mysql-k8s-0: 13:09:04 INFO juju.downloader downloading from ch:amd64/jammy/mysql-k8s-80 unit-mysql-k8s-0: 13:09:04 INFO juju.downloader download complete ("ch:amd64/jammy/mysql-k8s-80") unit-mysql-k8s-0: 13:09:04 INFO juju.downloader download verified ("ch:amd64/jammy/mysql-k8s-80") controller-0: 13:09:09 INFO juju.worker.caasapplicationprovisioner.runner stopped "mysql-router-k8s", err: Operation cannot be fulfilled on pods "mysql-router-k8s-0": the object has been modified; please apply your changes to the latest version and try again controller-0: 13:09:09 ERROR juju.worker.caasapplicationprovisioner.runner exited "mysql-router-k8s": Operation cannot be fulfilled on pods "mysql-router-k8s-0": the object has been modified; please apply your changes to the latest version and try again controller-0: 13:09:09 INFO juju.worker.caasapplicationprovisioner.runner restarting "mysql-router-k8s" in 3s unit-mysql-router-k8s-0: 13:09:08 INFO juju.cmd running containerAgent [2.9.29 54b87ef5071691a2f089e26395908794321009a7 gc go1.17.9] unit-mysql-router-k8s-0: 13:09:08 INFO juju.cmd.containeragent.unit start "unit" unit-mysql-router-k8s-0: 13:09:08 INFO juju.worker.upgradesteps upgrade steps for 2.9.29 have already been run. unit-mysql-router-k8s-0: 13:09:08 INFO juju.worker.probehttpserver starting http server on [::]:3856 unit-mysql-router-k8s-0: 13:09:08 INFO juju.api connection established to "wss://controller-service.controller-micro.svc.cluster.local:17070/model/8c6aa098-7056-47e5-8e3f-9f515f87a48e/api" unit-mysql-router-k8s-0: 13:09:09 INFO juju.worker.apicaller [8c6aa0] "unit-mysql-router-k8s-0" successfully connected to "controller-service.controller-micro.svc.cluster.local:17070" unit-mysql-router-k8s-0: 13:09:09 INFO juju.worker.migrationminion migration phase is now: NONE unit-mysql-router-k8s-0: 13:09:09 INFO juju.worker.logger logger worker started unit-mysql-router-k8s-0: 13:09:09 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: "" unit-mysql-router-k8s-0: 13:09:09 INFO juju.worker.leadership mysql-router-k8s/0 promoted to leadership of mysql-router-k8s unit-mysql-router-k8s-0: 13:09:09 INFO juju.worker.caasupgrader abort check blocked until version event received unit-mysql-router-k8s-0: 13:09:09 INFO juju.worker.caasupgrader unblocking abort check unit-mysql-router-k8s-0: 13:09:09 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-mysql-router-k8s-0 unit-mysql-router-k8s-0: 13:09:09 INFO juju.worker.uniter unit "mysql-router-k8s/0" started unit-mysql-router-k8s-0: 13:09:09 INFO juju.worker.uniter resuming charm install unit-mysql-router-k8s-0: 13:09:09 INFO juju.worker.uniter.charm downloading ch:amd64/jammy/mysql-router-k8s-31 from API server unit-mysql-router-k8s-0: 13:09:09 INFO juju.downloader downloading from ch:amd64/jammy/mysql-router-k8s-31 unit-mysql-router-k8s-0: 13:09:09 INFO juju.downloader download complete ("ch:amd64/jammy/mysql-router-k8s-31") unit-mysql-router-k8s-0: 13:09:09 INFO juju.downloader download verified ("ch:amd64/jammy/mysql-router-k8s-31") controller-0: 13:09:12 INFO juju.worker.caasapplicationprovisioner.runner start "mysql-router-k8s" unit-mysql-k8s-0: 13:09:15 INFO juju.worker.uniter hooks are retried true unit-mysql-k8s-0: 13:09:15 INFO juju.worker.uniter found queued "install" hook unit-mysql-k8s-0: 13:09:17 DEBUG unit.mysql-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 13:09:17 INFO unit.mysql-k8s/0.juju-log Running legacy hooks/install. unit-mysql-router-k8s-0: 13:09:17 INFO juju.worker.uniter hooks are retried true unit-mysql-k8s-0: 13:09:18 DEBUG unit.mysql-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 13:09:18 DEBUG unit.mysql-k8s/0.juju-log Charm called itself via hooks/install. unit-mysql-router-k8s-0: 13:09:18 INFO juju.worker.uniter found queued "install" hook unit-mysql-k8s-0: 13:09:18 DEBUG unit.mysql-k8s/0.juju-log Legacy hooks/install exited with status 0. unit-mysql-k8s-0: 13:09:18 DEBUG unit.mysql-k8s/0.juju-log Using local storage: not a Kubernetes podspec charm unit-mysql-k8s-0: 13:09:18 DEBUG unit.mysql-k8s/0.juju-log Initializing SQLite local storage: /var/lib/juju/agents/unit-mysql-k8s-0/charm/.unit-state.db. unit-mysql-k8s-0: 13:09:18 DEBUG unit.mysql-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 13:09:18 DEBUG unit.mysql-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 13:09:18 DEBUG unit.mysql-k8s/0.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:18 DEBUG unit.mysql-k8s/0.juju-log Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:18 DEBUG unit.mysql-k8s/0.juju-log Emitting Juju event install. unit-mysql-k8s-0: 13:09:18 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 13:09:19 INFO unit.mysql-router-k8s/0.juju-log Running legacy hooks/install. unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 13:09:19 DEBUG unit.mysql-k8s/0.juju-log database:2: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log Charm called itself via hooks/install. unit-mysql-k8s-0: 13:09:19 DEBUG unit.mysql-k8s/0.juju-log database:2: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 13:09:19 DEBUG unit.mysql-k8s/0.juju-log database:2: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 13:09:19 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:19 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log Legacy hooks/install exited with status 0. unit-mysql-k8s-0: 13:09:19 DEBUG unit.mysql-k8s/0.juju-log database:2: Emitting Juju event database_relation_created. unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log Using local storage: not a Kubernetes podspec charm unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log Initializing SQLite local storage: /var/lib/juju/agents/unit-mysql-router-k8s-0/charm/.unit-state.db. unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log Emitting Juju event install. unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log Patching k8s service name='mysql-router-k8s', ro_port=6447, rw_port=6446 unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log connection.connect_tcp.started host='10.152.183.1' port=443 local_address=None timeout=None unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log connection.connect_tcp.complete return_value= unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log connection.start_tls.started ssl_context= server_hostname='10.152.183.1' timeout=None unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log connection.start_tls.complete return_value= unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.send_request_headers.started request= unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.send_request_headers.complete unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.send_request_body.started request= unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.send_request_body.complete unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.receive_response_headers.started request= unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'0f4a0d1e-27a6-4322-be51-2671d3a9a74a'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'd178a7e5-16a6-4c1d-b375-269b5c847538'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'6aafa5ee-718d-41e3-ac79-a7986543b90b'), (b'Date', b'Fri, 19 May 2023 13:09:19 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-router-k8s-0: 13:09:19 INFO unit.mysql-router-k8s/0.juju-log HTTP Request: GET https://10.152.183.1/api/v1/namespaces/foo3/pods/mysql-router-k8s-0 "HTTP/1.1 200 OK" unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.receive_response_body.started request= unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.receive_response_body.complete unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.response_closed.started unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.response_closed.complete unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.send_request_headers.started request= unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.send_request_headers.complete unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.send_request_body.started request= unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.send_request_body.complete unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.receive_response_headers.started request= unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'496a8f1a-274e-4928-9f52-7f2adb34c0fe'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'd178a7e5-16a6-4c1d-b375-269b5c847538'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'6aafa5ee-718d-41e3-ac79-a7986543b90b'), (b'Date', b'Fri, 19 May 2023 13:09:19 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-router-k8s-0: 13:09:19 INFO unit.mysql-router-k8s/0.juju-log HTTP Request: PATCH https://10.152.183.1/api/v1/namespaces/foo3/services/mysql-router-k8s?fieldManager=mysql-router-k8s "HTTP/1.1 200 OK" unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.receive_response_body.started request= unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.receive_response_body.complete unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.response_closed.started unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log http11.response_closed.complete unit-mysql-router-k8s-0: 13:09:19 DEBUG unit.mysql-router-k8s/0.juju-log Patched k8s service name='mysql-router-k8s', ro_port=6447, rw_port=6446 unit-mysql-k8s-0: 13:09:19 INFO juju.worker.uniter.operation ran "database-relation-created" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 13:09:20 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 13:09:20 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 13:09:20 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 13:09:20 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 13:09:20 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:20 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:20 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Emitting Juju event database_peers_relation_created. unit-mysql-router-k8s-0: 13:09:20 DEBUG unit.mysql-router-k8s/0.juju-log mysql-router-peers:1: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 13:09:20 DEBUG unit.mysql-router-k8s/0.juju-log mysql-router-peers:1: Emitting Juju event mysql_router_peers_relation_created. unit-mysql-k8s-0: 13:09:21 INFO juju.worker.uniter.operation ran "database-peers-relation-created" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 13:09:21 INFO juju.worker.uniter found queued "leader-elected" hook unit-mysql-router-k8s-0: 13:09:21 INFO juju.worker.uniter.operation ran "mysql-router-peers-relation-created" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 13:09:21 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 13:09:21 DEBUG unit.mysql-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 13:09:21 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Emitting Juju event backend_database_relation_created. unit-mysql-k8s-0: 13:09:21 DEBUG unit.mysql-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 13:09:21 DEBUG unit.mysql-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 13:09:21 DEBUG unit.mysql-k8s/0.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:21 DEBUG unit.mysql-k8s/0.juju-log Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:21 DEBUG unit.mysql-k8s/0.juju-log Emitting Juju event leader_elected. unit-mysql-k8s-0: 13:09:21 DEBUG unit.mysql-k8s/0.juju-log Setting root-password unit-mysql-k8s-0: 13:09:21 DEBUG unit.mysql-k8s/0.juju-log Setting server-config-password unit-mysql-k8s-0: 13:09:21 DEBUG unit.mysql-k8s/0.juju-log Setting cluster-admin-password unit-mysql-k8s-0: 13:09:21 DEBUG unit.mysql-k8s/0.juju-log Setting monitoring-password unit-mysql-router-k8s-0: 13:09:21 INFO juju.worker.uniter.operation ran "backend-database-relation-created" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 13:09:22 INFO juju.worker.uniter found queued "leader-elected" hook unit-mysql-k8s-0: 13:09:22 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 13:09:22 DEBUG unit.mysql-router-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 13:09:22 DEBUG unit.mysql-router-k8s/0.juju-log Emitting Juju event leader_elected. unit-mysql-router-k8s-0: 13:09:22 DEBUG unit.mysql-router-k8s/0.juju-log Set status to BlockedStatus('Missing relation: database') unit-mysql-k8s-0: 13:09:22 DEBUG unit.mysql-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 13:09:22 DEBUG unit.mysql-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 13:09:22 DEBUG unit.mysql-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 13:09:22 DEBUG unit.mysql-k8s/0.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:22 DEBUG unit.mysql-k8s/0.juju-log Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:22 DEBUG unit.mysql-k8s/0.juju-log Emitting Juju event mysql_pebble_ready. unit-mysql-k8s-0: 13:09:22 DEBUG unit.mysql-k8s/0.juju-log Leader not ready yet, waiting... unit-mysql-k8s-0: 13:09:22 DEBUG unit.mysql-k8s/0.juju-log Deferring . unit-mysql-router-k8s-0: 13:09:22 INFO juju.worker.uniter.operation ran "leader-elected" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 13:09:23 INFO juju.worker.uniter.operation ran "mysql-pebble-ready" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 13:09:23 DEBUG unit.mysql-router-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 13:09:23 DEBUG unit.mysql-router-k8s/0.juju-log Emitting Juju event mysql_router_pebble_ready. unit-mysql-router-k8s-0: 13:09:23 DEBUG unit.mysql-router-k8s/0.juju-log State of reconcile self.unit.is_leader()=True, isinstance(self.workload, workload.AuthenticatedWorkload)=False, self.database_requires.relation and self.database_requires.relation.is_breaking(event)=None, self.workload.container_ready=True, isinstance(event, ops.UpgradeCharmEvent)=False unit-mysql-router-k8s-0: 13:09:23 DEBUG unit.mysql-router-k8s/0.juju-log Set status to BlockedStatus('Missing relation: database') unit-mysql-k8s-0: 13:09:23 DEBUG unit.mysql-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 13:09:23 INFO juju.worker.uniter.operation ran "mysql-router-pebble-ready" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 13:09:23 DEBUG unit.mysql-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 13:09:23 DEBUG unit.mysql-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 13:09:23 DEBUG unit.mysql-k8s/0.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:23 DEBUG unit.mysql-k8s/0.juju-log Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:23 DEBUG unit.mysql-k8s/0.juju-log Re-emitting deferred event . unit-mysql-k8s-0: 13:09:24 DEBUG unit.mysql-k8s/0.juju-log Leader not ready yet, waiting... unit-mysql-k8s-0: 13:09:24 DEBUG unit.mysql-k8s/0.juju-log Deferring . unit-mysql-k8s-0: 13:09:24 DEBUG unit.mysql-k8s/0.juju-log Emitting Juju event database_storage_attached. unit-mysql-k8s-0: 13:09:24 INFO juju.worker.uniter.operation ran "database-storage-attached" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 13:09:24 DEBUG unit.mysql-router-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 13:09:24 DEBUG unit.mysql-router-k8s/0.juju-log Emitting Juju event config_changed. unit-mysql-router-k8s-0: 13:09:24 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 13:09:24 INFO juju.worker.uniter found queued "start" hook unit-mysql-k8s-0: 13:09:24 DEBUG unit.mysql-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 13:09:25 DEBUG unit.mysql-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 13:09:25 DEBUG unit.mysql-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 13:09:25 DEBUG unit.mysql-k8s/0.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:25 DEBUG unit.mysql-k8s/0.juju-log Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:25 DEBUG unit.mysql-k8s/0.juju-log Re-emitting deferred event . unit-mysql-k8s-0: 13:09:25 DEBUG unit.mysql-k8s/0.juju-log Leader not ready yet, waiting... unit-mysql-k8s-0: 13:09:25 DEBUG unit.mysql-k8s/0.juju-log Deferring . unit-mysql-k8s-0: 13:09:25 DEBUG unit.mysql-k8s/0.juju-log Emitting Juju event config_changed. unit-mysql-router-k8s-0: 13:09:25 DEBUG unit.mysql-router-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 13:09:25 INFO unit.mysql-router-k8s/0.juju-log Running legacy hooks/start. unit-mysql-k8s-0: 13:09:25 INFO juju.worker.uniter.operation ran "config-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 13:09:25 INFO juju.worker.uniter found queued "start" hook unit-mysql-router-k8s-0: 13:09:25 DEBUG unit.mysql-router-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 13:09:25 DEBUG unit.mysql-router-k8s/0.juju-log Charm called itself via hooks/start. unit-mysql-router-k8s-0: 13:09:25 DEBUG unit.mysql-router-k8s/0.juju-log Legacy hooks/start exited with status 0. unit-mysql-router-k8s-0: 13:09:25 DEBUG unit.mysql-router-k8s/0.juju-log Emitting Juju event start. unit-mysql-router-k8s-0: 13:09:25 DEBUG unit.mysql-router-k8s/0.juju-log State of reconcile self.unit.is_leader()=True, isinstance(self.workload, workload.AuthenticatedWorkload)=False, self.database_requires.relation and self.database_requires.relation.is_breaking(event)=None, self.workload.container_ready=True, isinstance(event, ops.UpgradeCharmEvent)=False unit-mysql-router-k8s-0: 13:09:25 DEBUG unit.mysql-router-k8s/0.juju-log Set status to BlockedStatus('Missing relation: database') unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 13:09:26 INFO unit.mysql-k8s/0.juju-log Running legacy hooks/start. unit-mysql-router-k8s-0: 13:09:26 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 13:09:26 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 13:09:26 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Emitting Juju event backend_database_relation_joined. unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log Charm called itself via hooks/start. unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log Legacy hooks/start exited with status 0. unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log Re-emitting deferred event . unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log connection.connect_tcp.started host='10.152.183.1' port=443 local_address=None timeout=None unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log connection.connect_tcp.complete return_value= unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log connection.start_tls.started ssl_context= server_hostname='10.152.183.1' timeout=None unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log connection.start_tls.complete return_value= unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log http11.send_request_headers.started request= unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log http11.send_request_headers.complete unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log http11.send_request_body.started request= unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log http11.send_request_body.complete unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log http11.receive_response_headers.started request= unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'0ad9f7ed-404b-4347-a76d-0e8d6e192bdf'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'd178a7e5-16a6-4c1d-b375-269b5c847538'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'6aafa5ee-718d-41e3-ac79-a7986543b90b'), (b'Date', b'Fri, 19 May 2023 13:09:26 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-k8s-0: 13:09:26 INFO unit.mysql-k8s/0.juju-log HTTP Request: GET https://10.152.183.1/api/v1/namespaces/foo3/pods/mysql-k8s-0 "HTTP/1.1 200 OK" unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log http11.receive_response_body.started request= unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log http11.receive_response_body.complete unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log http11.response_closed.started unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log http11.response_closed.complete unit-mysql-k8s-0: 13:09:26 INFO unit.mysql-k8s/0.juju-log Retrieving the total memory of the server unit-mysql-router-k8s-0: 13:09:26 INFO juju.worker.uniter.operation ran "backend-database-relation-joined" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 13:09:26 DEBUG unit.mysql-k8s/0.juju-log Initializing instance unit-mysql-router-k8s-0: 13:09:27 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 13:09:27 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Emitting Juju event backend_database_relation_changed. unit-mysql-router-k8s-0: 13:09:27 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 13:09:34 DEBUG unit.mysql-k8s/0.juju-log Adding pebble layer unit-mysql-k8s-0: 13:09:34 DEBUG unit.mysql-k8s/0.juju-log Safe stopping mysqld safe unit-mysql-k8s-0: 13:09:35 DEBUG unit.mysql-k8s/0.juju-log Waiting for instance to be ready unit-mysql-k8s-0: 13:09:40 INFO unit.mysql-k8s/0.juju-log Configuring instance unit-mysql-k8s-0: 13:09:40 DEBUG unit.mysql-k8s/0.juju-log Configuring users unit-mysql-k8s-0: 13:09:40 DEBUG unit.mysql-k8s/0.juju-log Configuring instance for InnoDB on mysql-k8s-0.mysql-k8s-endpoints unit-mysql-k8s-0: 13:09:47 DEBUG unit.mysql-k8s/0.juju-log Getting InnoDB version unit-mysql-k8s-0: 13:09:47 INFO unit.mysql-k8s/0.juju-log Creating cluster on the leader unit unit-mysql-k8s-0: 13:09:47 DEBUG unit.mysql-k8s/0.juju-log Creating a MySQL InnoDB cluster on mysql-k8s-0.mysql-k8s-endpoints unit-mysql-k8s-0: 13:09:50 DEBUG unit.mysql-k8s/0.juju-log Emitting Juju event start. unit-mysql-k8s-0: 13:09:50 DEBUG unit.mysql-k8s/0.juju-log Alert rules path does not exist: src/prometheus_alert_rules unit-mysql-k8s-0: 13:09:50 INFO juju.worker.uniter.operation ran "start" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: Emitting Juju event database_relation_changed. unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: Emitting custom event . unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: Getting InnoDB version unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: Updating pod labels unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: connection.connect_tcp.started host='10.152.183.1' port=443 local_address=None timeout=None unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: connection.connect_tcp.complete return_value= unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: connection.start_tls.started ssl_context= server_hostname='10.152.183.1' timeout=None unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: connection.start_tls.complete return_value= unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.started request= unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.complete unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.started request= unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.complete unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.started request= unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'0a93a27e-00a6-42fd-88cf-8a861aacae54'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'd178a7e5-16a6-4c1d-b375-269b5c847538'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'6aafa5ee-718d-41e3-ac79-a7986543b90b'), (b'Date', b'Fri, 19 May 2023 13:09:51 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-k8s-0: 13:09:51 INFO unit.mysql-k8s/0.juju-log database:2: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/foo3/pods/mysql-k8s-0 "HTTP/1.1 200 OK" unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.started request= unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.complete unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.started unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.complete unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.started request= unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.complete unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.started request= unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.complete unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.started request= unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'bc256b01-c726-4243-96f9-ae06992c8c2f'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'd178a7e5-16a6-4c1d-b375-269b5c847538'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'6aafa5ee-718d-41e3-ac79-a7986543b90b'), (b'Date', b'Fri, 19 May 2023 13:09:51 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-k8s-0: 13:09:51 INFO unit.mysql-k8s/0.juju-log database:2: HTTP Request: PATCH https://10.152.183.1/api/v1/namespaces/foo3/pods/mysql-k8s-0 "HTTP/1.1 200 OK" unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.started request= unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.complete unit-mysql-k8s-0: 13:09:51 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.started unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.complete unit-mysql-k8s-0: 13:09:52 INFO unit.mysql-k8s/0.juju-log database:2: Kubernetes pod label primary created unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.started request= unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.complete unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.started request= unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.complete unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.started request= unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'adc3207f-05c4-4231-b1e6-aa189f8d544d'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'd178a7e5-16a6-4c1d-b375-269b5c847538'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'6aafa5ee-718d-41e3-ac79-a7986543b90b'), (b'Date', b'Fri, 19 May 2023 13:09:52 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-k8s-0: 13:09:52 INFO unit.mysql-k8s/0.juju-log database:2: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/foo3/pods/mysql-k8s-0 "HTTP/1.1 200 OK" unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.started request= unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.complete unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.started unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.complete unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.started request= unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.complete unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.started request= unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.complete unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.started request= unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 201, b'Created', [(b'Audit-Id', b'6cd88e7b-50f0-4834-9b18-02046a71b7e0'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'd178a7e5-16a6-4c1d-b375-269b5c847538'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'6aafa5ee-718d-41e3-ac79-a7986543b90b'), (b'Date', b'Fri, 19 May 2023 13:09:52 GMT'), (b'Content-Length', b'1268')]) unit-mysql-k8s-0: 13:09:52 INFO unit.mysql-k8s/0.juju-log database:2: HTTP Request: POST https://10.152.183.1/api/v1/namespaces/foo3/services "HTTP/1.1 201 Created" unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.started request= unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.complete unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.started unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.complete unit-mysql-k8s-0: 13:09:52 INFO unit.mysql-k8s/0.juju-log database:2: Kubernetes service mysql-k8s-primary created unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.started request= unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.complete unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.started request= unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.complete unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.started request= unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'02cb496c-ee03-4742-8059-298ea8ecc585'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'd178a7e5-16a6-4c1d-b375-269b5c847538'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'6aafa5ee-718d-41e3-ac79-a7986543b90b'), (b'Date', b'Fri, 19 May 2023 13:09:52 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-k8s-0: 13:09:52 INFO unit.mysql-k8s/0.juju-log database:2: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/foo3/pods/mysql-k8s-0 "HTTP/1.1 200 OK" unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.started request= unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.complete unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.started unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.complete unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.started request= unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.complete unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.started request= unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.complete unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.started request= unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 201, b'Created', [(b'Audit-Id', b'444e96ed-31c4-4586-aa62-a1a3c690dd9a'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'd178a7e5-16a6-4c1d-b375-269b5c847538'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'6aafa5ee-718d-41e3-ac79-a7986543b90b'), (b'Date', b'Fri, 19 May 2023 13:09:52 GMT'), (b'Content-Length', b'1270')]) unit-mysql-k8s-0: 13:09:52 INFO unit.mysql-k8s/0.juju-log database:2: HTTP Request: POST https://10.152.183.1/api/v1/namespaces/foo3/services "HTTP/1.1 201 Created" unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.started request= unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.complete unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.started unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.complete unit-mysql-k8s-0: 13:09:52 INFO unit.mysql-k8s/0.juju-log database:2: Kubernetes service mysql-k8s-replicas created unit-mysql-k8s-0: 13:09:52 DEBUG unit.mysql-k8s/0.juju-log database:2: Kubernetes service endpoint not ready yet unit-mysql-k8s-0: 13:09:53 DEBUG unit.mysql-k8s/0.juju-log database:2: Kubernetes service endpoint ready unit-mysql-k8s-0: 13:09:53 DEBUG unit.mysql-k8s/0.juju-log database:2: Getting cluster primary member's address from mysql-k8s-0.mysql-k8s-endpoints unit-mysql-k8s-0: 13:09:54 DEBUG unit.mysql-k8s/0.juju-log database:2: Getting cluster primary member's address from mysql-k8s-0.mysql-k8s-endpoints unit-mysql-k8s-0: 13:09:54 INFO unit.mysql-k8s/0.juju-log database:2: Created user for app mysql-router-k8s unit-mysql-k8s-0: 13:09:55 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 13:09:55 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 13:09:55 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Emitting Juju event backend_database_relation_changed. unit-mysql-router-k8s-0: 13:09:55 INFO unit.mysql-router-k8s/0.juju-log backend-database:2: database created at 2023-05-19 13:09:55.570090 unit-mysql-k8s-0: 13:09:55 DEBUG unit.mysql-k8s/0.juju-log database:2: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 13:09:55 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Emitting custom event . unit-mysql-router-k8s-0: 13:09:55 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: State of reconcile self.unit.is_leader()=True, isinstance(self.workload, workload.AuthenticatedWorkload)=True, self.database_requires.relation and self.database_requires.relation.is_breaking(event)=False, self.workload.container_ready=True, isinstance(event, ops.UpgradeCharmEvent)=False unit-mysql-router-k8s-0: 13:09:55 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Reconciling users event=, router_endpoint='mysql-router-k8s.foo3.svc.cluster.local' unit-mysql-k8s-0: 13:09:55 DEBUG unit.mysql-k8s/0.juju-log database:2: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-router-k8s-0: 13:09:55 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: State of reconcile users requested_users=[], created_users=[] unit-mysql-router-k8s-0: 13:09:55 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Reconciled users event=, router_endpoint='mysql-router-k8s.foo3.svc.cluster.local' unit-mysql-k8s-0: 13:09:55 DEBUG unit.mysql-k8s/0.juju-log database:2: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 13:09:55 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-router-k8s-0: 13:09:55 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Enabling MySQL Router service unit-mysql-k8s-0: 13:09:55 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-router-k8s-0: 13:09:55 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Bootstrapping router tls=False, self._database_requires_relation.host='mysql-k8s-primary.foo3.svc.cluster.local', self._database_requires_relation.port='3306' unit-mysql-k8s-0: 13:09:55 DEBUG unit.mysql-k8s/0.juju-log database:2: Emitting Juju event database_relation_joined. unit-mysql-k8s-0: 13:09:55 INFO juju.worker.uniter.operation ran "database-relation-joined" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 13:09:56 DEBUG unit.mysql-k8s/0.juju-log database:2: Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 13:09:56 DEBUG unit.mysql-k8s/0.juju-log database:2: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 13:09:56 DEBUG unit.mysql-k8s/0.juju-log database:2: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 13:09:56 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:56 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:56 DEBUG unit.mysql-k8s/0.juju-log database:2: Emitting Juju event database_relation_changed. unit-mysql-k8s-0: 13:09:56 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 13:09:57 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 13:09:57 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 13:09:57 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 13:09:57 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:57 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:57 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Emitting Juju event database_peers_relation_changed. unit-mysql-k8s-0: 13:09:57 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: Updating pod labels unit-mysql-router-k8s-0: 13:09:57 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Bootstrapped router tls=False, self._database_requires_relation.host='mysql-k8s-primary.foo3.svc.cluster.local', self._database_requires_relation.port='3306' unit-mysql-router-k8s-0: 13:09:57 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Adding attributes='{"created_by_user": "relation-2", "router_id": "mysql-router-k8s-0.mysql-router-k8s-endpoints.foo3.svc.cluster.local::system", "created_by_juju_unit": "mysql-router-k8s/0"}' to username='mysql_router1_nx6cpoupug9s' unit-mysql-k8s-0: 13:09:57 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: connection.connect_tcp.started host='10.152.183.1' port=443 local_address=None timeout=None unit-mysql-k8s-0: 13:09:57 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: connection.connect_tcp.complete return_value= unit-mysql-k8s-0: 13:09:57 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: connection.start_tls.started ssl_context= server_hostname='10.152.183.1' timeout=None unit-mysql-k8s-0: 13:09:57 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: connection.start_tls.complete return_value= unit-mysql-k8s-0: 13:09:57 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_headers.started request= unit-mysql-k8s-0: 13:09:57 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_headers.complete unit-mysql-k8s-0: 13:09:57 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_body.started request= unit-mysql-k8s-0: 13:09:57 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_body.complete unit-mysql-k8s-0: 13:09:58 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_headers.started request= unit-mysql-k8s-0: 13:09:58 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'06564560-c0c4-448a-8a48-5ff434691fec'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'd178a7e5-16a6-4c1d-b375-269b5c847538'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'6aafa5ee-718d-41e3-ac79-a7986543b90b'), (b'Date', b'Fri, 19 May 2023 13:09:57 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-k8s-0: 13:09:58 INFO unit.mysql-k8s/0.juju-log database-peers:0: HTTP Request: GET https://10.152.183.1/api/v1/namespaces/foo3/pods/mysql-k8s-0 "HTTP/1.1 200 OK" unit-mysql-k8s-0: 13:09:58 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_body.started request= unit-mysql-k8s-0: 13:09:58 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_body.complete unit-mysql-k8s-0: 13:09:58 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.response_closed.started unit-mysql-k8s-0: 13:09:58 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.response_closed.complete unit-mysql-k8s-0: 13:09:58 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_headers.started request= unit-mysql-k8s-0: 13:09:58 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_headers.complete unit-mysql-k8s-0: 13:09:58 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_body.started request= unit-mysql-k8s-0: 13:09:58 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.send_request_body.complete unit-mysql-k8s-0: 13:09:58 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_headers.started request= unit-mysql-k8s-0: 13:09:58 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'977e39c5-1262-4bc9-81c4-13cc27bb5a54'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'd178a7e5-16a6-4c1d-b375-269b5c847538'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'6aafa5ee-718d-41e3-ac79-a7986543b90b'), (b'Date', b'Fri, 19 May 2023 13:09:58 GMT'), (b'Transfer-Encoding', b'chunked')]) unit-mysql-k8s-0: 13:09:58 INFO unit.mysql-k8s/0.juju-log database-peers:0: HTTP Request: PATCH https://10.152.183.1/api/v1/namespaces/foo3/pods/mysql-k8s-0 "HTTP/1.1 200 OK" unit-mysql-k8s-0: 13:09:58 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_body.started request= unit-mysql-k8s-0: 13:09:58 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.receive_response_body.complete unit-mysql-k8s-0: 13:09:58 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.response_closed.started unit-mysql-k8s-0: 13:09:58 DEBUG unit.mysql-k8s/0.juju-log database-peers:0: http11.response_closed.complete unit-mysql-k8s-0: 13:09:58 INFO unit.mysql-k8s/0.juju-log database-peers:0: Kubernetes pod label primary created unit-mysql-k8s-0: 13:09:58 INFO juju.worker.uniter.operation ran "database-peers-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 13:09:58 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Added attributes='{"created_by_user": "relation-2", "router_id": "mysql-router-k8s-0.mysql-router-k8s-endpoints.foo3.svc.cluster.local::system", "created_by_juju_unit": "mysql-router-k8s/0"}' to username='mysql_router1_nx6cpoupug9s' unit-mysql-router-k8s-0: 13:09:58 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Enabled MySQL Router service unit-mysql-router-k8s-0: 13:09:58 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Waiting until MySQL Router is ready unit-mysql-router-k8s-0: 13:09:58 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: MySQL Router is ready unit-mysql-router-k8s-0: 13:09:58 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Set status to BlockedStatus('Missing relation: database') unit-mysql-router-k8s-0: 13:09:59 INFO juju.worker.uniter.operation ran "backend-database-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 13:09:59 DEBUG unit.mysql-k8s/0.juju-log database:2: Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 13:09:59 DEBUG unit.mysql-k8s/0.juju-log database:2: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 13:09:59 DEBUG unit.mysql-k8s/0.juju-log database:2: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 13:09:59 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:59 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 13:09:59 DEBUG unit.mysql-k8s/0.juju-log database:2: Emitting Juju event database_relation_changed. unit-mysql-k8s-0: 13:10:00 INFO juju.worker.uniter.operation ran "database-relation-changed" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 13:10:27 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 13:10:27 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Emitting Juju event backend_database_relation_departed. unit-mysql-k8s-0: 13:10:27 DEBUG unit.mysql-k8s/0.juju-log database:2: Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 13:10:27 DEBUG unit.mysql-k8s/0.juju-log database:2: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 13:10:27 DEBUG unit.mysql-k8s/0.juju-log database:2: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 13:10:27 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 13:10:27 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 13:10:27 DEBUG unit.mysql-k8s/0.juju-log database:2: Emitting Juju event database_relation_departed. unit-mysql-router-k8s-0: 13:10:28 INFO juju.worker.uniter.operation ran "backend-database-relation-departed" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 13:10:28 DEBUG unit.mysql-k8s/0.juju-log database:2: Getting cluster primary member's address from mysql-k8s-0.mysql-k8s-endpoints unit-mysql-router-k8s-0: 13:10:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Operator Framework 2.2.0 up and running. unit-mysql-router-k8s-0: 13:10:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Emitting Juju event backend_database_relation_broken. unit-mysql-router-k8s-0: 13:10:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: State of reconcile self.unit.is_leader()=True, isinstance(self.workload, workload.AuthenticatedWorkload)=True, self.database_requires.relation and self.database_requires.relation.is_breaking(event)=True, self.workload.container_ready=True, isinstance(event, ops.UpgradeCharmEvent)=False unit-mysql-router-k8s-0: 13:10:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Reconciling users event=, router_endpoint='mysql-router-k8s.foo3.svc.cluster.local' unit-mysql-router-k8s-0: 13:10:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: State of reconcile users requested_users=[], created_users=[] unit-mysql-router-k8s-0: 13:10:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Reconciled users event=, router_endpoint='mysql-router-k8s.foo3.svc.cluster.local' unit-mysql-router-k8s-0: 13:10:28 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Disabling MySQL Router service unit-mysql-k8s-0: 13:10:29 INFO unit.mysql-k8s/0.juju-log database:2: Deleted router user mysql_router1_nx6cpoupug9s unit-mysql-k8s-0: 13:10:29 DEBUG unit.mysql-k8s/0.juju-log database:2: Getting cluster primary member's address from mysql-k8s-0.mysql-k8s-endpoints unit-mysql-k8s-0: 13:10:29 INFO unit.mysql-k8s/0.juju-log database:2: Removed router from metadata mysql-router-k8s-0.mysql-router-k8s-endpoints.foo3.svc.cluster.local::system unit-mysql-router-k8s-0: 13:10:29 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Disabled MySQL Router service unit-mysql-router-k8s-0: 13:10:29 DEBUG unit.mysql-router-k8s/0.juju-log backend-database:2: Set status to BlockedStatus('Missing relations: backend-database, database') unit-mysql-k8s-0: 13:10:29 INFO juju.worker.uniter.operation ran "database-relation-departed" hook (via hook dispatching script: dispatch) unit-mysql-router-k8s-0: 13:10:30 INFO juju.worker.uniter.operation ran "backend-database-relation-broken" hook (via hook dispatching script: dispatch) unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: Operator Framework 2.2.0 up and running. unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: load_ssl_context verify='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' cert=None trust_env=True http2=False unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: load_verify_locations cafile='/var/run/secrets/kubernetes.io/serviceaccount/ca.crt' unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Prometheus alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/prometheus_alert_rules: directory does not exist unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: Invalid Loki alert rules folder at /var/lib/juju/agents/unit-mysql-k8s-0/charm/src/loki_alert_rules: directory does not exist unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: Emitting Juju event database_relation_broken. unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: connection.connect_tcp.started host='10.152.183.1' port=443 local_address=None timeout=None unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: connection.connect_tcp.complete return_value= unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: connection.start_tls.started ssl_context= server_hostname='10.152.183.1' timeout=None unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: connection.start_tls.complete return_value= unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.started request= unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.complete unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.started request= unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.complete unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.started request= unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'5db102a9-b0bf-40c9-9610-721411f8d403'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'd178a7e5-16a6-4c1d-b375-269b5c847538'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'6aafa5ee-718d-41e3-ac79-a7986543b90b'), (b'Date', b'Fri, 19 May 2023 13:10:30 GMT'), (b'Content-Length', b'1268')]) unit-mysql-k8s-0: 13:10:30 INFO unit.mysql-k8s/0.juju-log database:2: HTTP Request: DELETE https://10.152.183.1/api/v1/namespaces/foo3/services/mysql-k8s-primary "HTTP/1.1 200 OK" unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.started request= unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.complete unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.started unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.complete unit-mysql-k8s-0: 13:10:30 INFO unit.mysql-k8s/0.juju-log database:2: Kubernetes service mysql-k8s-primary deleted unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.started request= unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_headers.complete unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.started request= unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.send_request_body.complete unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.started request= unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Audit-Id', b'48f3ebd6-02a3-453f-b2b4-25b82c330f49'), (b'Cache-Control', b'no-cache, private'), (b'Content-Type', b'application/json'), (b'X-Kubernetes-Pf-Flowschema-Uid', b'd178a7e5-16a6-4c1d-b375-269b5c847538'), (b'X-Kubernetes-Pf-Prioritylevel-Uid', b'6aafa5ee-718d-41e3-ac79-a7986543b90b'), (b'Date', b'Fri, 19 May 2023 13:10:30 GMT'), (b'Content-Length', b'1270')]) unit-mysql-k8s-0: 13:10:30 INFO unit.mysql-k8s/0.juju-log database:2: HTTP Request: DELETE https://10.152.183.1/api/v1/namespaces/foo3/services/mysql-k8s-replicas "HTTP/1.1 200 OK" unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.started request= unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.receive_response_body.complete unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.started unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: http11.response_closed.complete unit-mysql-k8s-0: 13:10:30 INFO unit.mysql-k8s/0.juju-log database:2: Kubernetes service mysql-k8s-replicas deleted unit-mysql-k8s-0: 13:10:30 DEBUG unit.mysql-k8s/0.juju-log database:2: Getting cluster primary member's address from mysql-k8s-0.mysql-k8s-endpoints unit-mysql-k8s-0: 13:10:31 ERROR unit.mysql-k8s/0.juju-log database:2: Failed to delete users for relation 2 Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/src/mysql_k8s_helpers.py", line 658, in _run_mysqlsh_script stdout, _ = process.wait_output() File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/venv/ops/pebble.py", line 1252, in wait_output raise ExecError(self._command, exit_code, out_value, err_value) ops.pebble.ExecError: non-zero exit code 1 executing ['/usr/bin/mysqlsh', '--no-wizard', '--python', '--verbose=1', '-f', '/tmp/script.py', ';', 'rm', '/tmp/script.py'], stdout='', stderr='Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory\nverbose: 2023-05-19T13:10:31Z: Loading startup files...\nverbose: 2023-05-19T13:10:31Z: Loading plugins...\nverbose: 2023-05-19T13:10:31Z: Connecting to MySQL at: serverconfig@mysql-k8s-0.mysql-k8s-endpoints.foo3.svc.cluster.local:3306\nverbose: 2023-05-19T13:10:31Z: Shell.connect: tid=87: CONNECTED: mysql-k8s-0.mysql-k8s-endpoints.foo3.svc.cluster.local:3306\nverbose: 2023-05-19T13:10:31Z: Now tracking \'sql_mode\' system variable.\nverbose: 2023-05-19T13:10:31Z: ClassicSession.run_sql: tid=87: MySQL Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near \'NULL\' at line 1, SQL: PREPARE stmt FROM @sql\nTraceback (most recent call last):\n File "", line 4, in \nmysqlsh.DBError: MySQL Error (1064): ClassicSession.run_sql: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax ' [truncated] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/lib/charms/mysql/v0/mysql.py", line 626, in delete_users_for_relation self._run_mysqlsh_script("\n".join(drop_users_command)) File "/var/lib/juju/agents/unit-mysql-k8s-0/charm/src/mysql_k8s_helpers.py", line 661, in _run_mysqlsh_script raise MySQLClientError(e.stderr) charms.mysql.v0.mysql.MySQLClientError: Cannot set LC_ALL to locale en_US.UTF-8: No such file or directory verbose: 2023-05-19T13:10:31Z: Loading startup files... verbose: 2023-05-19T13:10:31Z: Loading plugins... verbose: 2023-05-19T13:10:31Z: Connecting to MySQL at: serverconfig@mysql-k8s-0.mysql-k8s-endpoints.foo3.svc.cluster.local:3306 verbose: 2023-05-19T13:10:31Z: Shell.connect: tid=87: CONNECTED: mysql-k8s-0.mysql-k8s-endpoints.foo3.svc.cluster.local:3306 verbose: 2023-05-19T13:10:31Z: Now tracking 'sql_mode' system variable. verbose: 2023-05-19T13:10:31Z: ClassicSession.run_sql: tid=87: MySQL Error 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'NULL' at line 1, SQL: PREPARE stmt FROM @sql Traceback (most recent call last): File "", line 4, in mysqlsh.DBError: MySQL Error (1064): ClassicSession.run_sql: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'NULL' at line 1 unit-mysql-k8s-0: 13:10:31 ERROR unit.mysql-k8s/0.juju-log database:2: Failed to delete user(s) for relation 2 unit-mysql-k8s-0: 13:10:31 INFO juju.worker.uniter.operation ran "database-relation-broken" hook (via hook dispatching script: dispatch) ```
github-actions[bot] commented 1 year ago

https://warthogs.atlassian.net/browse/DPE-1984