search

canonical / mysql-k8s-operator

A Charmed Operator for running MySQL on Kubernetes
https://charmhub.io/mysql-k8s
Apache License 2.0
8 stars 15 forks source link

cannot stat /etc/logrotate.d/flush_mysql_logs when joining metrics-endpoint relation #384

Closed gboutry closed 3 months ago

gboutry commented 7 months ago

Steps to reproduce

  1. juju add-model cos;deploy cos...
  2. juju offer prometheus:metrics-endpoint pm-me
  3. juju add-model squirrel
  4. juju deploy mysql --channel 8.0/stable
  5. juju integrate mysql admin/squirrel.pm-me

Expected behavior

Does not error out

Actual behavior

Failure during relation-join: charms.mysql.v0.mysql.MySQLExecError: error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory

Versions

Operating system: Ubuntu 22.04.4 LTS

Juju CLI: 3.2.4-genericlinux-amd64

Juju agent: 3.2.4

Charm revision: 113

microk8s: MicroK8s v1.28.7 revision 6532

Log output

Juju debug log: https://pastebin.canonical.com/p/rfP7thNKVN/

2024-02-26T17:52:14.213Z [container-agent] 2024-02-26 17:52:14 INFO juju.worker.uniter.operation runhook.go:186 ran "update-status" hook (via hook dispatching script: dispatch)
2024-02-26T17:57:10.836Z [container-agent] 2024-02-26 17:57:10 INFO juju-log Unit workload member-state is online with member-role secondary
2024-02-26T17:57:13.032Z [container-agent] 2024-02-26 17:57:13 WARNING juju-log No relation: certificates
2024-02-26T17:57:13.342Z [container-agent] 2024-02-26 17:57:13 INFO juju.worker.uniter.operation runhook.go:186 ran "update-status" hook (via hook dispatching script: dispatch)
2024-02-26T17:57:33.136Z [container-agent] 2024-02-26 17:57:33 INFO juju.worker.uniter.operation runhook.go:186 ran "grafana-dashboard-relation-created" hook (via hook dispatching script: dispatch)
2024-02-26T17:57:34.264Z [container-agent] 2024-02-26 17:57:34 INFO juju-log metrics-endpoint:266: Adding pebble layer
2024-02-26T17:57:39.680Z [container-agent] 2024-02-26 17:57:39 ERROR juju-log metrics-endpoint:266: Uncaught exception while in charm code:
2024-02-26T17:57:39.680Z [container-agent] Traceback (most recent call last):
2024-02-26T17:57:39.680Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/./src/charm.py", line 786, in <module>
2024-02-26T17:57:39.680Z [container-agent]     main(MySQLOperatorCharm)
2024-02-26T17:57:39.680Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/main.py", line 436, in main
2024-02-26T17:57:39.680Z [container-agent]     _emit_charm_event(charm, dispatcher.event_name)
2024-02-26T17:57:39.680Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/main.py", line 144, in _emit_charm_event
2024-02-26T17:57:39.680Z [container-agent]     event_to_emit.emit(*args, **kwargs)
2024-02-26T17:57:39.680Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/framework.py", line 351, in emit
2024-02-26T17:57:39.680Z [container-agent]     framework._emit(event)
2024-02-26T17:57:39.680Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/framework.py", line 853, in _emit
2024-02-26T17:57:39.680Z [container-agent]     self._reemit(event_path)
2024-02-26T17:57:39.680Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/framework.py", line 942, in _reemit
2024-02-26T17:57:39.680Z [container-agent]     custom_handler(event)
2024-02-26T17:57:39.680Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/./src/charm.py", line 393, in _reconcile_mysqld_exporter
2024-02-26T17:57:39.680Z [container-agent]     self._reconcile_pebble_layer(container)
2024-02-26T17:57:39.680Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/./src/charm.py", line 344, in _reconcile_pebble_layer
2024-02-26T17:57:39.680Z [container-agent]     container.replan()
2024-02-26T17:57:39.680Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/model.py", line 2104, in replan
2024-02-26T17:57:39.680Z [container-agent]     self._pebble.replan_services()
2024-02-26T17:57:39.680Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1782, in replan_services
2024-02-26T17:57:39.680Z [container-agent]     return self._services_action('replan', [], timeout, delay)
2024-02-26T17:57:39.680Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1864, in _services_action
2024-02-26T17:57:39.680Z [container-agent]     change = self.wait_change(change_id, timeout=timeout, delay=delay)
2024-02-26T17:57:39.680Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1893, in wait_change
2024-02-26T17:57:39.680Z [container-agent]     return self._wait_change_using_wait(change_id, timeout)
2024-02-26T17:57:39.680Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1914, in _wait_change_using_wait
2024-02-26T17:57:39.680Z [container-agent]     return self._wait_change(change_id, this_timeout)
2024-02-26T17:57:39.680Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1928, in _wait_change
2024-02-26T17:57:39.680Z [container-agent]     resp = self._request('GET', f'/v1/changes/{change_id}/wait', query)
2024-02-26T17:57:39.680Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1655, in _request
2024-02-26T17:57:39.680Z [container-agent]     response = self._request_raw(method, path, query, headers, data)
2024-02-26T17:57:39.680Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1690, in _request_raw
2024-02-26T17:57:39.680Z [container-agent]     response = self.opener.open(request, timeout=self.timeout)
2024-02-26T17:57:39.680Z [container-agent]   File "/usr/lib/python3.10/urllib/request.py", line 519, in open
2024-02-26T17:57:39.680Z [container-agent]     response = self._open(req, data)
2024-02-26T17:57:39.680Z [container-agent]   File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
2024-02-26T17:57:39.680Z [container-agent]     result = self._call_chain(self.handle_open, protocol, protocol +
2024-02-26T17:57:39.680Z [container-agent]   File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
2024-02-26T17:57:39.680Z [container-agent]     result = func(*args)
2024-02-26T17:57:39.680Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 311, in http_open
2024-02-26T17:57:39.680Z [container-agent]     return self.do_open(_UnixSocketConnection, req,  # type:ignore
2024-02-26T17:57:39.680Z [container-agent]   File "/usr/lib/python3.10/urllib/request.py", line 1352, in do_open
2024-02-26T17:57:39.680Z [container-agent]     r = h.getresponse()
2024-02-26T17:57:39.680Z [container-agent]   File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
2024-02-26T17:57:39.680Z [container-agent]     response.begin()
2024-02-26T17:57:39.680Z [container-agent]   File "/usr/lib/python3.10/http/client.py", line 318, in begin
2024-02-26T17:57:39.680Z [container-agent]     version, status, reason = self._read_status()
2024-02-26T17:57:39.680Z [container-agent]   File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
2024-02-26T17:57:39.680Z [container-agent]     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2024-02-26T17:57:39.680Z [container-agent]   File "/usr/lib/python3.10/socket.py", line 705, in readinto
2024-02-26T17:57:39.680Z [container-agent]     return self._sock.recv_into(b)
2024-02-26T17:57:39.680Z [container-agent] TimeoutError: timed out
2024-02-26T17:57:39.970Z [container-agent] 2024-02-26 17:57:39 ERROR juju.worker.uniter.operation runhook.go:180 hook "metrics-endpoint-relation-created" (via hook dispatching script: dispatch) failed: exit status 1
2024-02-26T17:57:39.973Z [container-agent] 2024-02-26 17:57:39 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T17:57:43.551Z [container-agent] 2024-02-26 17:57:43 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T17:57:44.467Z [container-agent] 2024-02-26 17:57:44 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T17:57:45.082Z [container-agent] 2024-02-26 17:57:45 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T17:58:08.414Z [container-agent] 2024-02-26 17:58:08 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T17:58:18.234Z [container-agent] 2024-02-26 17:58:18 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T17:58:22.938Z [container-agent] 2024-02-26 17:58:22 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T17:58:23.894Z [container-agent] 2024-02-26 17:58:23 INFO juju-log metrics-endpoint:266: Adding pebble layer
2024-02-26T17:58:27.142Z [container-agent] 2024-02-26 17:58:27 ERROR juju-log metrics-endpoint:266: Uncaught exception while in charm code:
2024-02-26T17:58:27.142Z [container-agent] Traceback (most recent call last):
2024-02-26T17:58:27.142Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/./src/charm.py", line 786, in <module>
2024-02-26T17:58:27.142Z [container-agent]     main(MySQLOperatorCharm)
2024-02-26T17:58:27.142Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/main.py", line 436, in main
2024-02-26T17:58:27.142Z [container-agent]     _emit_charm_event(charm, dispatcher.event_name)
2024-02-26T17:58:27.142Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/main.py", line 144, in _emit_charm_event
2024-02-26T17:58:27.142Z [container-agent]     event_to_emit.emit(*args, **kwargs)
2024-02-26T17:58:27.142Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/framework.py", line 351, in emit
2024-02-26T17:58:27.142Z [container-agent]     framework._emit(event)
2024-02-26T17:58:27.142Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/framework.py", line 853, in _emit
2024-02-26T17:58:27.142Z [container-agent]     self._reemit(event_path)
2024-02-26T17:58:27.142Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/framework.py", line 942, in _reemit
2024-02-26T17:58:27.142Z [container-agent]     custom_handler(event)
2024-02-26T17:58:27.142Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/./src/charm.py", line 393, in _reconcile_mysqld_exporter
2024-02-26T17:58:27.142Z [container-agent]     self._reconcile_pebble_layer(container)
2024-02-26T17:58:27.142Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/./src/charm.py", line 344, in _reconcile_pebble_layer
2024-02-26T17:58:27.142Z [container-agent]     container.replan()
2024-02-26T17:58:27.142Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/model.py", line 2104, in replan
2024-02-26T17:58:27.142Z [container-agent]     self._pebble.replan_services()
2024-02-26T17:58:27.142Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1782, in replan_services
2024-02-26T17:58:27.142Z [container-agent]     return self._services_action('replan', [], timeout, delay)
2024-02-26T17:58:27.142Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1866, in _services_action
2024-02-26T17:58:27.142Z [container-agent]     raise ChangeError(change.err, change)
2024-02-26T17:58:27.142Z [container-agent] ops.pebble.ChangeError: cannot perform the following tasks:
2024-02-26T17:58:27.142Z [container-agent] - Start service "mysqld_safe" (cannot start service: exited quickly with code 1)
2024-02-26T17:58:27.142Z [container-agent] ----- Logs from task 0 -----
2024-02-26T17:58:27.142Z [container-agent] 2024-02-26T17:58:24Z INFO Service "mysqld_exporter" already started.
2024-02-26T17:58:27.142Z [container-agent] ----- Logs from task 1 -----
2024-02-26T17:58:27.142Z [container-agent] 2024-02-26T17:58:25Z INFO Most recent service output:
2024-02-26T17:58:27.142Z [container-agent]     2024-02-26T17:58:25.932632Z mysqld_safe Logging to '/var/log/mysql/error.log'.
2024-02-26T17:58:27.142Z [container-agent]     2024-02-26T17:58:25.949431Z mysqld_safe A mysqld process already exists
2024-02-26T17:58:27.142Z [container-agent] 2024-02-26T17:58:26Z ERROR cannot start service: exited quickly with code 1
2024-02-26T17:58:27.142Z [container-agent] -----
2024-02-26T17:58:27.442Z [container-agent] 2024-02-26 17:58:27 ERROR juju.worker.uniter.operation runhook.go:180 hook "metrics-endpoint-relation-created" (via hook dispatching script: dispatch) failed: exit status 1
2024-02-26T17:58:27.444Z [container-agent] 2024-02-26 17:58:27 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T17:58:32.752Z [container-agent] 2024-02-26 17:58:32 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T17:58:35.026Z [container-agent] 2024-02-26 17:58:35 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T17:58:42.653Z [container-agent] 2024-02-26 17:58:42 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T17:59:09.012Z [container-agent] 2024-02-26 17:59:09 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T17:59:55.304Z [container-agent] 2024-02-26 17:59:55 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T17:59:56.221Z [container-agent] 2024-02-26 17:59:56 INFO juju-log metrics-endpoint:266: Adding pebble layer
2024-02-26T17:59:59.421Z [container-agent] 2024-02-26 17:59:59 ERROR juju-log metrics-endpoint:266: Uncaught exception while in charm code:
2024-02-26T17:59:59.421Z [container-agent] Traceback (most recent call last):
2024-02-26T17:59:59.421Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/./src/charm.py", line 786, in <module>
2024-02-26T17:59:59.421Z [container-agent]     main(MySQLOperatorCharm)
2024-02-26T17:59:59.421Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/main.py", line 436, in main
2024-02-26T17:59:59.421Z [container-agent]     _emit_charm_event(charm, dispatcher.event_name)
2024-02-26T17:59:59.421Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/main.py", line 144, in _emit_charm_event
2024-02-26T17:59:59.421Z [container-agent]     event_to_emit.emit(*args, **kwargs)
2024-02-26T17:59:59.421Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/framework.py", line 351, in emit
2024-02-26T17:59:59.421Z [container-agent]     framework._emit(event)
2024-02-26T17:59:59.421Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/framework.py", line 853, in _emit
2024-02-26T17:59:59.421Z [container-agent]     self._reemit(event_path)
2024-02-26T17:59:59.421Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/framework.py", line 942, in _reemit
2024-02-26T17:59:59.421Z [container-agent]     custom_handler(event)
2024-02-26T17:59:59.421Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/./src/charm.py", line 393, in _reconcile_mysqld_exporter
2024-02-26T17:59:59.421Z [container-agent]     self._reconcile_pebble_layer(container)
2024-02-26T17:59:59.421Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/./src/charm.py", line 344, in _reconcile_pebble_layer
2024-02-26T17:59:59.421Z [container-agent]     container.replan()
2024-02-26T17:59:59.421Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/model.py", line 2104, in replan
2024-02-26T17:59:59.421Z [container-agent]     self._pebble.replan_services()
2024-02-26T17:59:59.421Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1782, in replan_services
2024-02-26T17:59:59.421Z [container-agent]     return self._services_action('replan', [], timeout, delay)
2024-02-26T17:59:59.421Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1866, in _services_action
2024-02-26T17:59:59.421Z [container-agent]     raise ChangeError(change.err, change)
2024-02-26T17:59:59.421Z [container-agent] ops.pebble.ChangeError: cannot perform the following tasks:
2024-02-26T17:59:59.421Z [container-agent] - Start service "mysqld_safe" (cannot start service: exited quickly with code 1)
2024-02-26T17:59:59.421Z [container-agent] ----- Logs from task 0 -----
2024-02-26T17:59:59.421Z [container-agent] 2024-02-26T17:59:56Z INFO Service "mysqld_exporter" already started.
2024-02-26T17:59:59.421Z [container-agent] ----- Logs from task 1 -----
2024-02-26T17:59:59.421Z [container-agent] 2024-02-26T17:59:58Z INFO Most recent service output:
2024-02-26T17:59:59.421Z [container-agent]     2024-02-26T17:59:58.267224Z mysqld_safe Logging to '/var/log/mysql/error.log'.
2024-02-26T17:59:59.421Z [container-agent]     2024-02-26T17:59:58.284154Z mysqld_safe A mysqld process already exists
2024-02-26T17:59:59.421Z [container-agent] 2024-02-26T17:59:58Z ERROR cannot start service: exited quickly with code 1
2024-02-26T17:59:59.421Z [container-agent] -----
2024-02-26T17:59:59.698Z [container-agent] 2024-02-26 17:59:59 ERROR juju.worker.uniter.operation runhook.go:180 hook "metrics-endpoint-relation-created" (via hook dispatching script: dispatch) failed: exit status 1
2024-02-26T17:59:59.700Z [container-agent] 2024-02-26 17:59:59 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:00:09.221Z [container-agent] 2024-02-26 18:00:09 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:01:08.861Z [container-agent] 2024-02-26 18:01:08 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:02:09.041Z [container-agent] 2024-02-26 18:02:09 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:02:33.675Z [container-agent] 2024-02-26 18:02:33 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:02:37.923Z [container-agent] 2024-02-26 18:02:37 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:02:38.563Z [container-agent] 2024-02-26 18:02:38 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:02:49.557Z [container-agent] 2024-02-26 18:02:49 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:02:57.555Z [container-agent] 2024-02-26 18:02:57 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:02:58.471Z [container-agent] 2024-02-26 18:02:58 INFO juju-log metrics-endpoint:266: Adding pebble layer
2024-02-26T18:03:32.540Z [container-agent] 2024-02-26 18:03:32 ERROR juju-log metrics-endpoint:266: Uncaught exception while in charm code:
2024-02-26T18:03:32.540Z [container-agent] Traceback (most recent call last):
2024-02-26T18:03:32.540Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/./src/charm.py", line 786, in <module>
2024-02-26T18:03:32.540Z [container-agent]     main(MySQLOperatorCharm)
2024-02-26T18:03:32.540Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/main.py", line 436, in main
2024-02-26T18:03:32.540Z [container-agent]     _emit_charm_event(charm, dispatcher.event_name)
2024-02-26T18:03:32.540Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/main.py", line 144, in _emit_charm_event
2024-02-26T18:03:32.540Z [container-agent]     event_to_emit.emit(*args, **kwargs)
2024-02-26T18:03:32.540Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/framework.py", line 351, in emit
2024-02-26T18:03:32.540Z [container-agent]     framework._emit(event)
2024-02-26T18:03:32.540Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/framework.py", line 853, in _emit
2024-02-26T18:03:32.540Z [container-agent]     self._reemit(event_path)
2024-02-26T18:03:32.540Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/framework.py", line 942, in _reemit
2024-02-26T18:03:32.540Z [container-agent]     custom_handler(event)
2024-02-26T18:03:32.540Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/./src/charm.py", line 393, in _reconcile_mysqld_exporter
2024-02-26T18:03:32.540Z [container-agent]     self._reconcile_pebble_layer(container)
2024-02-26T18:03:32.540Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/./src/charm.py", line 345, in _reconcile_pebble_layer
2024-02-26T18:03:32.540Z [container-agent]     self._mysql.wait_until_mysql_connection()
2024-02-26T18:03:32.540Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/tenacity/__init__.py", line 289, in wrapped_f
2024-02-26T18:03:32.540Z [container-agent]     return self(f, *args, **kw)
2024-02-26T18:03:32.540Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/tenacity/__init__.py", line 379, in __call__
2024-02-26T18:03:32.540Z [container-agent]     do = self.iter(retry_state=retry_state)
2024-02-26T18:03:32.540Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/tenacity/__init__.py", line 325, in iter
2024-02-26T18:03:32.540Z [container-agent]     raise retry_exc.reraise()
2024-02-26T18:03:32.540Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/tenacity/__init__.py", line 158, in reraise
2024-02-26T18:03:32.540Z [container-agent]     raise self.last_attempt.result()
2024-02-26T18:03:32.540Z [container-agent]   File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2024-02-26T18:03:32.540Z [container-agent]     return self.__get_result()
2024-02-26T18:03:32.540Z [container-agent]   File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2024-02-26T18:03:32.540Z [container-agent]     raise self._exception
2024-02-26T18:03:32.540Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/tenacity/__init__.py", line 382, in __call__
2024-02-26T18:03:32.540Z [container-agent]     result = fn(*args, **kwargs)
2024-02-26T18:03:32.540Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/mysql_k8s_helpers.py", line 237, in wait_until_mysql_connection
2024-02-26T18:03:32.540Z [container-agent]     raise MySQLServiceNotRunningError("Connection with mysqlsh not possible")
2024-02-26T18:03:32.540Z [container-agent] mysql_k8s_helpers.MySQLServiceNotRunningError: Connection with mysqlsh not possible
2024-02-26T18:03:32.829Z [container-agent] 2024-02-26 18:03:32 ERROR juju.worker.uniter.operation runhook.go:180 hook "metrics-endpoint-relation-created" (via hook dispatching script: dispatch) failed: exit status 1
2024-02-26T18:03:33.878Z [container-agent] 2024-02-26 18:03:33 ERROR juju-log Failed to rotate mysql logs
2024-02-26T18:03:33.878Z [container-agent] Traceback (most recent call last):
2024-02-26T18:03:33.878Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/mysql_k8s_helpers.py", line 628, in _execute_commands
2024-02-26T18:03:33.878Z [container-agent]     stdout, stderr = process.wait_output()
2024-02-26T18:03:33.878Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1441, in wait_output
2024-02-26T18:03:33.878Z [container-agent]     raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
2024-02-26T18:03:33.878Z [container-agent] ops.pebble.ExecError: non-zero exit code 1 executing ['logrotate', '-f', '/etc/logrotate.d/flush_mysql_logs'], stdout='', stderr='error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory\n'
2024-02-26T18:03:33.878Z [container-agent] 
2024-02-26T18:03:33.878Z [container-agent] During handling of the above exception, another exception occurred:
2024-02-26T18:03:33.878Z [container-agent] 
2024-02-26T18:03:33.878Z [container-agent] Traceback (most recent call last):
2024-02-26T18:03:33.878Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/rotate_mysql_logs.py", line 50, in _rotate_mysql_logs
2024-02-26T18:03:33.878Z [container-agent]     self.charm._mysql._execute_commands(["logrotate", "-f", LOG_ROTATE_CONFIG_FILE])
2024-02-26T18:03:33.878Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/mysql_k8s_helpers.py", line 632, in _execute_commands
2024-02-26T18:03:33.878Z [container-agent]     raise MySQLExecError(e.stderr)
2024-02-26T18:03:33.878Z [container-agent] charms.mysql.v0.mysql.MySQLExecError: error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory
2024-02-26T18:03:33.878Z [container-agent] 
2024-02-26T18:03:34.001Z [container-agent] 2024-02-26 18:03:34 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:04:01.090Z [container-agent] 2024-02-26 18:04:01 ERROR juju-log Failed to rotate mysql logs
2024-02-26T18:04:01.090Z [container-agent] Traceback (most recent call last):
2024-02-26T18:04:01.090Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/mysql_k8s_helpers.py", line 628, in _execute_commands
2024-02-26T18:04:01.090Z [container-agent]     stdout, stderr = process.wait_output()
2024-02-26T18:04:01.090Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1441, in wait_output
2024-02-26T18:04:01.090Z [container-agent]     raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
2024-02-26T18:04:01.090Z [container-agent] ops.pebble.ExecError: non-zero exit code 1 executing ['logrotate', '-f', '/etc/logrotate.d/flush_mysql_logs'], stdout='', stderr='error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory\n'
2024-02-26T18:04:01.090Z [container-agent] 
2024-02-26T18:04:01.090Z [container-agent] During handling of the above exception, another exception occurred:
2024-02-26T18:04:01.090Z [container-agent] 
2024-02-26T18:04:01.090Z [container-agent] Traceback (most recent call last):
2024-02-26T18:04:01.090Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/rotate_mysql_logs.py", line 50, in _rotate_mysql_logs
2024-02-26T18:04:01.090Z [container-agent]     self.charm._mysql._execute_commands(["logrotate", "-f", LOG_ROTATE_CONFIG_FILE])
2024-02-26T18:04:01.090Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/mysql_k8s_helpers.py", line 632, in _execute_commands
2024-02-26T18:04:01.090Z [container-agent]     raise MySQLExecError(e.stderr)
2024-02-26T18:04:01.090Z [container-agent] charms.mysql.v0.mysql.MySQLExecError: error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory
2024-02-26T18:04:01.090Z [container-agent] 
2024-02-26T18:04:01.184Z [container-agent] 2024-02-26 18:04:01 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:05:01.224Z [container-agent] 2024-02-26 18:05:01 ERROR juju-log Failed to rotate mysql logs
2024-02-26T18:05:01.224Z [container-agent] Traceback (most recent call last):
2024-02-26T18:05:01.224Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/mysql_k8s_helpers.py", line 628, in _execute_commands
2024-02-26T18:05:01.224Z [container-agent]     stdout, stderr = process.wait_output()
2024-02-26T18:05:01.224Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1441, in wait_output
2024-02-26T18:05:01.224Z [container-agent]     raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
2024-02-26T18:05:01.224Z [container-agent] ops.pebble.ExecError: non-zero exit code 1 executing ['logrotate', '-f', '/etc/logrotate.d/flush_mysql_logs'], stdout='', stderr='error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory\n'
2024-02-26T18:05:01.224Z [container-agent] 
2024-02-26T18:05:01.224Z [container-agent] During handling of the above exception, another exception occurred:
2024-02-26T18:05:01.224Z [container-agent] 
2024-02-26T18:05:01.224Z [container-agent] Traceback (most recent call last):
2024-02-26T18:05:01.224Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/rotate_mysql_logs.py", line 50, in _rotate_mysql_logs
2024-02-26T18:05:01.224Z [container-agent]     self.charm._mysql._execute_commands(["logrotate", "-f", LOG_ROTATE_CONFIG_FILE])
2024-02-26T18:05:01.224Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/mysql_k8s_helpers.py", line 632, in _execute_commands
2024-02-26T18:05:01.224Z [container-agent]     raise MySQLExecError(e.stderr)
2024-02-26T18:05:01.224Z [container-agent] charms.mysql.v0.mysql.MySQLExecError: error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory
2024-02-26T18:05:01.224Z [container-agent] 
2024-02-26T18:05:01.358Z [container-agent] 2024-02-26 18:05:01 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:06:01.233Z [container-agent] 2024-02-26 18:06:01 ERROR juju-log Failed to rotate mysql logs
2024-02-26T18:06:01.233Z [container-agent] Traceback (most recent call last):
2024-02-26T18:06:01.233Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/mysql_k8s_helpers.py", line 628, in _execute_commands
2024-02-26T18:06:01.233Z [container-agent]     stdout, stderr = process.wait_output()
2024-02-26T18:06:01.233Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1441, in wait_output
2024-02-26T18:06:01.233Z [container-agent]     raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
2024-02-26T18:06:01.233Z [container-agent] ops.pebble.ExecError: non-zero exit code 1 executing ['logrotate', '-f', '/etc/logrotate.d/flush_mysql_logs'], stdout='', stderr='error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory\n'
2024-02-26T18:06:01.233Z [container-agent] 
2024-02-26T18:06:01.233Z [container-agent] During handling of the above exception, another exception occurred:
2024-02-26T18:06:01.233Z [container-agent] 
2024-02-26T18:06:01.233Z [container-agent] Traceback (most recent call last):
2024-02-26T18:06:01.233Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/rotate_mysql_logs.py", line 50, in _rotate_mysql_logs
2024-02-26T18:06:01.233Z [container-agent]     self.charm._mysql._execute_commands(["logrotate", "-f", LOG_ROTATE_CONFIG_FILE])
2024-02-26T18:06:01.233Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/mysql_k8s_helpers.py", line 632, in _execute_commands
2024-02-26T18:06:01.233Z [container-agent]     raise MySQLExecError(e.stderr)
2024-02-26T18:06:01.233Z [container-agent] charms.mysql.v0.mysql.MySQLExecError: error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory
2024-02-26T18:06:01.233Z [container-agent] 
2024-02-26T18:06:01.359Z [container-agent] 2024-02-26 18:06:01 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:07:01.197Z [container-agent] 2024-02-26 18:07:01 ERROR juju-log Failed to rotate mysql logs
2024-02-26T18:07:01.197Z [container-agent] Traceback (most recent call last):
2024-02-26T18:07:01.197Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/mysql_k8s_helpers.py", line 628, in _execute_commands
2024-02-26T18:07:01.197Z [container-agent]     stdout, stderr = process.wait_output()
2024-02-26T18:07:01.197Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1441, in wait_output
2024-02-26T18:07:01.197Z [container-agent]     raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
2024-02-26T18:07:01.197Z [container-agent] ops.pebble.ExecError: non-zero exit code 1 executing ['logrotate', '-f', '/etc/logrotate.d/flush_mysql_logs'], stdout='', stderr='error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory\n'
2024-02-26T18:07:01.197Z [container-agent] 
2024-02-26T18:07:01.197Z [container-agent] During handling of the above exception, another exception occurred:
2024-02-26T18:07:01.197Z [container-agent] 
2024-02-26T18:07:01.197Z [container-agent] Traceback (most recent call last):
2024-02-26T18:07:01.197Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/rotate_mysql_logs.py", line 50, in _rotate_mysql_logs
2024-02-26T18:07:01.197Z [container-agent]     self.charm._mysql._execute_commands(["logrotate", "-f", LOG_ROTATE_CONFIG_FILE])
2024-02-26T18:07:01.197Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/mysql_k8s_helpers.py", line 632, in _execute_commands
2024-02-26T18:07:01.197Z [container-agent]     raise MySQLExecError(e.stderr)
2024-02-26T18:07:01.197Z [container-agent] charms.mysql.v0.mysql.MySQLExecError: error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory
2024-02-26T18:07:01.197Z [container-agent] 
2024-02-26T18:07:01.309Z [container-agent] 2024-02-26 18:07:01 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:08:01.252Z [container-agent] 2024-02-26 18:08:01 ERROR juju-log Failed to rotate mysql logs
2024-02-26T18:08:01.252Z [container-agent] Traceback (most recent call last):
2024-02-26T18:08:01.252Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/mysql_k8s_helpers.py", line 628, in _execute_commands
2024-02-26T18:08:01.252Z [container-agent]     stdout, stderr = process.wait_output()
2024-02-26T18:08:01.252Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1441, in wait_output
2024-02-26T18:08:01.252Z [container-agent]     raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
2024-02-26T18:08:01.252Z [container-agent] ops.pebble.ExecError: non-zero exit code 1 executing ['logrotate', '-f', '/etc/logrotate.d/flush_mysql_logs'], stdout='', stderr='error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory\n'
2024-02-26T18:08:01.252Z [container-agent] 
2024-02-26T18:08:01.252Z [container-agent] During handling of the above exception, another exception occurred:
2024-02-26T18:08:01.252Z [container-agent] 
2024-02-26T18:08:01.252Z [container-agent] Traceback (most recent call last):
2024-02-26T18:08:01.252Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/rotate_mysql_logs.py", line 50, in _rotate_mysql_logs
2024-02-26T18:08:01.252Z [container-agent]     self.charm._mysql._execute_commands(["logrotate", "-f", LOG_ROTATE_CONFIG_FILE])
2024-02-26T18:08:01.252Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/mysql_k8s_helpers.py", line 632, in _execute_commands
2024-02-26T18:08:01.252Z [container-agent]     raise MySQLExecError(e.stderr)
2024-02-26T18:08:01.252Z [container-agent] charms.mysql.v0.mysql.MySQLExecError: error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory
2024-02-26T18:08:01.252Z [container-agent] 
2024-02-26T18:08:01.390Z [container-agent] 2024-02-26 18:08:01 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:08:19.296Z [container-agent] 2024-02-26 18:08:19 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:08:26.593Z [container-agent] 2024-02-26 18:08:26 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:08:34.017Z [container-agent] 2024-02-26 18:08:34 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:08:35.014Z [container-agent] 2024-02-26 18:08:35 INFO juju-log metrics-endpoint:266: Adding pebble layer
2024-02-26T18:09:07.788Z [container-agent] 2024-02-26 18:09:07 ERROR juju-log metrics-endpoint:266: Uncaught exception while in charm code:
2024-02-26T18:09:07.788Z [container-agent] Traceback (most recent call last):
2024-02-26T18:09:07.788Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/./src/charm.py", line 786, in <module>
2024-02-26T18:09:07.788Z [container-agent]     main(MySQLOperatorCharm)
2024-02-26T18:09:07.788Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/main.py", line 436, in main
2024-02-26T18:09:07.788Z [container-agent]     _emit_charm_event(charm, dispatcher.event_name)
2024-02-26T18:09:07.788Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/main.py", line 144, in _emit_charm_event
2024-02-26T18:09:07.788Z [container-agent]     event_to_emit.emit(*args, **kwargs)
2024-02-26T18:09:07.788Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/framework.py", line 351, in emit
2024-02-26T18:09:07.788Z [container-agent]     framework._emit(event)
2024-02-26T18:09:07.788Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/framework.py", line 853, in _emit
2024-02-26T18:09:07.788Z [container-agent]     self._reemit(event_path)
2024-02-26T18:09:07.788Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/framework.py", line 942, in _reemit
2024-02-26T18:09:07.788Z [container-agent]     custom_handler(event)
2024-02-26T18:09:07.788Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/./src/charm.py", line 393, in _reconcile_mysqld_exporter
2024-02-26T18:09:07.788Z [container-agent]     self._reconcile_pebble_layer(container)
2024-02-26T18:09:07.788Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/./src/charm.py", line 345, in _reconcile_pebble_layer
2024-02-26T18:09:07.788Z [container-agent]     self._mysql.wait_until_mysql_connection()
2024-02-26T18:09:07.788Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/tenacity/__init__.py", line 289, in wrapped_f
2024-02-26T18:09:07.788Z [container-agent]     return self(f, *args, **kw)
2024-02-26T18:09:07.788Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/tenacity/__init__.py", line 379, in __call__
2024-02-26T18:09:07.788Z [container-agent]     do = self.iter(retry_state=retry_state)
2024-02-26T18:09:07.788Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/tenacity/__init__.py", line 325, in iter
2024-02-26T18:09:07.788Z [container-agent]     raise retry_exc.reraise()
2024-02-26T18:09:07.788Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/tenacity/__init__.py", line 158, in reraise
2024-02-26T18:09:07.788Z [container-agent]     raise self.last_attempt.result()
2024-02-26T18:09:07.788Z [container-agent]   File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
2024-02-26T18:09:07.788Z [container-agent]     return self.__get_result()
2024-02-26T18:09:07.788Z [container-agent]   File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
2024-02-26T18:09:07.788Z [container-agent]     raise self._exception
2024-02-26T18:09:07.788Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/tenacity/__init__.py", line 382, in __call__
2024-02-26T18:09:07.788Z [container-agent]     result = fn(*args, **kwargs)
2024-02-26T18:09:07.788Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/mysql_k8s_helpers.py", line 237, in wait_until_mysql_connection
2024-02-26T18:09:07.788Z [container-agent]     raise MySQLServiceNotRunningError("Connection with mysqlsh not possible")
2024-02-26T18:09:07.788Z [container-agent] mysql_k8s_helpers.MySQLServiceNotRunningError: Connection with mysqlsh not possible
2024-02-26T18:09:08.085Z [container-agent] 2024-02-26 18:09:08 ERROR juju.worker.uniter.operation runhook.go:180 hook "metrics-endpoint-relation-created" (via hook dispatching script: dispatch) failed: exit status 1
2024-02-26T18:09:09.163Z [container-agent] 2024-02-26 18:09:09 ERROR juju-log Failed to rotate mysql logs
2024-02-26T18:09:09.163Z [container-agent] Traceback (most recent call last):
2024-02-26T18:09:09.163Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/mysql_k8s_helpers.py", line 628, in _execute_commands
2024-02-26T18:09:09.163Z [container-agent]     stdout, stderr = process.wait_output()
2024-02-26T18:09:09.163Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1441, in wait_output
2024-02-26T18:09:09.163Z [container-agent]     raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
2024-02-26T18:09:09.163Z [container-agent] ops.pebble.ExecError: non-zero exit code 1 executing ['logrotate', '-f', '/etc/logrotate.d/flush_mysql_logs'], stdout='', stderr='error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory\n'
2024-02-26T18:09:09.163Z [container-agent] 
2024-02-26T18:09:09.163Z [container-agent] During handling of the above exception, another exception occurred:
2024-02-26T18:09:09.163Z [container-agent] 
2024-02-26T18:09:09.163Z [container-agent] Traceback (most recent call last):
2024-02-26T18:09:09.163Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/rotate_mysql_logs.py", line 50, in _rotate_mysql_logs
2024-02-26T18:09:09.163Z [container-agent]     self.charm._mysql._execute_commands(["logrotate", "-f", LOG_ROTATE_CONFIG_FILE])
2024-02-26T18:09:09.163Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/mysql_k8s_helpers.py", line 632, in _execute_commands
2024-02-26T18:09:09.163Z [container-agent]     raise MySQLExecError(e.stderr)
2024-02-26T18:09:09.163Z [container-agent] charms.mysql.v0.mysql.MySQLExecError: error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory
2024-02-26T18:09:09.163Z [container-agent] 
2024-02-26T18:09:09.267Z [container-agent] 2024-02-26 18:09:09 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook
2024-02-26T18:10:01.296Z [container-agent] 2024-02-26 18:10:01 ERROR juju-log Failed to rotate mysql logs
2024-02-26T18:10:01.296Z [container-agent] Traceback (most recent call last):
2024-02-26T18:10:01.296Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/mysql_k8s_helpers.py", line 628, in _execute_commands
2024-02-26T18:10:01.296Z [container-agent]     stdout, stderr = process.wait_output()
2024-02-26T18:10:01.296Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/venv/ops/pebble.py", line 1441, in wait_output
2024-02-26T18:10:01.296Z [container-agent]     raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
2024-02-26T18:10:01.296Z [container-agent] ops.pebble.ExecError: non-zero exit code 1 executing ['logrotate', '-f', '/etc/logrotate.d/flush_mysql_logs'], stdout='', stderr='error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory\n'
2024-02-26T18:10:01.296Z [container-agent] 
2024-02-26T18:10:01.296Z [container-agent] During handling of the above exception, another exception occurred:
2024-02-26T18:10:01.296Z [container-agent] 
2024-02-26T18:10:01.296Z [container-agent] Traceback (most recent call last):
2024-02-26T18:10:01.296Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/rotate_mysql_logs.py", line 50, in _rotate_mysql_logs
2024-02-26T18:10:01.296Z [container-agent]     self.charm._mysql._execute_commands(["logrotate", "-f", LOG_ROTATE_CONFIG_FILE])
2024-02-26T18:10:01.296Z [container-agent]   File "/var/lib/juju/agents/unit-horizon-mysql-1/charm/src/mysql_k8s_helpers.py", line 632, in _execute_commands
2024-02-26T18:10:01.296Z [container-agent]     raise MySQLExecError(e.stderr)
2024-02-26T18:10:01.296Z [container-agent] charms.mysql.v0.mysql.MySQLExecError: error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory
2024-02-26T18:10:01.296Z [container-agent] 
2024-02-26T18:10:01.412Z [container-agent] 2024-02-26 18:10:01 INFO juju.worker.uniter resolver.go:161 awaiting error resolution for "relation-created" hook

Additional context

Got the charm going further by killing the pod Happened to 3 out of my 9 MySQL deployments (yes, MySQL is deployed as 9 different applications inside the same model) Each MySQL is deployed with 3 units, it's the unit receiving the event first that failed each time.

github-actions[bot] commented 7 months ago

https://warthogs.atlassian.net/browse/DPE-3679

paulomach commented 7 months ago

@gboutry Although I've identified at least one race condition that could lead to the error, I could not reproduce locally.

I'm running

juju add-model cos
juju deploy cos-lite --overlay ${HOME}/storage-small-overlay.yaml --trust
juju offer prometheus:metrics-endpoint pm-me
juju add-model squirrel
juju deploy ch:mysql-k8s --channel 8.0/stable --trust -n 3
juju consume admin/cos.pm-me
juju integrate mysql-k8s pm-me
juju model-config logging-config="<root>=WARNING;unit=DEBUG" -m squirrel
juju debug-log

Do you see any fundamental difference from what you are doing?

gboutry commented 7 months ago

I don't see any fundamental difference, except that we have grafana-agent-k8s in the middle, but that should not have an effect (I hope).

I did a deployment from 8.0/edge, related all the endpoints and did not encounter that issue.

paulomach commented 3 months ago

Closing this as unable to reproduce on current revisions.