When we scale up to more than one unit, it is possible that we start the log rotate dispatcher before the unit is fully initialized (and the logrotate config file is copied over the workload container).
unit-mysql-k8s-2: 18:00:00 ERROR unit.mysql-k8s/2.juju-log Failed to rotate mysql logs
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/src/mysql_k8s_helpers.py", line 646, in _execute_commands
stdout, stderr = process.wait_output()
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/venv/ops/pebble.py", line 1359, in wait_output
raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
ops.pebble.ExecError: non-zero exit code 1 executing ['logrotate', '-f', '/etc/logrotate.d/flush_mysql_logs'], stdout='', stderr='error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory\n'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/src/rotate_mysql_logs.py", line 47, in _rotate_mysql_logs
self.charm._mysql._execute_commands(["logrotate", "-f", LOG_ROTATE_CONFIG_FILE])
File "/var/lib/juju/agents/unit-mysql-k8s-2/charm/src/mysql_k8s_helpers.py", line 650, in _execute_commands
raise MySQLExecError(e.stderr)
charms.mysql.v0.mysql.MySQLExecError: error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory
unit-mysql-k8s-1: 18:00:00 DEBUG unit.mysql-k8s/1.juju-log Failed command: commands=['logrotate', '-f', '/etc/logrotate.d/flush_mysql_logs'], user=None, group=None
unit-mysql-k8s-1: 18:00:00 ERROR unit.mysql-k8s/1.juju-log Failed to rotate mysql logs
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-mysql-k8s-1/charm/src/mysql_k8s_helpers.py", line 646, in _execute_commands
stdout, stderr = process.wait_output()
File "/var/lib/juju/agents/unit-mysql-k8s-1/charm/venv/ops/pebble.py", line 1359, in wait_output
raise ExecError[AnyStr](self._command, exit_code, out_value, err_value)
ops.pebble.ExecError: non-zero exit code 1 executing ['logrotate', '-f', '/etc/logrotate.d/flush_mysql_logs'], stdout='', stderr='error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory\n'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-mysql-k8s-1/charm/src/rotate_mysql_logs.py", line 47, in _rotate_mysql_logs
self.charm._mysql._execute_commands(["logrotate", "-f", LOG_ROTATE_CONFIG_FILE])
File "/var/lib/juju/agents/unit-mysql-k8s-1/charm/src/mysql_k8s_helpers.py", line 650, in _execute_commands
raise MySQLExecError(e.stderr)
charms.mysql.v0.mysql.MySQLExecError: error: cannot stat /etc/logrotate.d/flush_mysql_logs: No such file or directory
unit-mysql-k8s-0: 18:01:00 DEBUG unit.mysql-k8s/0.juju-log ops 2.7.0 up and running.
Solution
Only start the log rotate dispatcher if the unit has been initialized and only flush mysql logs in the custom event handler if the unit has been initialized.
Issue
When we scale up to more than one unit, it is possible that we start the log rotate dispatcher before the unit is fully initialized (and the logrotate config file is copied over the workload container).
Solution
Only start the log rotate dispatcher if the unit has been initialized and only flush mysql logs in the custom event handler if the unit has been initialized.