The PostgreSQL VM charm rollback test started to fail by the combination of adding TimescaleDB to the Charmed PostgreSQL Snap + updating the upgrade library to LIBPATCH 16 (which removed a defer call that was hiding the issue fixed by this PR).
Explanation of the issue (considering the tests from the PostgreSQL VM charm):
In the test_upgrade_from_edge test:
TimescaleDB in the snap is leading to a longer snap refresh (only some more seconds).
With that, Patroni is performing a failover, so a new unit, different from the Juju leader unit (the former primary), becomes the primary.
Also with that, the former primary unit, the Juju leader unit, becomes a replica (not a synchronous standby).
In the test_fail_and_rollback test:
After the failed upgrade, the pre-upgrade-check is run, which sets the recovery state for the Juju leader unit (which is a replica).
When the refresh is started, the upgrade charm event is fired in each of the three units, setting the state to ready in all of them.
The upgrade changed event is fired only in the non-primary units because only the relation data from the Juju leader unit changed (the state changed from recovery to ready, while in the other units, it was already set to ready, so no change, no events firing for the Juju leader unit).
As the Juju leader unit is the replica, it’s the top of the stack unit (the one that should be upgraded first). However, as the upgrade relation changed, the event hasn’t fired for it; it cannot start the upgrade (in this case, it is a rollback).
For Kafka and MySQL operators on VM, it hasn’t happened because both always set the upgrade stack with the Juju leader unit as the last unit to be upgraded. For the following codes, remember that the build_upgrade_stack is always executed in the Juju leader unit.
If the Juju leader unit is the top unit of upgrading the stack, execute the on_upgrade_changed handler logic during a rollback. This way, it can run the logic that starts the upgrade process, in the end of the on_upgrade_changed handler.
This fix was tested on the following charms to avoid breaking them:
Issue:
The PostgreSQL VM charm rollback test started to fail by the combination of adding TimescaleDB to the Charmed PostgreSQL Snap + updating the
upgradelibrary to LIBPATCH 16 (which removed adefercall that was hiding the issue fixed by this PR).Explanation of the issue (considering the tests from the PostgreSQL VM charm):
In the
test_upgrade_from_edgetest:In the
test_fail_and_rollbacktest:pre-upgrade-checkis run, which sets therecoverystate for the Juju leader unit (which is a replica).recoverytoready, while in the other units, it was already set toready, so no change, no events firing for the Juju leader unit).For Kafka and MySQL operators on VM, it hasn’t happened because both always set the upgrade stack with the Juju leader unit as the last unit to be upgraded. For the following codes, remember that the
build_upgrade_stackis always executed in the Juju leader unit.Solution
If the Juju leader unit is the top unit of upgrading the stack, execute the
on_upgrade_changedhandler logic during a rollback. This way, it can run the logic that starts the upgrade process, in the end of theon_upgrade_changedhandler.This fix was tested on the following charms to avoid breaking them: