Closed paulomach closed 4 months ago
As discussed in private: it is not possible to re-join the cluster after recreate-cluster
. The juju stuck on:
Unit Workload Agent Address Ports Message
db2/0* blocked idle 10.1.204.228 User data found, aborting async replication setup
db2/1 waiting idle 10.1.204.196 waiting replica cluster be configured
db2/2 waiting idle 10.1.204.207 waiting replica cluster be configured
STR:
juju run -m az2 db2/leader recreate-cluster
it is not possible to re-join db2 to az1.db1 because az2 has the previos data which is impossible to wipe.Proposal:
1) the blocked cluster after juju remove-relation -m az2 async-primary db2
can be re-joined back using juju relate -m az2 async-primary db2
(the local az2 data will be wiped)
2) if the cluster re-create, it cannot re-join, it must be destroyed and re-deployed. Safety first.
@taurus-forever
2. if the cluster re-create, it cannot re-join, it must be destroyed and re-deployed. Safety first.
Should we make this parametrized for the action?
@taurus-forever
2. if the cluster re-create, it cannot re-join, it must be destroyed and re-deployed. Safety first.
Should we make this parametrized for the action?
IMHO, not now. It is a dangerous action. If you want to re-attach the re-created cluster:
juju remove-application mysql && juju remove-storage ... && juju deploy mysql
(c) Safety first.
P.S. we can implement such action later after the proper spec review by PMs.
For the history, revision 128 from edge/arepl fails:
unit-db2-0: 14:13:57 DEBUG unit.db2/0.juju-log async-replica:3: Syncing credentials from primary cluster
unit-db2-0: 14:13:57 ERROR unit.db2/0.juju-log async-replica:3: Uncaught exception while in charm code:
Traceback (most recent call last):
File "/var/lib/juju/agents/unit-db2-0/charm/venv/ops/model.py", line 3019, in _run
result = subprocess.run(args, **kwargs) # type: ignore
File "/usr/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-db2-0/secret-get', 'secret:cnpfe3nmp25c77tiud60', '--format=json')' returned non-zero exit status 1.
For the history, revision 128 from edge/arepl fails:
unit-db2-0: 14:13:57 DEBUG unit.db2/0.juju-log async-replica:3: Syncing credentials from primary cluster unit-db2-0: 14:13:57 ERROR unit.db2/0.juju-log async-replica:3: Uncaught exception while in charm code: Traceback (most recent call last): File "/var/lib/juju/agents/unit-db2-0/charm/venv/ops/model.py", line 3019, in _run result = subprocess.run(args, **kwargs) # type: ignore File "/usr/lib/python3.10/subprocess.py", line 526, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-db2-0/secret-get', 'secret:cnpfe3nmp25c77tiud60', '--format=json')' returned non-zero exit status 1.
Updated revision ~r129~ ~r130~ r134 with fixes
Issue
Need to have async replication
Solution
Use async replication library (owned by vm charm). Implementation as described by this workflow How-to draft in the discourse page
Also implements DPE-3761
Integration test included
TODO:
Unit tests tests on a fup PR on VM only(lib owner)