Upgrade to 0.24 without downtime?

DaanDD commented 3 weeks ago

When looking at the instructions on how to upgrade to 0.24 it involves deleting the old CHK installation and creating a new one. The ClickHouse instance I'm running is constantly being written to so I cannot just turn off Keeper. Is there a way to upgrade to 0.24 without downtime? Maybe I could create a new Keeper installation that syncs data with the existing one, then hot swap between old and new Keeper?

UnamedRus commented 3 weeks ago

Hi,

We are currently in process of writing procedure to migrate CHK from old to new syntax.

Maybe I could create a new Keeper installation that syncs data with the existing one, then hot swap between old and new Keeper?

Yeah, it's basically how it will work:

Detach old CHK from operator management.
Upgrade Operator + CRDs
Create new CHK, which will also include old nodes in it's files.xml configuration.
Update config for old keeper installation (ie include new nodes)
Check sync
Switch CH to new keeper ensemble.
Remove old nodes from ensemble
Drop old nodes.

zeev1079 commented 2 weeks ago

have a couple of questions regarding this migration process(which looks much straight forward):

What happens if the old Keeper nodes have different volume mount paths, such as:

keeper_server/log_storage_path: /var/lib/clickhouse-keeper/logs 
keeper_server/snapshot_storage_path: /var/lib/clickhouse-keeper/snapshots
keeper_server/storage_path: /var/lib/clickhouse-keeper

Is adding the old Keeper nodes in the files.xml configuration the correct way to include them in the new installation? For example:


configuration:
files:
  files.xml: |
    <yandex>
      <zookeeper>
        <node>
          <host>old-keeper-node-1</host>
          <port>2181</port>
        </node>
        <node>
          <host>old-keeper-node-2</host>
          <port>2181</port>
        </node>
        <node>
          <host>old-keeper-node-3</host>
          <port>2181</port>
        </node>
        <node>
          <host>new-keeper-node-1</host>
          <port>2181</port>
        </node>
        <node>
          <host>new-keeper-node-2</host>
          <port>2181</port>
        </node>
        <node>
          <host>new-keeper-node-3</host>
          <port>2181</port>
        </node>
      </zookeeper>
    </yandex>```

clement94310 commented 3 days ago

hello i read the upgrade doc : https://github.com/Altinity/clickhouse-operator/blob/0.24.0/docs/keeper_migration_from_23_to_24.md

in point 7 how i have to modify the pvc if i have 3 servers ? volumeClaimTemplates:

name: default spec: ... volumeName: $PV

Altinity / clickhouse-operator

Upgrade to 0.24 without downtime? #1550