Setup replication on production clickhouse instance

[hellais]

There is a scheduled downtime for backend-fsn set for:

Start:  2024-12-05, 03:30 AM UTC
Estimated end:  2024-12-05, 05:30 AM UTC

We should use this as an opportunity and motivation to setup replication between backend-fsn and backend-hel so that we can be fault tolerant to these kinds of downtimes.

Relevant bits of documentation are:

• https://clickhouse.com/docs/en/architecture/replication
• https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication#converting-from-mergetree-to-replicatedmergetree

We also need to run a dedicated clickhouse-zookeeper instance in addition to one on the two replicas. It's probably good to set that up on the monitoring host since the memory requirements are relatively light compared to clickhouse.

[hellais]

This is being setup inside of https://github.com/ooni/devops/pull/112 as part of the work to get Pipeline v5 into production. Once it's finished we should move the fastpath clickhouse database over and set it up as a replicated table.

I wrote some notes on this can be done here: https://github.com/ooni/devops/blob/prod-pipeline-v5/docs/merge-tree-replication.md