[Enhancement] Implement automatic backup

[ibumarskov]

Description

Automatic backup of zookeeper db is a useful mechanism in demand in many real-world scenarios. It would be great to implement a separate controller responsible for automatic backup by schedule or on-demand backups. As the simplest implementation we can provide backup of ZooKeeper by copying its transaction and snapshot logs. It's a reasonable method, there is a short article regarding backup/restore procedure for zookeeper: Apache ZooKeeper Backup, a Treatise

Importance

Should-have

Location

Zookeeper operator.

Suggestions for an improvement

Separate controller can be added to watch for a new type of resource (zookeeper-backup for example). Controller should take following responsibilities:

• Provide cluster health check and cancel backup operation if required.
• Detect ZK leader pod and prepare/reconfigure CronJob configuration (Backup pod should land on node where leader is elected).
• Schedule CronJob with a backup script.
• Provide mechanism to periodic checks to make sure CronJob configuration is updated and valid (for example in case of new leader election)

Backup script should contains following steps:

1. May contain some additional checks (for example check that it really landed on pod with leader, check datadir folder exists, check free storage space, etc.)
2. Create archive of datadir folder and put it in a backup folder (It should be a volume intended for storage backups)
3. Check amount of backups and delete oldest if necessary (should be specified in zookeeper-backup cr)

[anishakj]

@ibumarskov Do you find the need of backup when we are using ephemeral storage? In the case of persistent volumes , data is not lost even if there is a crash.

[ibumarskov]

You can lost data in different ways. Data failures can be the result of hardware or software failure, data corruption, or a human-caused event, such as a malicious attack or accidental deletion of data. So, data backup is a good practice.

[anishakj]

@ibumarskov Would you like to contribute adding backup support by raising a PR.

[ibumarskov]

@anishakj Yes, I'm going to provide an implementation soon.