Design - RDPC Backup and Manual Restore: Kafka

[akachru-github]

Investigation ticket: https://github.com/icgc-argo/rdpc-infra/issues/71

• Original investigation targeted Kafka connector for MDB. We may want to switch this to something more appropriate like another cold Kafka as we do not want to limit the proliferation of technology and moving pieces in our syste. If we use a cold Kafka, we can fall back to this if the primary Kafka becomes unavailable. We will be mirroring one topic called weblog.
• Mirroring events + metadata to target
• Kafka is append only

Expected output:

• Design wiki for our finalized approach for backing up Kafka.

---

Docs: https://wiki.oicr.on.ca/display/icgcargotech/Kafka+backup

[akachru-github]

After meeting with @lepsalex, @andricDu , @yalturmes where @henro001 presented POC design:

• Backing up to a file was agreed as the best solution
• Look into single file size limit of 2TB in Linux system
• Look into strategy for backing up the txt file: Either stopping Kafka Connect for the duration of the backup (with large files, this could take some time) or backups roll over to new files on some set time period (daily, monthly, yearly?) to avoid large file sizes which will allow backups of those files to be taken with either minimum Kafka Connect downtime or no downtime at all.

Moving ticket back to in progress as we will spend some time planning for the above cases before starting implementation.