search

kahkhang / kube-linode

:whale: Provision a Kubernetes/CoreOS cluster on Linode
MIT License
212 stars 31 forks source link

Question: Kube Linode Backups? #94

Open cboothe opened 6 years ago

cboothe commented 6 years ago

Hi,

I have setup a cluster using this project. However the native Linode Backup service fails.

"Your backups are currently failing because the Linode Backup Service is unable to mount your images. This is likely due to an incompatibility between your disk images and our service."

Specifically: "The backup system must be able to mount your disk images on the host. If you've used fdisk on your images to create partitions, or created encrypted volumes, or LVM, or done anything other than use our deployment or disk image creation tools

So my question is since you have partitioned two disk images (CoreOS and Storage) how do I roll my own backups? I assume I have to backup etcd, etc. But don't know what from the Master node needs backup. Assuming that the worker nodes can just be rebuilt automatically on restore.

Any suggestions would be great.

Thanks, Chris

kahkhang commented 6 years ago

Hi @cboothe, thanks for raising this issue! Unfortunately Linode backups will not work on coreos images because of their unique partitioning scheme (see https://linode.com/docs/tools-reference/custom-kernels-distros/install-coreos-on-your-linode/).

For etcd backups, you can follow this guide (https://coreos.com/etcd/docs/latest/v2/admin_guide.html) with the appropriate environmental variables set pointing to the etcd certs located in the cluster/ folder.

As for persistent storage backups, assuming you are using persistent volumes via Rook, unfortunately there is no out of the box solution for snapshots, though I see work is planned to support such a feature (see https://github.com/rook/rook/issues/1552) would should hopefully be available soon :) If you are just looking for high availability, you can tune the number of replicas / error correction etc in the manifest before running the script (see https://github.com/kahkhang/kube-linode/blob/master/manifests/rook/rook-storageclass.yaml)

Assuming you're using the kube-linode setup out of the box, I think etcd and persistent volumes are probably the only things that you need to backup in order for disaster recovery.