search

davidmirror-ops / flyte-the-hard-way

Prepare requirements and deploy Flyte using Helm
Creative Commons Zero v1.0 Universal
57 stars 15 forks source link

Set MINIO_DATA_DIR to use mounted PVC #17

Closed BroderPeters closed 6 months ago

BroderPeters commented 6 months ago

If MINIO_DATA_DIR is not set, minio will use the default "local" container path and will lose data if the container crashes.

To double-check this, connect to your server running the currently documented config and find the locally PV mounted by e.g. lsblk

└─sda3                      8:3    0  248G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0  100G  0 lvm  /var/lib/kubelet/pods/152f21af-1e1d-48d6-aed6-91779432aa80/volume-subpaths/installinfo/agent/1
                                                   /var/lib/kubelet/pods/9fcdbb7f-8c5c-4b10-91f3-960b299da759/volume-subpaths/local-pv-ad0a0b8d/registry/1
                                                   /var/lib/kubelet/pods/5bd3e4ff-1c59-4f2f-81b0-cc32bfe6086d/volume-subpaths/tigera-ca-bundle/calico-node/1
                                                   /var/lib/kubelet/pods/b9b63cea-102d-4c17-bb54-48ab44afa554/volume-subpaths/tigera-ca-bundle/calico-typha/1
                                                   /var/lib/kubelet/pods/1cba5f34-7f3c-4670-a347-6fc0f8529227/volumes/kubernetes.io~local-volume/local-pv-c67e4d39
                                                   /var/lib/kubelet/pods/9fcdbb7f-8c5c-4b10-91f3-960b299da759/volumes/kubernetes.io~local-volume/local-pv-ad0a0b8d
                                                   /var/lib/kubelet/pods/87643f0c-6509-47f7-8a0b-d5585ca52cda/volumes/kubernetes.io~local-volume/local-pv-583e891a

Listing the mounted PV shows no entries

$ sudo ls -la /var/lib/kubelet/pods/1cba5f34-7f3c-4670-a347-6fc0f8529227/volumes/kubernetes.io~local-volume/local-pv-c67e4d39
total 8
drwxr-xr-x 2 root root 4096 Jul 28  2023 .
drwxr-x--- 3 root root 4096 Jan  3 08:25 ..

Or you simply just check the /data/ dir in the minio container:

I have no name!@minio-6bd54747b-7b76m:/opt/bitnami/minio-client$ ls -la /data/
total 8
drwxr-xr-x 2 root root 4096 Jul 28  2023 .
drwxr-xr-x 1 root root 4096 Feb 16 08:59 ..

To back up and restore your data, I can recommend using the MinIO client. Additionally, you should make sure that your PVs have persistentVolumeReclaimPolicy: Retain so the don't delete local data by accident.

BroderPeters commented 6 months ago

Thanks so much @BroderPeters!

Do you think we should also add to the docs the note regarding the PV's Reclaim Policy?

Totally, will get back to it in another PR once I'm back on this topic at work