search

hpe-storage / truenas-csp

TrueNAS Container Storage Provider for HPE CSI Driver for Kubernetes
https://scod.hpedev.io
MIT License
65 stars 8 forks source link

Volumesnapshots component missing? #33

Closed ishioni closed 4 months ago

ishioni commented 1 year ago

I wanted to roll out velero for backups which seems to require volumesnapshots

However, if I try to test it out with an example volumesnapshot from the link above I get

no matches for kind "VolumeSnapshot" in version "snapshot.storage.k8s.io/v1beta1"

This is on k3s 1.23.12. I see in your e2e test you have a volumesnapshotclass, and I see democratic-csi deploys the snapshot-controller as a component for this to work. I know this is not strictly truenas-csp related, but which component am I missing for snapshots here?

datamattsson commented 1 year ago

Where did you find the example VolumeSnapshot stanza? You need to call out "snapshot.storage.k8s.io/v1" since Kubernetes 1.20 (well, I'm not sure when "v1beta1" got fully removed). Check example here: https://kubernetes.io/docs/concepts/storage/volume-snapshots/#volumesnapshots

(I also noticed my error in the e2e test suite that calls out the wrong API version)

Anyhow, it's supposed to be up to the Kubernetes distribution vendor to install the external snapshot controller. Not many distributions actually provide this. So, if you want to enable volume snapshots for the TrueNAS CSP you may follow the instructions found with the HPE CSI Driver.

ishioni commented 1 year ago

Where did you find the example VolumeSnapshot stanza?

Exactly the same link from the one you posted :)

So, if you want to enable volume snapshots for the TrueNAS CSP you may follow the instructions found with the HPE CSI Driver.

Ah, so as I suspected, I need external-snapshot-controller. Could you maybe link that info in the Readme somewhere?

Thanks for the support!

datamattsson commented 1 year ago

Thanks for the suggestion. I'll have a look to have this outlined on the INSTALL to ensure there are pointers to enable and use VolumeSnapshots.

ishioni commented 1 year ago

@datamattsson I'm playing around with this now, snapshot-controller in tow. However, I think that's not how it's supposed to work

Consider this scenario I create a pvc - my-pv I create a volumesnapshot out of it - my-pv-snapshot - I delete the original pvc I create a new pvc out of the snapshot - It now references (in my case) SSD_k3s_pvc-f72e0ff5-33cf-40fb-b488-6b960928f53b@snapshot-2f5d1c03-9dba-448b-b426-c71ac2f1c3f9, which is a zfs snapshot of a zvol that doesn't exist anymore, and subsequently fails

Hence all snapshots are bound to a backing zvol. If that zvol were to be deleted by for example killing the underlying pv, then those snapshots are dead, providing no protection

datamattsson commented 1 year ago

Would we like zvols which has snapshots created by the CSI driver to be retained on the backend? I'm not sure why PVs with snapshotcontent derived from them aren't protected. It's odd.

ishioni commented 1 year ago

Yeah, it's a quirk of how snapshots are handled by zfs, where the snapshots are bound to the originating zvol/dataset. Democratic CSI has an option to clone the zvol/dataset outright.

That being said, you should know i've moved on to using nvmeof, so if you do change the handling I won't be able to test it out

datamattsson commented 1 year ago

Democratic CSI has an option to clone the zvol/dataset outright.

Interesting, like doing a send/recv asynchronously within the same pool?

datamattsson commented 4 months ago

Add hints to INSTALL.md.