search

openebs / zfs-localpv

Dynamically provision Stateful Persistent Node-Local Volumes & Filesystems for Kubernetes that is integrated with a backend ZFS data storage stack.
https://openebs.io
Apache License 2.0
419 stars 102 forks source link

adding encryption in ZFSVolume CR,volume creation failed #410

Closed ptthon closed 2 years ago

ptthon commented 2 years ago

feat(ZFSPV): adding encryption in ZFSVolume CR According to the following steps to create the PVC failed Steps are as follows:

  1. truncate -s 100G /tmp/disk.img zpool create zfspv-pool sudo losetup -f /tmp/disk.img --show root@master3:~# zfs list NAME USED AVAIL REFER MOUNTPOINT zfspv-pool 327K 48.2G 24K /zfspv-pool
  2. Put the secret key in the corresponding node (/home/keys)
  3. I use the encryption option in storage class

apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: openebs-zfspv allowVolumeExpansion: true parameters: recordsize: "4k" compression: "on" dedup: "on" thinprovision: "yes" encryption: "on" keyformat: "raw" keylocation: "file:///home/keys/key" fstype: "zfs" poolname: "zfspv-pool" provisioner: zfs.csi.openebs.io allowedTopologies:

Events: Type Reason Age From Message

Normal Provisioning 4m7s (x14 over 23m) zfs.csi.openebs.io_openebs-zfs-controller-0_5028c459-a44a-4063-9e5b-60efc06d2281 External provisioner is provisioning volume for claim "default/csi-zfspv-enc" Warning ProvisioningFailed 4m6s (x14 over 23m) zfs.csi.openebs.io_openebs-zfs-controller-0_5028c459-a44a-4063-9e5b-60efc06d2281 failed to provision volume with StorageClass "openebs-zfspv-enc": rpc error: code = Internal desc = not able to provision the volume, nodes [master3], err : zfs: volume creation failed Normal ExternalProvisioning 3m30s (x82 over 23m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "zfs.csi.openebs.io" or manually created by system administrator

pawanpraka1 commented 2 years ago

Can you share the node daemonset log also?

ptthon commented 2 years ago

Can you share the node daemonset log also

I0506 11:23:09.221454 1 volume.go:123] Got add event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:23:09.243247 1 zfs_util.go:424] zfs: could not create volume zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b cmd [create -s -V 4294967296 -o dedup=on -o compression=on -o encryption=on -o keylocation=file:///home/keys/key -o keyformat=hex zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b] error: cannot create 'zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b': invalid property 'encryption' I0506 11:23:09.270411 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' I0506 11:23:10.249497 1 volume.go:144] Got update event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:23:10.288702 1 zfs_util.go:635] destroy: volume zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b is not present, error: exit status 1 I0506 11:23:10.316435 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' I0506 11:23:10.316481 1 volume.go:169] Got delete event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:23:10.316841 1 volume.go:50] zfsvolume 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' has been deleted I0506 11:23:10.316940 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' I0506 11:23:11.309064 1 volume.go:123] Got add event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:23:11.325652 1 zfs_util.go:424] zfs: could not create volume zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b cmd [create -s -V 4294967296 -o dedup=on -o compression=on -o encryption=on -o keylocation=file:///home/keys/key -o keyformat=hex zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b] error: cannot create 'zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b': invalid property 'encryption' I0506 11:23:11.341643 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' I0506 11:23:12.339655 1 volume.go:144] Got update event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:23:12.367379 1 zfs_util.go:635] destroy: volume zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b is not present, error: exit status 1 I0506 11:23:12.368990 1 volume.go:169] Got delete event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:23:12.380740 1 volume.go:259] error syncing 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b': Operation cannot be fulfilled on zfsvolumes.zfs.openebs.io "pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b": StorageError: invalid object, Code: 4, Key: /registry/zfs.openebs.io/zfsvolumes/openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 05d59d86-9cf9-4d19-a171-01bcfa4888a6, UID in object meta: , requeuing E0506 11:23:12.380792 1 volume.go:50] zfsvolume 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' has been deleted I0506 11:23:12.381977 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' E0506 11:23:12.386075 1 volume.go:50] zfsvolume 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' has been deleted I0506 11:23:12.386162 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' I0506 11:23:14.397563 1 volume.go:123] Got add event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:23:14.449786 1 zfs_util.go:424] zfs: could not create volume zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b cmd [create -s -V 4294967296 -o dedup=on -o compression=on -o encryption=on -o keylocation=file:///home/keys/key -o keyformat=hex zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b] error: cannot create 'zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b': invalid property 'encryption' I0506 11:23:14.471993 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' I0506 11:23:15.428971 1 volume.go:144] Got update event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:23:15.453532 1 zfs_util.go:635] destroy: volume zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b is not present, error: exit status 1 I0506 11:23:15.464239 1 volume.go:169] Got delete event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:23:15.467879 1 volume.go:50] zfsvolume 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' has been deleted I0506 11:23:15.468024 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' E0506 11:23:15.468062 1 volume.go:259] error syncing 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b': Operation cannot be fulfilled on zfsvolumes.zfs.openebs.io "pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b": StorageError: invalid object, Code: 4, Key: /registry/zfs.openebs.io/zfsvolumes/openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 4f20ad5a-0836-46f4-910b-f12f2f9f60c9, UID in object meta: , requeuing E0506 11:23:15.472890 1 volume.go:50] zfsvolume 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' has been deleted I0506 11:23:15.472929 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' I0506 11:23:19.500420 1 volume.go:123] Got add event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:23:19.528423 1 zfs_util.go:424] zfs: could not create volume zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b cmd [create -s -V 4294967296 -o dedup=on -o compression=on -o encryption=on -o keylocation=file:///home/keys/key -o keyformat=hex zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b] error: cannot create 'zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b': invalid property 'encryption' I0506 11:23:19.557130 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' I0506 11:23:20.538978 1 volume.go:144] Got update event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:23:20.564250 1 zfs_util.go:635] destroy: volume zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b is not present, error: exit status 1 I0506 11:23:20.578597 1 volume.go:169] Got delete event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:23:20.581164 1 volume.go:259] error syncing 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b': Operation cannot be fulfilled on zfsvolumes.zfs.openebs.io "pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b": StorageError: invalid object, Code: 4, Key: /registry/zfs.openebs.io/zfsvolumes/openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 847b5d73-bd0c-4704-9609-ec39435a2594, UID in object meta: , requeuing E0506 11:23:20.581199 1 volume.go:50] zfsvolume 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' has been deleted I0506 11:23:20.582227 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' E0506 11:23:20.586292 1 volume.go:50] zfsvolume 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' has been deleted I0506 11:23:20.586323 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' I0506 11:23:28.613471 1 volume.go:123] Got add event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:23:28.633030 1 zfs_util.go:424] zfs: could not create volume zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b cmd [create -s -V 4294967296 -o dedup=on -o compression=on -o encryption=on -o keylocation=file:///home/keys/key -o keyformat=hex zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b] error: cannot create 'zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b': invalid property 'encryption' I0506 11:23:28.644789 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' I0506 11:23:29.638508 1 volume.go:144] Got update event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:23:29.674204 1 zfs_util.go:635] destroy: volume zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b is not present, error: exit status 1 I0506 11:23:29.690008 1 volume.go:169] Got delete event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:23:29.696678 1 volume.go:259] error syncing 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b': Operation cannot be fulfilled on zfsvolumes.zfs.openebs.io "pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b": StorageError: invalid object, Code: 4, Key: /registry/zfs.openebs.io/zfsvolumes/openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: bb3ad4a4-e8a2-4bfa-a302-30314c573153, UID in object meta: , requeuing E0506 11:23:29.696880 1 volume.go:50] zfsvolume 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' has been deleted I0506 11:23:29.697893 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' E0506 11:23:29.701850 1 volume.go:50] zfsvolume 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' has been deleted I0506 11:23:29.701875 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' I0506 11:23:45.728735 1 volume.go:123] Got add event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:23:45.761243 1 zfs_util.go:424] zfs: could not create volume zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b cmd [create -s -V 4294967296 -o dedup=on -o compression=on -o encryption=on -o keylocation=file:///home/keys/key -o keyformat=hex zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b] error: cannot create 'zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b': invalid property 'encryption' I0506 11:23:45.791692 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' I0506 11:23:46.759773 1 volume.go:144] Got update event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:23:46.798991 1 zfs_util.go:635] destroy: volume zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b is not present, error: exit status 1 I0506 11:23:46.821024 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' I0506 11:23:46.821994 1 volume.go:169] Got delete event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:23:46.822194 1 volume.go:50] zfsvolume 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' has been deleted I0506 11:23:46.822354 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' I0506 11:24:18.816365 1 volume.go:123] Got add event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:24:18.839799 1 zfs_util.go:424] zfs: could not create volume zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b cmd [create -s -V 4294967296 -o dedup=on -o compression=on -o encryption=on -o keylocation=file:///home/keys/key -o keyformat=hex zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b] error: cannot create 'zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b': invalid property 'encryption' I0506 11:24:18.863473 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' I0506 11:24:19.857952 1 volume.go:144] Got update event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:24:19.883272 1 zfs_util.go:635] destroy: volume zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b is not present, error: exit status 1 I0506 11:24:19.904159 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' I0506 11:24:19.904479 1 volume.go:169] Got delete event for ZV zfspv-pool/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b E0506 11:24:19.904710 1 volume.go:50] zfsvolume 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b' has been deleted I0506 11:24:19.904873 1 volume.go:254] Successfully synced 'openebs/pvc-b605df95-05d2-45a8-8b89-0833a23b7c8b'

pawanpraka1 commented 2 years ago

What is the zfs version?

ptthon commented 2 years ago

What is the zfs version? zfsutils-linux version is 0.7.5-1ubuntu16.12 zfs-localpv url is https://openebs.github.io/charts/zfs-operator.yaml,the version maybe is newest

pawanpraka1 commented 2 years ago

@ptthon you need zfs 0.8 or higher version to have encryption. It will not work on zfs 0.7.

ptthon commented 2 years ago

@ptthon you need zfs 0.8 or higher version to have encryption. It will not work on zfs 0.7.

I understand, thank you very much!