tareksha commented 1 year ago

Describe the feature you'd like to have

dm-crypt devices are mounted without discards option by default. this means even if the storage class has the "discards" option set, it will apply between the file system and the mapped device only. the mapped device itself won't discard blocks from the underlying rbd image that is encrypted at rest. the effective result is that encrypted rbd volumes that thin provisioned grow irreversibly which impacts the cost on underlying storage system.

the theoretical risk of exposing information about the size of the encrypted volume is already minimized because thin-provisioned rbd images grow incrementally anyway.

the administrator should have the freedom to decide whether they want the discards option on encrypted volumes or not.

What new functionality do you want?

Allow mounting encrypted rbd volumes with --allow-discards option

What is the value to the end user? (why is it a priority?)

Reduce storage costs in system that use volumes provision by ceph csi with encryption enabled.

How will we know we have a good solution? (acceptance criteria)

A storage class can specify a parameter that tells ceph-csi to mount the encrypted volume with discards enabled.

Additional Context

I run ceph-csi v3.6.1 (rook-operated ceph cluster) and provision volumes with a storage class that has "discard" mount option and enables volume encryption. Before enabling encryption, the volumes were efficient as unused blocks were dynamically reclaimed back to the unused storage space in the OSDs. Since encryption was enabled the volumes grow only and never reclaim unused space. Apparently while the file system does discard unused blocks, the mapped device itself does not discard blocks back to the underlying storage (rbd image). This is a known behavior in dm-crypt: https://wiki.archlinux.org/title/Dm-crypt/Specialties#Discard/TRIM_support_for_solid_state_drives_(SSD)

The administrator should have the freedom to enable dm-crypt discards option has he decided to accept the theoretical risk. Mostly this should not be an issue because rbd images are thin provisioned anyway - again up to the specific system's administrator to decide.

I tested this successfully in my kubernetes 1.23 cluster - exec'ed into csi-rbdplugin daemonset and executed the following command on an already mounted pvc:

echo "<passphrase>" | cryptsetup --allow-discards --persistent refresh /dev/mapper/luks-rbd-...

then I started creating an deleting large files in the volumes - suddenly the rbd image grows and shrinks just like non-encrypted ones.

https://github.com/ceph/ceph-csi/discussions/3561

Rakshith-R commented 1 year ago

@tareqhs Does it make sense to pass it through cmd line args of nodeplugin daemonset pod instead of storageclass?

cc @ceph/ceph-csi-contributors

tareksha commented 1 year ago

@Rakshith-R i think this makes it less convenient. especially that it is off by default (for a reason)

nixpanic commented 1 year ago

On Mon, Dec 05, 2022 at 06:37:07AM -0800, Rakshith R wrote:

@tareqhs Does it make sense to pass it through cmd line args of nodeplugin daemonset pod instead of storageclass?

It should be a StorageClass option. If users want a higher level of security, they would not enable discard for encrypted volumes.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.