Allow for striped LVM storage volumes

[danielrobbins]

As a sysadmin, sometimes I would like to be able to create a striped lvm logical volume to use as a backing store for a container to take advantage of its performance characteristics. Currently, it appears that lxd does not provide a mechanism to do this. Special options would need to be passed to lvcreate by lxd to accomplish this. Without these options, the underlying vg will be used as a concatenated (linear) volume rather than a stripe, since lvcreate will not be "told" to set up the logical volume as a stripe.

Proposed solution: at container creation time, provide a mechansim for a sysadmin to specify parameters to be used for creating the thinpool to allow optimal use of the LVM volume group. Also allow a sysadmin to set defaults as desired. I am not sure where the best place for this is, but I could see the option varying per container. But it does need to be set at container creation time.

[stgraber]

A LVM specific option on the storage pool, similar to that we have for thinpool would likely be acceptable, though I don't expect us to have much of an interest in doing this work, so we'd need someone who cares about this to send a pull request.

[danielrobbins]

I could potentially send you a pull request -- can you give me an idea on the optimal way to specify this option -- either command-line -- or where it would be stored -- in which part of the config? And it's preferred name?

[danielrobbins]

Also wondering if maybe there should be some user-friendly option for creation like lxc launch --lvm_mode=striped, which would reference some specific some specific configuration defined somewhere else? Is there any way that these default settings could be placed inside profiles?

[danielrobbins]

I guess my dilemma is this... As a sysadmin, when I do create a container with "special options" for the underlying storage pool, I would like a way -- after the fact -- to be able to "see" that I did this, so I can know that I did it without doing a deep dive into the storage technology to re-discover it.

So I am wondering what the best way would be to store this information in lxd. Maybe a container-specific configuration variable that can only be set on creation time, but is immutable after the container is created? If you can give me some hints about this part and the best way to approach it from an lxd perspective, then I should be able to come up with a patch that is more suitable to be merged upstream, rather than a "hack" :)

[stgraber]

I think we'd want a volume.lvm.striped boolean option at the storage pool level which would then affect any LV created after the value is changed. That value is then copied as lvm.striped on the storage volume as a read-only property.

For custom storage volumes, that'd let you do something like lxc storage create default blah lvm.striped=true to override it just for that one custom volume.

For containers, there is no easy way to do so right now, but you'd be able to temporarily flip volume.lvm.striped on your storage pool, switching it to true, create the container and switch back to false.

[mparfan]

Hi, I'm a student taking a virtualization class at UT Austin, and I'm currently collaborating with two others. @stgraber Is it possible that we can get more information about this issue if it's still available?

[stgraber]

So my understanding of that LVM feature is pretty limited so we'll need @danielrobbins to confirm that this would work for him. My understanding is that for striping, you'll want to configure both the number of stripes and the stripe size.

For LXD, we'll want this applied on a per-volume basis with support for a default volume as would typically be used for creating a new container from an image.

So I think we'd be looking at adding two pool config keys:

• volume.lvm.stripes (must be >= 1, defaults to 1, no striping options passed if it's <= 1)
• volume.lvm.stripes.size (amount of bytes (with our usual suffix support), default is empty and causes lvcreate to use the default)

And the matching keys for the volumes:

• lvm.stripes
• lvm.stripes.size

If lvm.stripes is set to > 1, then the -i option will be passed to lvcreate, if lvm.stripes.size is set as well, then both -i and -I will be passed.

[rcash]

Thank you for the additional info! I'm one of @mparfan 's collaborators, we're interested in this one as well, just pending instructor approval.

[stgraber]

Cool, let me know when you get the go ahead and I'll assign it to the two of you.

[rcash]

Hi @stgraber , we're approved! Could you assign @lay-ton @mparfan and I?

[stgraber]

Hi @rcash I've assigned it to you and @mparfan, I can't assign to @Lay-ton as Github won't let me assign to someone who hasn't commented in the issue.

[Lay-ton]

Hello, could you assign me as well?

[rcash]

Hi @stgraber, I just wanted to give a status update since we've been silent for a few days and need some help/clarification on setting the volume keys.

We've spent awhile reading through files relevant to the issue and have added the storage pool config keys (volume.lvm.stripes and volume.lvm.stripes.size) and currently set the default number of stripes (volume.lvm.stripes) in StoragePoolCreate(). Where should the corresponding volume keys (changeableStoragePoolVolumeProperties for these be set? When the storage pool is created, when the volume is created, or another place? Our current assumption is the user will do something like lxc storage create poolA lvm volume.lvm.stripes=2 volume.lvm.stripes.size=64kB and the volumes that follow suit from this command will be striped like that as a result.

Edit: We are currently setting the default for number of stripes (volume.lvm.stripes) in StoragePoolInit() as well but have removed that since we know it is incorrect, that is not reflected in the repo linked above.

[stgraber]

lvm.stripes and lvm.stripes.size should go in changeableStoragePoolVolumeProperties, StorageVolumeConfigKeys and VolumeValidateConfig.

One quick way to see pretty much everywhere changes are needed would be to look for zfs.use_refquota which is a similar key in the zfs backend.

stgraber@castiana:~/data/code/lxc/lxd (lxc/master)$ grep -r zfs\.use_refquota . | grep -v storage\.zfs
./lxd/patches.go:           if !shared.IsTrue(pool.Config["volume.zfs.use_refquota"]) {
./lxd/patches.go:               pool.Config["volume.zfs.use_refquota"] = ""
./lxd/patches.go:               if !shared.IsTrue(volume.Config["zfs.use_refquota"]) {
./lxd/patches.go:                   volume.Config["zfs.use_refquota"] = ""
./lxd/storage_pools_config.go:      "volume.zfs.use_refquota",
./lxd/storage_pools_config.go:  "volume.zfs.use_refquota":     shared.IsBool,
./lxd/storage_volumes_config.go:        "zfs.use_refquota",
./lxd/storage/utils.go: "zfs.use_refquota": func(value string) ([]string, error) {
./lxd/storage/utils.go:         if config["zfs.use_refquota"] != "" {
./lxd/storage/utils.go:             return fmt.Errorf("the key volume.zfs.use_refquota cannot be used with non zfs storage volumes")
./test/suites/storage.sh:      lxc storage create "lxdtest-$(basename "${LXD_DIR}")-valid-zfs-pool-config" zfs volume.zfs.use_refquota=true
./test/suites/storage.sh:      ! lxc storage create "lxdtest-$(basename "${LXD_DIR}")-invalid-btrfs-pool-config" btrfs volume.zfs.use_refquota=true || false
./test/suites/storage.sh:    ! lxc storage create "lxdtest-$(basename "${LXD_DIR}")-invalid-dir-pool-config" dir volume.zfs.use_refquota=true || false
./test/suites/storage.sh:      ! lxc storage create "lxdtest-$(basename "${LXD_DIR}")-invalid-lvm-pool-config" lvm volume.zfs_use_refquota=true || false
./test/suites/storage.sh:      lxc storage volume set "lxdtest-$(basename "${LXD_DIR}")-pool1" c1pool1 zfs.use_refquota true
./scripts/bash/lxd-client:      volume.zfs.use_refquota zfs.clone_copy zfs.pool_name"
./scripts/bash/lxd-client:      security.unmapped security.shifted zfs.remove_snapshots zfs.use_refquota"

[Lay-ton]

Hi @stgraber , we've implemented all the logic that we think is necessary to support striped volumes, however, we're running into some issues testing lvcreate-are there instances where where striping is not valid on thinly provisioned volumes (such on when there is only one physical volume in a volume group)? This is the error we've been seeing when trying to create a striped thin volume here. Error: Error Creating LVM LV for new image: Could not create thin LV named custom_vol0: Failed to run: lvcreate -i 2 -I 64K -Wy --yes --thin -n custom_vol0 --virtualsize 10GB poolA/LXDThinPool: Command does not accept option: --stripes 2. Command does not accept option: --stripesize 64K This is the underlying volume group: VG #PV #LV #SN Attr VSize VFree poolA 1 1 0 wz--n- <27.94g 0 We understand you mentioned not knowing much about this part of lvm, but any help from you or someone you could recommend would be appreciated.

[tomponline]

Does it work when using normal volumes rather than thin pools?

[tomponline]

Yes it looks like when using thin volumes its the thin pool lvm that needs the stripe settings.

"A thin pool can be striped like any other logical volume. All the thin volumes created from the pool inherit that settings - do not specify it manually when creating a thin volume. " https://wiki.gentoo.org/wiki/LVM