"vdo_slab_size_mb" and "vdo_physical_threads" incompatible values ?

[tigerblue77]

Hello there,

I'm encountering a problem with the following LVM VDO metadata profile:

# Custom configuration for VDO not using compression/deduplication
# Custom parameters based on: https://is.muni.cz/th/rq7e2/petrovic_diploma_thesis.pdf

allocation {
  vdo_use_compression=0
  vdo_use_deduplication=0
  vdo_use_metadata_hints=1
  vdo_minimum_io_size=4096
  vdo_block_map_cache_size_mb=16384
  vdo_block_map_period=16380
  vdo_check_point_frequency=0
  vdo_use_sparse_index=0
  vdo_index_memory_size_mb=256
  vdo_slab_size_mb=8192
  vdo_ack_threads=$((VCPU_COUNT/4))
  vdo_bio_threads=$VCPU_COUNT
  vdo_bio_rotation=64
  vdo_cpu_threads=$((VCPU_COUNT/2))
  vdo_hash_zone_threads=$((VCPU_COUNT/4))
  vdo_logical_threads=$((VCPU_COUNT/4))
  vdo_physical_threads=$((VCPU_COUNT/4))
  vdo_write_policy=\"sync\"
  vdo_max_discard=1
}

FYI, on my system:

VCPU_COUNT=8

When I run lvcreate using the previous metadata profile:

DESTINATION_LVM_VG="/dev/Pepper-Potts-vg"
VDOLV_NAME="ZEROED-VDOLV-1"

lvcreate -y --vdopool "ZEROED-VDOPOOL-1" --name $VDOLV_NAME --size 12G --virtualsize 24G --metadataprofile $LVM_METADATA_PROFILE_NAME $DESTINATION_LVM_VG

I get the following error:

Wiping vdo signature on /dev/Pepper-Potts-vg/ZEROED-VDOPOOL-1.
  The VDO volume can address 8 GB in 1 data slab.
  It can grow to address at most 64 TB of physical storage in 8192 slabs.
  If a larger maximum size might be needed, use bigger slabs.
device-mapper: reload ioctl on  (254:6) failed: Erreur d'entrée/sortie
Failed to activate new LV Pepper-Potts-vg/ZEROED-VDOLV-1.
Failed to find logical volume "Pepper-Potts-vg/ZEROED-VDOPOOL-1"

I did some tests and found that the problem comes from these lines :

vdo_slab_size_mb=8192

and

vdo_physical_threads=$((VCPU_COUNT/4))

Here are the results of my different tests:

		vdo_physical_threads value
		1 (default value)	2	4	8
vdo_slab_size_mb value	2048 (default value)	✅	✅	✅	❌
	4096	✅	✅	❌	❌
	8192	✅	❌	❌	❌

I feel that I'm doing something wrong but don't really know what 😅 thanks per advance !

PS: I need 8GB slabs to make my VDO volume ready for 64TB total volume. I choosed 8GB even if 32GB are advised on RedHat documentation because this volume will never need to be expanded more than 64TB.

[lorelei-sakai]

The problem you're running into here is that each physical thread needs to have an least one slab available.

The size of your volume is 12GB, which it only enough space to create one 8GB slab. So you can one have one physical zone. If you want to use 8GB slabs, two physical threads would require at least 17GB, four physical threads would requires 33 GB, etc. (In addition to the slab space, we require some space for metadata, so using exactly 16GB would probably not be enough to create two slabs.)

The number of physical threads can be changed after formatting, so you can decide whether you want to create the VDO with a larger physical size, or start with just one thread.

[tigerblue77]

Thank you very much for this clear explanation ! I didn't find it in any documentation I read, is there any documentation link you can share ?

Also, I'm trying to get the maximum performance out of VDO, what could you advise me ?

About threads, my reasoning was to say: by default, this setting has 1 thread, this one has 2 and this one has 4; so I'll take the number of vCPUs in my system and distribute them according to the same weighting. What do you think of this choice? I also had the idea to simply put the total number of vCPUs for each of these parameters but I was afraid to overload the system

The number of physical threads can be changed after formatting, so you can decide whether you want to create the VDO with a larger physical size, or start with just one thread.

So I suppose that I will have to change my metadata profile and then... How do I apply the updated profile ? Will I need a reboot or another annoying thing like that?

[lorelei-sakai]

About threads, my reasoning was to say: by default, this setting has 1 thread, this one has 2 and this one has 4; so I'll take the number of vCPUs in my system and distribute them according to the same weighting. What do you think of this choice? I also had the idea to simply put the total number of vCPUs for each of these parameters but I was afraid to overload the system

This a very reasonable choice. The optimal settings depend a lot on the characteristics of your hardware and the data access pattern of the volume, so I can't make too many recommendations. The most CPU-intensive work (hashing) happen on the CPU threads, so it's good to have several of those. For physical threads specifically, you should have enough to keep the underlying storage busy. However, if there are too many, they will compete for shared resources and can reduce performance. The effect is likely to be modest in either case, though.

Also, using vdo_write_policy="async" will likely perform better if you don't need the volume to be synchronous. And vdo_use_deduplication=1 should be set so you can actually get deduplication of writes.

The number of physical threads can be changed after formatting, so you can decide whether you want to create the VDO with a larger physical size, or start with just one thread.

So I suppose that I will have to change my metadata profile and then... How do I apply the updated profile ? Will I need a reboot or another annoying thing like that?

You would need to shut the volume down entirely and restart it with the new setting. Also I am not sure lvm has tools yet to allow you to modify this so applying the new setting might be challenging. For now it might be better to create test volumes to try various configurations until you find one that seems to work well.