Getting zfs exception when resizing due to incompatible size to to volblocksize

[dniasoff]

Hi,

Using Linbit Server 1.27.0 on Ubuntu 22.04 with OpenZFS 2.2.2.

I have it integrated with OpenStack Cinder.

When creating a volume from an image with volume image cache enabled - part of the workflow creates a small volume and resizing it.

However, I am getting the following error inside Linbit.

Reported error:
===============

Category:                           LinStorException
Class name:                         StorageException
Class canonical name:               com.linbit.linstor.storage.StorageException
Generated at:                       Method 'checkExitCode', Source file 'ExtCmdUtils.java', Line #69

Error message:                      Failed to resize zfs volume

Error context:
        An error occurred while processing resource 'Node: 'node01.cloud5.cloudportal.app', Rsc: 'volume-251a8beb-d1e1-4200-9ebe-1a86c8089817''
ErrorContext:
  Details:     Command 'zfs set volsize=5244040KB pool1/volume-251a8beb-d1e1-4200-9ebe-1a86c8089817_00000' returned with exitcode 255.

Standard out:

Error message:
cannot set property for 'pool1/volume-251a8beb-d1e1-4200-9ebe-1a86c8089817_00000': 'volsize' must be a multiple of volume block size (16K)

Call backtrace:

    Method                                   Native Class:Line number
    checkExitCode                            N      com.linbit.extproc.ExtCmdUtils:69
    genericExecutor                          N      com.linbit.linstor.storage.utils.Commands:103
    genericExecutor                          N      com.linbit.linstor.storage.utils.Commands:63
    genericExecutor                          N      com.linbit.linstor.storage.utils.Commands:51
    resize                                   N      com.linbit.linstor.layer.storage.zfs.utils.ZfsCommands:121
    resizeLvImpl                             N      com.linbit.linstor.layer.storage.zfs.ZfsProvider:317
    resizeLvImpl                             N      com.linbit.linstor.layer.storage.zfs.ZfsProvider:67
    resizeVolumes                            N      com.linbit.linstor.layer.storage.AbsStorageProvider:717
    processVolumes                           N      com.linbit.linstor.layer.storage.AbsStorageProvider:361
    processResource                          N      com.linbit.linstor.layer.storage.StorageLayer:282
    lambda$processResource$4                 N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:908
    processGeneric                           N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:949
    processResource                          N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:904
    processChild                             N      com.linbit.linstor.layer.drbd.DrbdLayer:323
    adjustDrbd                               N      com.linbit.linstor.layer.drbd.DrbdLayer:447
    processResource                          N      com.linbit.linstor.layer.drbd.DrbdLayer:250
    lambda$processResource$4                 N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:908
    processGeneric                           N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:949
    processResource                          N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:904
    processResources                         N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:370
    dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceHandlerImpl:217
    dispatchResources                        N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:331
    phaseDispatchDeviceHandlers              N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:1204
    devMgrLoop                               N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:778
    run                                      N      com.linbit.linstor.core.devmgr.DeviceManagerImpl:672
    run                                      N      java.lang.Thread:829

END OF ERROR REPORT.

The volblocksize is a default setting on my system - I haven't changed it anywhere.

Can I suggest the neatest approach would be for Linstor to check the volblocksize and then roundup the requested size.

Thanks

Daniel

[ghernadi]

Thanks for the report, we are aware of this issue with newer ZFS versions and are already working on a fix for this issue.

In order to get LINSTOR out of the resizing state, I'd suggest the following steps:

• Resize ZFS by hand:
  That means, take the command from the ErrorReport, add 8k to the size so it is a multiple of 16k and run the command manually on the node (or all nodes) that reported this error.
• (Optional) Find out the exact size of the VD so that the next command does not additionally grow your VD:
  This step is obviously only needed if you are unsure about the exact size of your VD. If you have just resized your VD to for example 100GiB, feel free to skip this step and use the very same size argument again in the next step. To find out the exact size of a VD is to check linstor -m vd l -r $rsc_name and look for volume_definitions[$your_vlm_nr].size_kib. If you have jq installed and only have a single VD for the given resource, you could use something like this:
  linstor -m vd l -r $rsc_name | jq '.[0][0].volume_definitions[0].size_kib'.
  If you are using this method, please make sure to also properly use the KiB suffix for the size in the next step.
• Re-run the linstor vd size ... command.
  This should make LINSTOR find the ZFS volume is already in the expected state, resize DRBD (if needed) and properly finish the resize operation by removing the flags from the VD.

[dniasoff]

Thanks for a detailed and helpful response.

For now I have configured the storage pool to use a volblocksize of 4k which seems to solve the problem. I am guessing this might have an impact on disk usage/performance so look forward to the fix. This means any volumes already created before the fix will be stuck with a 4k volblocksize.