search

LINBIT / linstor-server

High Performance Software-Defined Block Storage for container, cloud and virtualisation. Fully integrated with Docker, Kubernetes, Openstack, Proxmox etc.
https://docs.linbit.com/docs/linstor-guide/
GNU General Public License v3.0
975 stars 76 forks source link

LINSTOR creates volumes larger than requested size #421

Open kvaps opened 1 week ago

kvaps commented 1 week ago

I create PersistentVolumeClaim with size of 1Gi:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: block-pvc
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Block
  resources:
    requests:
      storage: 1Gi

1Gi*1024*1024*1024 = 1073741824 bytes

But inside the pod the size of volume is only 1073750016:

/ # lsblk -bno SIZE /dev/drbd1014
1073750016
/ # qemu-img info /dev/drbd1014
image: /dev/drbd1014
file format: raw
virtual size: 1 GiB (1073750016 bytes)
disk size: 0 B
Child node '/file':
    filename: /dev/drbd1014
    protocol type: host_device
    file length: 1 GiB (1073750016 bytes)
    disk size: 0 B

link https://github.com/kubevirt/containerized-data-importer/issues/3159

kvaps commented 1 week ago

In LINSTOR the volumes have different 

                "allocated_size_kib": 20976040,
                "usable_size_kib": 20971528

Is it possible to order volumes in LINSTOR using usable_size, not allocated_size?

kvaps commented 1 week ago

I just tried to use --gross but created volume even smaller than without this flag

kvaps commented 1 week ago

Is there ant reliable way to request LINSTOR create a DRBD volume with exact size 1Gi (1073741824 bytes) that can be 100% usable by the client application?

ghernadi commented 6 days ago

Correct me if I'm wrong, but reading the linked issue it does not look like the question / request here is that LINSTOR should not give you a little bit more space, but exactly the space you asked for, right?

If so, please compare the sizes from linstor -m vd l | grep size_kib and linstor -m v l | grep '"allocated_size_kib":'. The volume-definition's size is exactly the size the user asked for, whereas the allocated_size_kib as well as the usable_size_kib might be a bit larger than the user wanted due to LVM / ZFS rounding up to the next extent size.

Getting back to your example, if you ask LINSTOR for a 1GiB large volume (i.e. 1073741824 bytes), you will get a bit more than 1GiB volume, in your case 1073750016. 1073750016 - 1073741824 = 8192, which means you got 8KiB more than requested.

Am I missing something or is the title of this issue the wrong way around?

kvaps commented 6 days ago

Hi @ghernadi thank you for investigating, but seems I was wrong here, here is my pvc:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prime-dbabbeef-a3f4-412e-aec6-cdbc8acd2c05
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: "1073741824"
  storageClassName: replicated
  volumeMode: Block
  volumeName: pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a
# linstor vd l -r pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a
+-------------------------------------------------------------------------------------------+
| ResourceName                             | VolumeNr | VolumeMinor | Size  | Gross | State |
|===========================================================================================|
| pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a | 0        | 1000        | 1 GiB |       | ok    |
+-------------------------------------------------------------------------------------------+
# linstor v l -r pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a
+---------------------------------------------------------------------------------------------------------------------------------+
| Node | Resource                                 | StoragePool | VolNr | MinorNr | DeviceName    | Allocated | InUse  |    State |
|=================================================================================================================================|
| srv1 | pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a | data        |     0 |    1000 | /dev/drbd1000 |  1.00 GiB | Unused | UpToDate |
| srv2 | pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a | data        |     0 |    1000 | /dev/drbd1000 |  1.00 GiB | InUse  | UpToDate |
| srv3 | pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a | data        |     0 |    1000 | /dev/drbd1000 |  1.00 GiB | Unused | UpToDate |
+---------------------------------------------------------------------------------------------------------------------------------+
# linstor -m v l -r pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a
[
  [
    {
      "name": "pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a",
      "node_name": "srv1",
      "props": {
        "StorPoolName": "data"
      },
      "layer_object": {
        "children": [
          {
            "type": "STORAGE",
            "storage": {
              "storage_volumes": [
                {
                  "volume_number": 0,
                  "device_path": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
                  "allocated_size_kib": 1048848,
                  "usable_size_kib": 1048848,
                  "disk_state": "[]"
                }
              ]
            }
          }
        ],
        "type": "DRBD",
        "drbd": {
          "drbd_resource_definition": {
            "peer_slots": 7,
            "al_stripes": 1,
            "al_stripe_size_kib": 32,
            "port": 7001,
            "transport_type": "IP",
            "secret": "uV4aAOXbn/YtSIryS9yf",
            "down": false
          },
          "node_id": 1,
          "peer_slots": 7,
          "al_stripes": 1,
          "al_size": 32,
          "flags": [
            "INITIALIZED"
          ],
          "drbd_volumes": [
            {
              "drbd_volume_definition": {
                "volume_number": 0,
                "minor_number": 1000
              },
              "device_path": "/dev/drbd1000",
              "backing_device": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
              "allocated_size_kib": 1048840,
              "usable_size_kib": 1048584
            }
          ],
          "connections": {
            "srv3": {
              "connected": true,
              "message": "Connected"
            },
            "srv2": {
              "connected": true,
              "message": "Connected"
            }
          },
          "promotion_score": 10103,
          "may_promote": false
        }
      },
      "state": {
        "in_use": false
      },
      "uuid": "98460b7d-97cb-44d7-8022-2970006aa361",
      "create_timestamp": 1728646426440,
      "volumes": [
        {
          "volume_number": 0,
          "storage_pool_name": "data",
          "provider_kind": "ZFS",
          "device_path": "/dev/drbd1000",
          "allocated_size_kib": 1048848,
          "state": {
            "disk_state": "UpToDate"
          },
          "layer_data_list": [
            {
              "type": "DRBD",
              "data": {
                "drbd_volume_definition": {
                  "volume_number": 0,
                  "minor_number": 1000
                },
                "device_path": "/dev/drbd1000",
                "backing_device": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
                "allocated_size_kib": 1048840,
                "usable_size_kib": 1048584
              }
            },
            {
              "type": "STORAGE",
              "data": {
                "volume_number": 0,
                "device_path": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
                "allocated_size_kib": 1048848,
                "usable_size_kib": 1048848,
                "disk_state": "[]"
              }
            }
          ],
          "uuid": "293063cd-fdd3-4e45-9178-802c61832290"
        }
      ]
    },
    {
      "name": "pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a",
      "node_name": "srv2",
      "props": {
        "StorPoolName": "data"
      },
      "layer_object": {
        "children": [
          {
            "type": "STORAGE",
            "storage": {
              "storage_volumes": [
                {
                  "volume_number": 0,
                  "device_path": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
                  "allocated_size_kib": 1048848,
                  "usable_size_kib": 1048848,
                  "disk_state": "[]"
                }
              ]
            }
          }
        ],
        "type": "DRBD",
        "drbd": {
          "drbd_resource_definition": {
            "peer_slots": 7,
            "al_stripes": 1,
            "al_stripe_size_kib": 32,
            "port": 7001,
            "transport_type": "IP",
            "secret": "uV4aAOXbn/YtSIryS9yf",
            "down": false
          },
          "node_id": 0,
          "peer_slots": 7,
          "al_stripes": 1,
          "al_size": 32,
          "flags": [
            "INITIALIZED"
          ],
          "drbd_volumes": [
            {
              "drbd_volume_definition": {
                "volume_number": 0,
                "minor_number": 1000
              },
              "device_path": "/dev/drbd1000",
              "backing_device": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
              "allocated_size_kib": 1048840,
              "usable_size_kib": 1048584
            }
          ],
          "connections": {
            "srv3": {
              "connected": true,
              "message": "Connected"
            },
            "srv1": {
              "connected": true,
              "message": "Connected"
            }
          },
          "promotion_score": 10103,
          "may_promote": false
        }
      },
      "state": {
        "in_use": true
      },
      "uuid": "9cffb7f6-bea8-48a0-9906-d74e8eb2694c",
      "create_timestamp": 1728646425721,
      "volumes": [
        {
          "volume_number": 0,
          "storage_pool_name": "data",
          "provider_kind": "ZFS",
          "device_path": "/dev/drbd1000",
          "allocated_size_kib": 1048848,
          "state": {
            "disk_state": "UpToDate"
          },
          "layer_data_list": [
            {
              "type": "DRBD",
              "data": {
                "drbd_volume_definition": {
                  "volume_number": 0,
                  "minor_number": 1000
                },
                "device_path": "/dev/drbd1000",
                "backing_device": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
                "allocated_size_kib": 1048840,
                "usable_size_kib": 1048584
              }
            },
            {
              "type": "STORAGE",
              "data": {
                "volume_number": 0,
                "device_path": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
                "allocated_size_kib": 1048848,
                "usable_size_kib": 1048848,
                "disk_state": "[]"
              }
            }
          ],
          "uuid": "66fbfb01-a4da-4643-8b50-dc33e97cf18d"
        }
      ]
    },
    {
      "name": "pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a",
      "node_name": "srv3",
      "props": {
        "StorPoolName": "data"
      },
      "layer_object": {
        "children": [
          {
            "type": "STORAGE",
            "storage": {
              "storage_volumes": [
                {
                  "volume_number": 0,
                  "device_path": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
                  "allocated_size_kib": 1048848,
                  "usable_size_kib": 1048848,
                  "disk_state": "[]"
                }
              ]
            }
          }
        ],
        "type": "DRBD",
        "drbd": {
          "drbd_resource_definition": {
            "peer_slots": 7,
            "al_stripes": 1,
            "al_stripe_size_kib": 32,
            "port": 7001,
            "transport_type": "IP",
            "secret": "uV4aAOXbn/YtSIryS9yf",
            "down": false
          },
          "node_id": 2,
          "peer_slots": 7,
          "al_stripes": 1,
          "al_size": 32,
          "flags": [
            "INITIALIZED"
          ],
          "drbd_volumes": [
            {
              "drbd_volume_definition": {
                "volume_number": 0,
                "minor_number": 1000
              },
              "device_path": "/dev/drbd1000",
              "backing_device": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
              "allocated_size_kib": 1048840,
              "usable_size_kib": 1048584
            }
          ],
          "connections": {
            "srv1": {
              "connected": true,
              "message": "Connected"
            },
            "srv2": {
              "connected": true,
              "message": "Connected"
            }
          },
          "promotion_score": 10103,
          "may_promote": false
        }
      },
      "state": {
        "in_use": false
      },
      "uuid": "b9441c59-3f16-413e-b8b1-b73918d098f0",
      "create_timestamp": 1728646426396,
      "volumes": [
        {
          "volume_number": 0,
          "storage_pool_name": "data",
          "provider_kind": "ZFS",
          "device_path": "/dev/drbd1000",
          "allocated_size_kib": 1048848,
          "state": {
            "disk_state": "UpToDate"
          },
          "layer_data_list": [
            {
              "type": "DRBD",
              "data": {
                "drbd_volume_definition": {
                  "volume_number": 0,
                  "minor_number": 1000
                },
                "device_path": "/dev/drbd1000",
                "backing_device": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
                "allocated_size_kib": 1048840,
                "usable_size_kib": 1048584
              }
            },
            {
              "type": "STORAGE",
              "data": {
                "volume_number": 0,
                "device_path": "/dev/zvol/data/pvc-5b902b3e-24ef-431d-afdd-0746414a3e1a_00000",
                "allocated_size_kib": 1048848,
                "usable_size_kib": 1048848,
                "disk_state": "[]"
              }
            }
          ],
          "uuid": "7fa0181c-3522-4a44-826c-ea027b35558e"
        }
      ]
    }
  ]
]

For DRBD layer it is:

                "allocated_size_kib": 1048840,
                "usable_size_kib": 1048584
/ # lsblk -bno SIZE /dev/drbd1000
1073750016
>>> 1073750016/1024/1024/1024
1.0000076293945312

It seems CDI issue, I will continue investigating, and let you know. /close for now

kvaps commented 6 days ago

It seems there is a bug (or feature?) in CDI. It expects that volume will have exact size that requested in PVC.

Personally I don't see the problem if the drive will be a n bit larger than expected, but it might affect some mechanics of copying PVCs between each other.

I didn't dig so far, but I see a potential issue, if CDI creates a new PVC with the same size as origin (and it reads this size from the PVC spec), but in fact that origin have larger volume inside it might block copying operations.

kvaps commented 5 days ago

I think LINSTOR should create drives with exact requested size from PVC, if it is not possible due to technical limitations, it should at least report correct usable_size into created PV after the volume is created