search

gluster / gluster-kubernetes

GlusterFS Native Storage Service for Kubernetes
Apache License 2.0
874 stars 390 forks source link

Heketi can not expand volume as Failed to allocate volume expansion: Maximum number of bricks reached #628

Closed Ayden-Franklin closed 4 years ago

Ayden-Franklin commented 4 years ago
I have a gluster managed by heketi running on kubernetes. Today, I added one device (/dev/sdc 300G) for each node for expanding the volume. But it said Failed to allocate volume expansion: Maximum number of bricks reached. Component Version
k8s 1.16.2
Hekiti heketi/heketi:dev

The existing volume information:

Name: xxxxxxx-gluster-file
Size: 200
Volume Id: 3b05ef061fadc0291bf4b315f1bc69f6
Cluster Id: c90f411e9274f2c386bfd9081d59d7c5
Mount: 10.168.0.105:runsdata-gluster-file
Mount Options: backup-volfile-servers=10.168.0.104
Block: false
Free Size: 0
Reserved Size: 0
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: replicate
Distribute Count: 1
Replica Count: 2

I tried to expand volume using this command:

 heketi-cli volume expand --volume=3b05ef061fadc0291bf4b315f1bc69f6  --user admin --secret <****key***> --expand-size=250
Error: Failed to allocate volume expansion: Maximum number of bricks reached.

I noticed that the new device has no bricks.


    Nodes:

    Node Id: 06b85d7d1c3e9011bd01e23e96bffe49
    State: online
    Cluster Id: c90f411e9274f2c386bfd9081d59d7c5
    Zone: 1
    Management Hostnames: 10.168.0.105
    Storage Hostnames: 10.168.0.105
    Devices:
        Id:94c3a9ae7842c7cdb20f9ab4c9ee72d2   Name:/dev/sdc            State:online    Size (GiB):299     Used (GiB):251     Free (GiB):48      
            Bricks:
        Id:adab7eb4d804d5202051bd53e30f0451   Name:/dev/sdb            State:online    Size (GiB):499     Used (GiB):402     Free (GiB):97      
            Bricks:
                Id:1b177e6160aa7d223d15ea193916a03f   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_adab7eb4d804d5202051bd53e30f0451/brick_1b177e6160aa7d223d15ea193916a03f/brick
                Id:264fa838fa4b09413f8792640ec146c0   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_adab7eb4d804d5202051bd53e30f0451/brick_264fa838fa4b09413f8792640ec146c0/brick
                Id:26a1c90362ffbfc16e110e77515e226f   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_adab7eb4d804d5202051bd53e30f0451/brick_26a1c90362ffbfc16e110e77515e226f/brick
                Id:2932dc41ff7e682d7babae7602645d92   Size (GiB):100     Path: /var/lib/heketi/mounts/vg_adab7eb4d804d5202051bd53e30f0451/brick_2932dc41ff7e682d7babae7602645d92/brick
                Id:4e2d4087447954e5ade7fddf106b3af2   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_adab7eb4d804d5202051bd53e30f0451/brick_4e2d4087447954e5ade7fddf106b3af2/brick
                Id:6735fd44ee294a955898189f8971e9e1   Size (GiB):200     Path: /var/lib/heketi/mounts/vg_adab7eb4d804d5202051bd53e30f0451/brick_6735fd44ee294a955898189f8971e9e1/brick
                Id:7c8dfe5665daee16b03b156c69442429   Size (GiB):20      Path: /var/lib/heketi/mounts/vg_adab7eb4d804d5202051bd53e30f0451/brick_7c8dfe5665daee16b03b156c69442429/brick
                Id:806bae061ac85d9b13aef42a8136eeb9   Size (GiB):10      Path: /var/lib/heketi/mounts/vg_adab7eb4d804d5202051bd53e30f0451/brick_806bae061ac85d9b13aef42a8136eeb9/brick
                Id:8b3be246c37c8e7080655af33f52b9de   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_adab7eb4d804d5202051bd53e30f0451/brick_8b3be246c37c8e7080655af33f52b9de/brick
                Id:b46f7a867bbcc61f81278c04f2dcd56f   Size (GiB):10      Path: /var/lib/heketi/mounts/vg_adab7eb4d804d5202051bd53e30f0451/brick_b46f7a867bbcc61f81278c04f2dcd56f/brick
                Id:dd38cb2261a7e4dc09facbc4673cb09d   Size (GiB):50      Path: /var/lib/heketi/mounts/vg_adab7eb4d804d5202051bd53e30f0451/brick_dd38cb2261a7e4dc09facbc4673cb09d/brick

    Node Id: a7c905ecb1e58c3ae6ffa6c608a15ebd
    State: online
    Cluster Id: c90f411e9274f2c386bfd9081d59d7c5
    Zone: 1
    Management Hostnames: 10.168.0.104
    Storage Hostnames: 10.168.0.104
    Devices:
        Id:2c9acdd4cb4c37d96a61bb526d1feb57   Name:/dev/sdc            State:online    Size (GiB):299     Used (GiB):251     Free (GiB):48      
            Bricks:
        Id:dc0b53e0f31376bb47b453a931521ae4   Name:/dev/sdb            State:online    Size (GiB):499     Used (GiB):404     Free (GiB):95      
            Bricks:
                Id:039121bf4b452a2b4ea0fbb41f78ac00   Size (GiB):20      Path: /var/lib/heketi/mounts/vg_dc0b53e0f31376bb47b453a931521ae4/brick_039121bf4b452a2b4ea0fbb41f78ac00/brick
                Id:0ba34f682b9e5cd8f3c1fa382ab5a04e   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_dc0b53e0f31376bb47b453a931521ae4/brick_0ba34f682b9e5cd8f3c1fa382ab5a04e/brick
                Id:3a3e589af06a23502d812afac3a20565   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_dc0b53e0f31376bb47b453a931521ae4/brick_3a3e589af06a23502d812afac3a20565/brick
                Id:4a1e1e0580eb2a55f2836383c9b47340   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_dc0b53e0f31376bb47b453a931521ae4/brick_4a1e1e0580eb2a55f2836383c9b47340/brick
                Id:8bfc13dad849a912f4f895a99a2f2f9d   Size (GiB):10      Path: /var/lib/heketi/mounts/vg_dc0b53e0f31376bb47b453a931521ae4/brick_8bfc13dad849a912f4f895a99a2f2f9d/brick
                Id:a0d3cd52b91f7c505371e662542734ea   Size (GiB):10      Path: /var/lib/heketi/mounts/vg_dc0b53e0f31376bb47b453a931521ae4/brick_a0d3cd52b91f7c505371e662542734ea/brick
                Id:a86eef2f8be8f1321b317a4882e873ad   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_dc0b53e0f31376bb47b453a931521ae4/brick_a86eef2f8be8f1321b317a4882e873ad/brick
                Id:b7c8315581335f3b525f015c0f7bc6ea   Size (GiB):200     Path: /var/lib/heketi/mounts/vg_dc0b53e0f31376bb47b453a931521ae4/brick_b7c8315581335f3b525f015c0f7bc6ea/brick
                Id:c7c3d09e99cdd3603e4e2aa95f93e8f4   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_dc0b53e0f31376bb47b453a931521ae4/brick_c7c3d09e99cdd3603e4e2aa95f93e8f4/brick
                Id:ee197858edc0e3429af207e6fc07d1bd   Size (GiB):100     Path: /var/lib/heketi/mounts/vg_dc0b53e0f31376bb47b453a931521ae4/brick_ee197858edc0e3429af207e6fc07d1bd/brick
                Id:ee8f161952e860466f84e2565123e8d6   Size (GiB):50      Path: /var/lib/heketi/mounts/vg_dc0b53e0f31376bb47b453a931521ae4/brick_ee8f161952e860466f84e2565123e8d6/brick
                Id:f50c7da046b8e590ab8a9b73e232e5ba   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_dc0b53e0f31376bb47b453a931521ae4/brick_f50c7da046b8e590ab8a9b73e232e5ba/brick

And the heketi pod output:

[heketi] ERROR 2019/12/26 08:22:33 heketi/apps/glusterfs/operations_manage.go:220:glusterfs.AsyncHttpOperation: Expand Volume Build Failed: Maximum number of bricks reached.
[negroni] 2019-12-26T08:22:33Z | 500 |   5.561901ms | 10.0.6.76:8080 | POST /volumes/3b05ef061fadc0291bf4b315f1bc69f6/expand
Ayden-Franklin commented 4 years ago

The /dev/sdc is new device added by me just now. What is inconceivable is that it had been used 250G. And there is no Bricks listed below it.

Ayden-Franklin commented 4 years ago

I solved this problem. I added the device by adding "/dev/sdc" to the device element in the topology.json. Unfortunately, it was marked as have been used for 251G. Today, I added the device by the command heketi-cli device add --name=/dev/sdc --node=xx and it works well.

Ayden-Franklin commented 4 years ago

The only one strange thing is there are two devices /dev/sdc on that node.

    Devices:
        Id:570107c155012c1ef9f3942248c1a194   Name:/dev/sdc            State:online    Size (GiB):299     Used (GiB):251     Free (GiB):48      
            Bricks:
                Id:948c4a2751e19e0643f105a1053b86d0   Size (GiB):250     Path: /var/lib/heketi/mounts/vg_570107c155012c1ef9f3942248c1a194/brick_948c4a2751e19e0643f105a1053b86d0/brick
        Id:94c3a9ae7842c7cdb20f9ab4c9ee72d2   Name:/dev/sdc            State:online    Size (GiB):299     Used (GiB):251     Free (GiB):48      
            Bricks:
        Id:adab7eb4d804d5202051bd53e30f0451   Name:/dev/sdb            State:online    Size (GiB):499     Used (GiB):402     Free (GiB):97      
            Bricks:
                Id:1b177e6160aa7d223d15ea193916a03f   Size (GiB):2       Path: /var/lib/heketi/mounts/vg_adab7eb4d804d5202051bd53e30f0451/brick_1b177e6160aa7d223d15ea193916a03f/brick