search

LINBIT / linstor-server

High Performance Software-Defined Block Storage for container, cloud and virtualisation. Fully integrated with Docker, Kubernetes, Openstack, Proxmox etc.
https://docs.linbit.com/docs/linstor-guide/
GNU General Public License v3.0
984 stars 76 forks source link

Resizing a volume used external metadata fails even though the storage pool has enough free space #360

Open kanno2inf opened 1 year ago

kanno2inf commented 1 year ago

Growing volume fails due to metadata pool capacity exceeded even though the storage pool has enough free space.

Error report

$ linstor error-reports show 64895E66-00000-000000
ERROR REPORT 64895E66-00000-000000

============================================================

Application:                        LINBIT® LINSTOR
Module:                             Controller
Version:                            1.23.0
Build ID:                           28dbd33ced60d75a2a0562bf5e9bc6b800ae8361
Build time:                         2023-05-23T07:06:58+00:00
Error time:                         2023-06-14 15:30:38
Node:                               sn1
Peer:                               RestClient(0:0:0:0:0:0:0:1; 'PythonLinstor/1.18.0 (API1.0.4): Client 1.18.0')

============================================================

Reported error:
===============

Category:                           RuntimeException
Class name:                         ApiRcException
Class canonical name:               com.linbit.linstor.core.apicallhandler.response.ApiRcException
Generated at:                       Method 'ensureAllStorPoolsHaveEnoughFreeSpace', Source file 'CtrlVlmDfnModifyApiCallHandler.java', Line #671

Error message:                      Cannot grow the volume definition by 4091904KiB, as the following storage pool do not have enough free space:
Node: sn1, StorPool: meta
Node: sn2, StorPool: meta

Error context:
    Cannot grow the volume definition by 4091904KiB, as the following storage pool do not have enough free space:
    Node: sn1, StorPool: meta
    Node: sn2, StorPool: meta

Asynchronous stage backtrace:

    Error has been observed at the following site(s):
        |_ checkpoint ⇢ Modify volume definition
    Stack trace:

Call backtrace:

    Method                                   Native Class:Line number
    ensureAllStorPoolsHaveEnoughFreeSpace    N      com.linbit.linstor.core.apicallhandler.controller.CtrlVlmDfnModifyApiCallHandler:671

Suppressed exception 1 of 1:
===============
Category:                           RuntimeException
Class name:                         OnAssemblyException
Class canonical name:               reactor.core.publisher.FluxOnAssembly.OnAssemblyException
Generated at:                       Method 'ensureAllStorPoolsHaveEnoughFreeSpace', Source file 'CtrlVlmDfnModifyApiCallHandler.java', Line #671

Error message:
Error has been observed at the following site(s):
        |_ checkpoint ⇢ Modify volume definition
Stack trace:

Error context:
    Cannot grow the volume definition by 4091904KiB, as the following storage pool do not have enough free space:
    Node: sn1, StorPool: meta
    Node: sn2, StorPool: meta

Call backtrace:

    Method                                   Native Class:Line number
    ensureAllStorPoolsHaveEnoughFreeSpace    N      com.linbit.linstor.core.apicallhandler.controller.CtrlVlmDfnModifyApiCallHandler:671
    modifyVlmDfnInTransaction                N      com.linbit.linstor.core.apicallhandler.controller.CtrlVlmDfnModifyApiCallHandler:315
    lambda$modifyVlmDfn$0                    N      com.linbit.linstor.core.apicallhandler.controller.CtrlVlmDfnModifyApiCallHandler:179
    doInScope                                N      com.linbit.linstor.core.apicallhandler.ScopeRunner:149
    lambda$fluxInScope$0                     N      com.linbit.linstor.core.apicallhandler.ScopeRunner:76
    call                                     N      reactor.core.publisher.MonoCallable:91
    trySubscribeScalarMap                    N      reactor.core.publisher.FluxFlatMap:126
    subscribeOrReturn                        N      reactor.core.publisher.MonoFlatMapMany:49
    subscribe                                N      reactor.core.publisher.Flux:8343
    onNext                                   N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:188
    request                                  N      reactor.core.publisher.Operators$ScalarSubscription:2344
    onSubscribe                              N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:134
    subscribe                                N      reactor.core.publisher.MonoCurrentContext:35
    subscribe                                N      reactor.core.publisher.Flux:8357
    onNext                                   N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:188
    request                                  N      reactor.core.publisher.Operators$ScalarSubscription:2344
    onSubscribe                              N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:134
    subscribe                                N      reactor.core.publisher.MonoCurrentContext:35
    subscribe                                N      reactor.core.publisher.Mono:4252
    subscribeWith                            N      reactor.core.publisher.Mono:4363
    subscribe                                N      reactor.core.publisher.Mono:4223
    subscribe                                N      reactor.core.publisher.Mono:4159
    subscribe                                N      reactor.core.publisher.Mono:4131
    doFlux                                   N      com.linbit.linstor.api.rest.v1.RequestHelper:302
    modifyVolumeDefinition                   N      com.linbit.linstor.api.rest.v1.VolumeDefinitions:209
    invoke0                                  Y      jdk.internal.reflect.NativeMethodAccessorImpl:unknown
    invoke                                   N      jdk.internal.reflect.NativeMethodAccessorImpl:62
    invoke                                   N      jdk.internal.reflect.DelegatingMethodAccessorImpl:43
    invoke                                   N      java.lang.reflect.Method:566
    lambda$static$0                          N      org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory:52
    run                                      N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1:124
    invoke                                   N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:167
    doDispatch                               N      org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker:159
    dispatch                                 N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:79
    invoke                                   N      org.glassfish.jersey.server.model.ResourceMethodInvoker:469
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:391
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:80
    run                                      N      org.glassfish.jersey.server.ServerRuntime$1:253
    call                                     N      org.glassfish.jersey.internal.Errors$1:248
    call                                     N      org.glassfish.jersey.internal.Errors$1:244
    process                                  N      org.glassfish.jersey.internal.Errors:292
    process                                  N      org.glassfish.jersey.internal.Errors:274
    process                                  N      org.glassfish.jersey.internal.Errors:244
    runInScope                               N      org.glassfish.jersey.process.internal.RequestScope:265
    process                                  N      org.glassfish.jersey.server.ServerRuntime:232
    handle                                   N      org.glassfish.jersey.server.ApplicationHandler:680
    service                                  N      org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer:356
    run                                      N      org.glassfish.grizzly.http.server.HttpHandler$1:200
    doWork                                   N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:569
    run                                      N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:549
    run                                      N      java.lang.Thread:829

END OF ERROR REPORT.

I created a new volume. An error occurred expanding this volume to a size that exceeds the size of the metadata pool.

linstor resource-definition create volume1
linstor resource-definition set-property volume1 StorPoolName storage
linstor resource-definition set-property volume1 StorPoolNameDrbdMeta meta
linstor volume-definition create volume1 100MiB
linstor resource create volume1 --auto-place 2 --storage-pool storage
linstor volume-definition set-size volume1 0 4096MiB
$ linstor storage-pool list -p
+----------------------------------------------------------------------------------------------------------------------+
| StoragePool          | Node | Driver   | PoolName | FreeCapacity | TotalCapacity | CanSnapshots | State | SharedName |
|======================================================================================================================|
| DfltDisklessStorPool | sn1  | DISKLESS |          |              |               | False        | Ok    |            |
| DfltDisklessStorPool | sn2  | DISKLESS |          |              |               | False        | Ok    |            |
| meta                 | sn1  | ZFS      | pool1    |     1.74 GiB |      1.88 GiB | True         | Ok    |            |
| meta                 | sn2  | ZFS      | pool1    |     1.75 GiB |      1.88 GiB | True         | Ok    |            |
| storage              | sn1  | ZFS      | pool0    |    18.47 GiB |        59 GiB | True         | Ok    |            |
| storage              | sn2  | ZFS      | pool0    |    18.87 GiB |        59 GiB | True         | Ok    |            |
+----------------------------------------------------------------------------------------------------------------------+

Storage pool hava enogh space for volume size (4GiB < 18.47Gib), but failed by metadata pool capacity check (4GiB > 1.75GiB).

sos_2023-06-14_06-33-41.tar.gz