LINBIT / linstor-server

High Performance Software-Defined Block Storage for container, cloud and virtualisation. Fully integrated with Docker, Kubernetes, Openstack, Proxmox etc.
https://docs.linbit.com/docs/linstor-guide/
GNU General Public License v3.0
945 stars 75 forks source link

Snapshoot Restore failed with not enough free space #411

Open bajizs opened 3 months ago

bajizs commented 3 months ago

Try to restore an snapshot. We have enough free space on storage-pool ( Free capacity 19.69 TiB, total capacity 29.02 TiB). SNapshot has 2 volume, first (0) 20Gb, second (1) 1Tb. We have another pool to, vith smaller storage, but snapshot and original resource not stored there.

ERROR REPORT 66391CD0-00000-000004

============================================================

Application:                        LINBIT® LINSTOR
Module:                             Controller
Version:                            1.27.1
Build ID:                           c6f8ceed9d50da2c4d37ae8ce20d09daf3046464
Build time:                         2024-04-25T11:55:38+00:00
Error time:                         2024-05-22 12:41:56
Node:                               hp01
Thread:                             grizzly-http-server-29
Access context information

Identity:                           PUBLIC
Role:                               PUBLIC
Domain:                             PUBLIC

Peer:                               RestClient(127.0.0.1; 'PythonLinstor/1.19.0 (API1.0.4): Client 1.19.0')

============================================================

Reported error:
===============

Category:                           RuntimeException
Class name:                         ApiRcException
Class canonical name:               com.linbit.linstor.core.apicallhandler.response.ApiRcException
Generated at:                       Method 'checkIfStorPoolsAreUsable', Source file 'CtrlVlmCrtApiHelper.java', Line #303

Error message:                      Not enough free space available for volume 0 of resource 'mal-backup'.

Error context:
        Not enough free space available for volume 0 of resource 'mal-backup'.
Call backtrace:

    Method                                   Native Class:Line number
    checkIfStorPoolsAreUsable                N      com.linbit.linstor.core.apicallhandler.controller.CtrlVlmCrtApiHelper:303
    createVlmImpl                            N      com.linbit.linstor.core.apicallhandler.controller.CtrlVlmCrtApiHelper:152
    createVolumeFromAbsVolume                N      com.linbit.linstor.core.apicallhandler.controller.CtrlVlmCrtApiHelper:129
    restoreOnNode                            N      com.linbit.linstor.core.apicallhandler.controller.CtrlSnapshotRestoreApiCallHandler:526
    restoreResourceInTransaction             N      com.linbit.linstor.core.apicallhandler.controller.CtrlSnapshotRestoreApiCallHandler:265
    lambda$restoreSnapshot$0                 N      com.linbit.linstor.core.apicallhandler.controller.CtrlSnapshotRestoreApiCallHandler:154
    doInScope                                N      com.linbit.linstor.core.apicallhandler.ScopeRunner:149
    lambda$fluxInScope$0                     N      com.linbit.linstor.core.apicallhandler.ScopeRunner:76
    call                                     N      reactor.core.publisher.MonoCallable:72
    trySubscribeScalarMap                    N      reactor.core.publisher.FluxFlatMap:127
    subscribeOrReturn                        N      reactor.core.publisher.MonoFlatMapMany:49
    subscribe                                N      reactor.core.publisher.Flux:8759
    onNext                                   N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:195
    request                                  N      reactor.core.publisher.Operators$ScalarSubscription:2545
    onSubscribe                              N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:141
    subscribe                                N      reactor.core.publisher.MonoJust:55
    subscribe                                N      reactor.core.publisher.MonoDeferContextual:55
    subscribe                                N      reactor.core.publisher.Flux:8773
    onNext                                   N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:195
    request                                  N      reactor.core.publisher.Operators$ScalarSubscription:2545
    onSubscribe                              N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:141
    subscribe                                N      reactor.core.publisher.MonoJust:55
    subscribe                                N      reactor.core.publisher.MonoDeferContextual:55
    subscribe                                N      reactor.core.publisher.Mono:4495
    subscribeWith                            N      reactor.core.publisher.Mono:4561
    subscribe                                N      reactor.core.publisher.Mono:4462
    subscribe                                N      reactor.core.publisher.Mono:4398
    subscribe                                N      reactor.core.publisher.Mono:4370
    doFlux                                   N      com.linbit.linstor.api.rest.v1.RequestHelper:326
    restoreResource                          N      com.linbit.linstor.api.rest.v1.SnapshotRestoreResource:68
    invoke0                                  Y      jdk.internal.reflect.NativeMethodAccessorImpl:unknown
    invoke                                   N      jdk.internal.reflect.NativeMethodAccessorImpl:62
    invoke                                   N      jdk.internal.reflect.DelegatingMethodAccessorImpl:43
    invoke                                   N      java.lang.reflect.Method:566
    lambda$static$0                          N      org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory:52
    run                                      N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1:146
    invoke                                   N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:189
    doDispatch                               N      org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker:159
    dispatch                                 N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:93
    invoke                                   N      org.glassfish.jersey.server.model.ResourceMethodInvoker:478
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:400
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:81
    run                                      N      org.glassfish.jersey.server.ServerRuntime$1:256
    call                                     N      org.glassfish.jersey.internal.Errors$1:248
    call                                     N      org.glassfish.jersey.internal.Errors$1:244
    process                                  N      org.glassfish.jersey.internal.Errors:292
    process                                  N      org.glassfish.jersey.internal.Errors:274
    process                                  N      org.glassfish.jersey.internal.Errors:244
    runInScope                               N      org.glassfish.jersey.process.internal.RequestScope:265
    process                                  N      org.glassfish.jersey.server.ServerRuntime:235
    handle                                   N      org.glassfish.jersey.server.ApplicationHandler:684
    service                                  N      org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer:356
    run                                      N      org.glassfish.grizzly.http.server.HttpHandler$1:190
    doWork                                   N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:535
    run                                      N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:515
    run                                      N      java.lang.Thread:829

END OF ERROR REPORT.
bajizs commented 3 months ago

restarted controller and satelite and now created successfully....

bajizs commented 2 months ago

I'ts happand again. After restart of controller and satelit now working. What is wrong? This issue I never have before version upgrade to 1.27.1

ERROR REPORT 664DD1CA-00000-000026

============================================================

Application:                        LINBIT® LINSTOR
Module:                             Controller
Version:                            1.27.1
Build ID:                           c6f8ceed9d50da2c4d37ae8ce20d09daf3046464
Build time:                         2024-04-25T11:55:38+00:00
Error time:                         2024-06-25 15:02:55
Node:                               hp01
Thread:                             grizzly-http-server-35
Access context information

Identity:                           PUBLIC
Role:                               PUBLIC
Domain:                             PUBLIC

Peer:                               RestClient(127.0.0.1; 'PythonLinstor/1.19.0 (API1.0.4): Client 1.19.0')

============================================================

Reported error:
===============

Category:                           RuntimeException
Class name:                         ApiRcException
Class canonical name:               com.linbit.linstor.core.apicallhandler.response.ApiRcException
Generated at:                       Method 'checkIfStorPoolsAreUsable', Source file 'CtrlVlmCrtApiHelper.java', Line #303

Error message:                      Not enough free space available for volume 0 of resource 'backup_db088c31'.

Error context:
        Not enough free space available for volume 0 of resource 'backup_db088c31'.
Call backtrace:

    Method                                   Native Class:Line number
    checkIfStorPoolsAreUsable                N      com.linbit.linstor.core.apicallhandler.controller.CtrlVlmCrtApiHelper:303
    createVlmImpl                            N      com.linbit.linstor.core.apicallhandler.controller.CtrlVlmCrtApiHelper:152
    createVolumeFromAbsVolume                N      com.linbit.linstor.core.apicallhandler.controller.CtrlVlmCrtApiHelper:129
    restoreOnNode                            N      com.linbit.linstor.core.apicallhandler.controller.CtrlSnapshotRestoreApiCallHandler:526
    restoreResourceInTransaction             N      com.linbit.linstor.core.apicallhandler.controller.CtrlSnapshotRestoreApiCallHandler:265
    lambda$restoreSnapshot$0                 N      com.linbit.linstor.core.apicallhandler.controller.CtrlSnapshotRestoreApiCallHandler:154
    doInScope                                N      com.linbit.linstor.core.apicallhandler.ScopeRunner:149
    lambda$fluxInScope$0                     N      com.linbit.linstor.core.apicallhandler.ScopeRunner:76
    call                                     N      reactor.core.publisher.MonoCallable:72
    trySubscribeScalarMap                    N      reactor.core.publisher.FluxFlatMap:127
    subscribeOrReturn                        N      reactor.core.publisher.MonoFlatMapMany:49
    subscribe                                N      reactor.core.publisher.Flux:8759
    onNext                                   N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:195
    request                                  N      reactor.core.publisher.Operators$ScalarSubscription:2545
    onSubscribe                              N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:141
    subscribe                                N      reactor.core.publisher.MonoJust:55
    subscribe                                N      reactor.core.publisher.MonoDeferContextual:55
    subscribe                                N      reactor.core.publisher.Flux:8773
    onNext                                   N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:195
    request                                  N      reactor.core.publisher.Operators$ScalarSubscription:2545
    onSubscribe                              N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:141
    subscribe                                N      reactor.core.publisher.MonoJust:55
    subscribe                                N      reactor.core.publisher.MonoDeferContextual:55
    subscribe                                N      reactor.core.publisher.Mono:4495
    subscribeWith                            N      reactor.core.publisher.Mono:4561
    subscribe                                N      reactor.core.publisher.Mono:4462
    subscribe                                N      reactor.core.publisher.Mono:4398
    subscribe                                N      reactor.core.publisher.Mono:4370
    doFlux                                   N      com.linbit.linstor.api.rest.v1.RequestHelper:326
    restoreResource                          N      com.linbit.linstor.api.rest.v1.SnapshotRestoreResource:68
    invoke                                   N      jdk.internal.reflect.GeneratedMethodAccessor5:unknown
    invoke                                   N      jdk.internal.reflect.DelegatingMethodAccessorImpl:43
    invoke                                   N      java.lang.reflect.Method:566
    lambda$static$0                          N      org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory:52
    run                                      N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1:146
    invoke                                   N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:189
    doDispatch                               N      org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker:159
    dispatch                                 N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:93
    invoke                                   N      org.glassfish.jersey.server.model.ResourceMethodInvoker:478
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:400
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:81
    run                                      N      org.glassfish.jersey.server.ServerRuntime$1:256
    call                                     N      org.glassfish.jersey.internal.Errors$1:248
    call                                     N      org.glassfish.jersey.internal.Errors$1:244
    process                                  N      org.glassfish.jersey.internal.Errors:292
    process                                  N      org.glassfish.jersey.internal.Errors:274
    process                                  N      org.glassfish.jersey.internal.Errors:244
    runInScope                               N      org.glassfish.jersey.process.internal.RequestScope:265
    process                                  N      org.glassfish.jersey.server.ServerRuntime:235
    handle                                   N      org.glassfish.jersey.server.ApplicationHandler:684
    service                                  N      org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer:356
    run                                      N      org.glassfish.grizzly.http.server.HttpHandler$1:190
    doWork                                   N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:535
    run                                      N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:515
    run                                      N      java.lang.Thread:829

END OF ERROR REPORT.