Storage pool can not be deleted...

Since the update to 1.7.0 the linstor-controller every few seconds throws the error in the log The specified storage pool 'lvm-thin' on node 'host9' can not be deleted as volumes / snapshot-volumes are still using it.. Even if all containers with DRBD resources are scaled down and no DRBD resource is mounted on the cluster nodes the message doesn't disappear.

Is this a bug during the upgrade routine or is it smth we have to resolve manually?

Here is the error report:



============================================================

Application:                        LINBIT�� LINSTOR
Module:                             Controller
Version:                            1.17.0
Build ID:                           7e646d83dbbadf1ec066e1bc8b29ae018aff1f66
Build time:                         2021-12-09T07:27:52+00:00
Error time:                         2022-02-01 18:08:56
Node:                               piraeus-op-cs-controller-6f7f457db-rs2q5
Peer:                               RestClient(10.42.1.105; 'Go-http-client/1.1')

============================================================

Reported error:
===============

Category:                           RuntimeException
Class name:                         ApiRcException
Class canonical name:               com.linbit.linstor.core.apicallhandler.response.ApiRcException
Generated at:                       Method 'deleteStorPoolInTransaction', Source file 'CtrlStorPoolApiCallHandler.java', Line #301

Error message:                      The specified storage pool 'lvm-thin' on node 'host9' can not be deleted as volumes / snapshot-volumes are still using it.

Error context:
    The specified storage pool 'lvm-thin' on node 'host9' can not be deleted as volumes / snapshot-volumes are still using it.

Call backtrace:

    Method                                   Native Class:Line number
    deleteStorPoolInTransaction              N      com.linbit.linstor.core.apicallhandler.controller.CtrlStorPoolApiCallHandler:301
    lambda$deleteStorPool$2                  N      com.linbit.linstor.core.apicallhandler.controller.CtrlStorPoolApiCallHandler:213
    doInScope                                N      com.linbit.linstor.core.apicallhandler.ScopeRunner:147
    lambda$fluxInScope$0                     N      com.linbit.linstor.core.apicallhandler.ScopeRunner:75
    call                                     N      reactor.core.publisher.MonoCallable:91
    trySubscribeScalarMap                    N      reactor.core.publisher.FluxFlatMap:126
    subscribeOrReturn                        N      reactor.core.publisher.MonoFlatMapMany:49
    subscribe                                N      reactor.core.publisher.Flux:8343
    onNext                                   N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:188
    request                                  N      reactor.core.publisher.Operators$ScalarSubscription:2344
    onSubscribe                              N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:134
    subscribe                                N      reactor.core.publisher.MonoCurrentContext:35
    subscribe                                N      reactor.core.publisher.Flux:8357
    onNext                                   N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:188
    request                                  N      reactor.core.publisher.Operators$ScalarSubscription:2344
    onSubscribe                              N      reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:134
    subscribe                                N      reactor.core.publisher.MonoCurrentContext:35
    subscribe                                N      reactor.core.publisher.Mono:4252
    subscribeWith                            N      reactor.core.publisher.Mono:4363
    subscribe                                N      reactor.core.publisher.Mono:4223
    subscribe                                N      reactor.core.publisher.Mono:4159
    subscribe                                N      reactor.core.publisher.Mono:4131
    doFlux                                   N      com.linbit.linstor.api.rest.v1.RequestHelper:304
    deleteStorPool                           N      com.linbit.linstor.api.rest.v1.StoragePools:330
    invoke                                   N      jdk.internal.reflect.GeneratedMethodAccessor26:unknown
    invoke                                   N      jdk.internal.reflect.DelegatingMethodAccessorImpl:43
    invoke                                   N      java.lang.reflect.Method:566
    lambda$static$0                          N      org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory:52
    run                                      N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1:124
    invoke                                   N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:167
    doDispatch                               N      org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker:159
    dispatch                                 N      org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher:79
    invoke                                   N      org.glassfish.jersey.server.model.ResourceMethodInvoker:469
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:391
    apply                                    N      org.glassfish.jersey.server.model.ResourceMethodInvoker:80
    run                                      N      org.glassfish.jersey.server.ServerRuntime$1:253
    call                                     N      org.glassfish.jersey.internal.Errors$1:248
    call                                     N      org.glassfish.jersey.internal.Errors$1:244
    process                                  N      org.glassfish.jersey.internal.Errors:292
    process                                  N      org.glassfish.jersey.internal.Errors:274
    process                                  N      org.glassfish.jersey.internal.Errors:244
    runInScope                               N      org.glassfish.jersey.process.internal.RequestScope:265
    process                                  N      org.glassfish.jersey.server.ServerRuntime:232
    handle                                   N      org.glassfish.jersey.server.ApplicationHandler:680
    service                                  N      org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer:356
    run                                      N      org.glassfish.grizzly.http.server.HttpHandler$1:200
    doWork                                   N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:569
    run                                      N      org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker:549
    run                                      N      java.lang.Thread:829

END OF ERROR REPORT.```

My question is why the storage pool should be deleted? The only way this is triggered is if operator.satelliteSet.storagePools was modified (removed entries).

There are no additional checks happening in the operator, so it just tries to delete the storage pool, even if there are resources or snapshots still present. Note that this does include simple replicas, even if the drbd device is not actively mounted.

So my advise would be:

Check that the LinstorSatelliteSet resource has the expected storage pools
- If not: edit it to have all expected storage pools
- If removal was intentional: you have to manually delete those resources on the node, there is no automatic removal by the operator.

Thanks @WanzenBug for the fast reply. Actually neither storage pool nor the list of nodes has changed. And these errors occur on almost all our clusters which have been updated to Piraeus 1.7.

Here are screenshots from the linstor cmd output: Bildschirmfoto 2022-02-02 um 11 30 54

The nodes and store pools look good and the storage pool is also referenced correctly for the resources.

I've checked the output of kubectl get LinstorSatelliteSet.piraeus.linbit.com piraeus-op-ns. Is it maybe a problem that the storage pools haven't be announced in the values file for the new CRD configuration? In the output the storagePools are empty (because we haven't declared it on the helm upgrade) but the SatelliteStatus OFC lists the storage pools which have been created on the first Linstor deployment on the cluster:


  sslSecret: null
  storagePools:
    lvmPools: []
    lvmThinPools: []
    zfsPools: []
  tolerations: []
status:
  SatelliteStatuses:
  - connectionStatus: ONLINE
    nodeName: de-fra-node10
    registeredOnController: true
    storagePoolStatus:
    - freeCapacity: 9223372036854775807
      name: DfltDisklessStorPool
      nodeName: de-fra-node10
      provider: DISKLESS
      totalCapacity: 9223372036854775807
    - freeCapacity: 1132126535
      name: lvm-thin
      nodeName: de-fra-node10
      provider: LVM_THIN
      totalCapacity: 1677721600
  - connectionStatus: ONLINE
    nodeName: de-fra-node8
    registeredOnController: true
    storagePoolStatus:
    - freeCapacity: 9223372036854775807
      name: DfltDisklessStorPool
      nodeName: de-fra-node8
      provider: DISKLESS
      totalCapacity: 9223372036854775807
    - freeCapacity: 1132126535
      name: lvm-thin
      nodeName: de-fra-node8
      provider: LVM_THIN
      totalCapacity: 1677721600
  - connectionStatus: ONLINE
    nodeName: de-fra-node9
    registeredOnController: true
    storagePoolStatus:
    - freeCapacity: 9223372036854775807
      name: DfltDisklessStorPool
      nodeName: de-fra-node9
      provider: DISKLESS
      totalCapacity: 9223372036854775807
    - freeCapacity: 1132126535
      name: lvm-thin
      nodeName: de-fra-node9
      provider: LVM_THIN
      totalCapacity: 1677721600
  errors:
  - "Message: 'The specified storage pool 'lvm-thin' on node 'de-fra-node9' can not
    be deleted as volumes / snapshot-volumes are still using it.'; Details: 'Volumes
    / snapshot-volumes that are still using the storage pool: \n   Node name: 'de-fra-node9',
    resource name: 'pvc-1e59589f-e04e-4aee-a1c6-0561a764a7e8', volume number: 0\n
    \  Node name: 'de-fra-node9', resource name: 'pvc-2c8ea040-9651-4501-a0d9-7b3920c82ec8',
    volume number: 0\n   Node name: 'de-fra-node9', resource name: 'pvc-4506c792-6fbb-43b6-a6ca-745084259e0d',
    volume number: 0\n   Node name: 'de-fra-node9', resource name: 'pvc-4e6cf7f8-2a42-4129-9a9b-cba310b3ed9e',
    volume number: 0\n   Node name: 'de-fra-node9', resource name: 'pvc-7afac331-4319-4ca7-b587-1ec267dc63b8',
    volume number: 0\n   Node name: 'de-fra-node9', resource name: 'pvc-7c07be83-43aa-44af-b9c6-c600339ad6a8',
    volume number: 0\n   Node name: 'de-fra-node9', resource name: 'pvc-8366a898-20ca-4f71-abee-cc3c40ff8bf1',
    volume number: 0\n   Node name: 'de-fra-node9', resource name: 'pvc-a2bff230-7ca6-4e93-a273-15d217967def',
    volume number: 0\n   Node name: 'de-fra-node9', resource name: 'pvc-b1cb2468-4c1b-4872-8040-9e9860c45c76',
    volume number: 0\n   Node name: 'de-fra-node9', resource name: 'pvc-cabb808d-d90b-4c59-831f-a3f08420effc',
    volume number: 0\nNode: de-fra-node9, Storage pool name: lvm-thin'; Correction:
    'Delete the listed volumes and snapshot-volumes first.'; Reports: '[61F976B7-00000-071670]'"
  - "Message: 'The specified storage pool 'lvm-thin' on node 'de-fra-node8' can not
    be deleted as volumes / snapshot-volumes are still using it.'; Details: 'Volumes
    / snapshot-volumes that are still using the storage pool: \n   Node name: 'de-fra-node8',
    resource name: 'pvc-1e59589f-e04e-4aee-a1c6-0561a764a7e8', volume number: 0\n
    \  Node name: 'de-fra-node8', resource name: 'pvc-2c8ea040-9651-4501-a0d9-7b3920c82ec8',
    volume number: 0\n   Node name: 'de-fra-node8', resource name: 'pvc-4506c792-6fbb-43b6-a6ca-745084259e0d',
    volume number: 0\n   Node name: 'de-fra-node8', resource name: 'pvc-4e6cf7f8-2a42-4129-9a9b-cba310b3ed9e',
    volume number: 0\n   Node name: 'de-fra-node8', resource name: 'pvc-7afac331-4319-4ca7-b587-1ec267dc63b8',
    volume number: 0\n   Node name: 'de-fra-node8', resource name: 'pvc-7c07be83-43aa-44af-b9c6-c600339ad6a8',
    volume number: 0\n   Node name: 'de-fra-node8', resource name: 'pvc-8366a898-20ca-4f71-abee-cc3c40ff8bf1',
    volume number: 0\n   Node name: 'de-fra-node8', resource name: 'pvc-a2bff230-7ca6-4e93-a273-15d217967def',
    volume number: 0\n   Node name: 'de-fra-node8', resource name: 'pvc-b1cb2468-4c1b-4872-8040-9e9860c45c76',
    volume number: 0\n   Node name: 'de-fra-node8', resource name: 'pvc-cabb808d-d90b-4c59-831f-a3f08420effc',
    volume number: 0\nNode: de-fra-node8, Storage pool name: lvm-thin'; Correction:
    'Delete the listed volumes and snapshot-volumes first.'; Reports: '[61F976B7-00000-071671]'"
  - "Message: 'The specified storage pool 'lvm-thin' on node 'de-fra-node10' can not
    be deleted as volumes / snapshot-volumes are still using it.'; Details: 'Volumes
    / snapshot-volumes that are still using the storage pool: \n   Node name: 'de-fra-node10',
    resource name: 'pvc-1e59589f-e04e-4aee-a1c6-0561a764a7e8', volume number: 0\n
    \  Node name: 'de-fra-node10', resource name: 'pvc-2c8ea040-9651-4501-a0d9-7b3920c82ec8',
    volume number: 0\n   Node name: 'de-fra-node10', resource name: 'pvc-4506c792-6fbb-43b6-a6ca-745084259e0d',
    volume number: 0\n   Node name: 'de-fra-node10', resource name: 'pvc-4e6cf7f8-2a42-4129-9a9b-cba310b3ed9e',
    volume number: 0\n   Node name: 'de-fra-node10', resource name: 'pvc-7afac331-4319-4ca7-b587-1ec267dc63b8',
    volume number: 0\n   Node name: 'de-fra-node10', resource name: 'pvc-7c07be83-43aa-44af-b9c6-c600339ad6a8',
    volume number: 0\n   Node name: 'de-fra-node10', resource name: 'pvc-8366a898-20ca-4f71-abee-cc3c40ff8bf1',
    volume number: 0\n   Node name: 'de-fra-node10', resource name: 'pvc-a2bff230-7ca6-4e93-a273-15d217967def',
    volume number: 0\n   Node name: 'de-fra-node10', resource name: 'pvc-b1cb2468-4c1b-4872-8040-9e9860c45c76',
    volume number: 0\n   Node name: 'de-fra-node10', resource name: 'pvc-cabb808d-d90b-4c59-831f-a3f08420effc',
    volume number: 0\nNode: de-fra-node10, Storage pool name: lvm-thin'; Correction:
    'Delete the listed volumes and snapshot-volumes first.'; Reports: '[61F976B7-00000-071672]'"```

Is it maybe a problem that the storage pools haven't be announced in the values file for the new CRD configuration?

Yes. If they are not set on helm upgrade, helm just removes them, and then the operator tries to delete the storage pool in a loop. It's a bit cumbersome, I know, but that's what helm does :shrug:

So you should edit the LinstorSatelliteSet to say:

spec:
  storagePools:
    lvmThinPools:
    - name: lvm-thin
      volumeGroup: vg-pool
      thinVolume: disk-redundant

And also save that to your helm overrides for the next upgrade.

Awesome, thanks for your help @WanzenBug , that actually resolved the problem. Our expectation was that the new CRD would get merged with the existing information in etcd, so we skipped the setting in the values.yaml. Maybe this should be added to

release notes
https://github.com/piraeusdatastore/piraeus-operator/blob/v1.7.0/UPGRADE.md
values.yaml

piraeusdatastore / piraeus-operator

Storage pool can not be deleted... #264