search

LINBIT / linstor-server

High Performance Software-Defined Block Storage for container, cloud and virtualisation. Fully integrated with Docker, Kubernetes, Openstack, Proxmox etc.
https://docs.linbit.com/docs/linstor-guide/
GNU General Public License v3.0
846 stars 73 forks source link

Attempt to create an instance of class com.linbit.linstor.core.objects.ResourceDefinition with a null com.linbit.linstor.core.objects.ResourceGroup reference #286

Open kvaps opened 2 years ago

kvaps commented 2 years ago

Hi I use k8s backend. After some days I found that linstor-controller can't start:

LINSTOR, Module Controller
Version:            1.18.0 (648ab925644f53039239c5aec366a11f046f5977)
Build time:         2022-03-28T18:25:25+00:00
Java Version:       11
Java VM:            Debian, Version 11.0.14+9-post-Debian-1deb10u1
Operating system:   Linux, Version 5.13.0-27-generic
Environment:        amd64, 1 processors, 29694 MiB memory reserved for allocations

System components initialization in progress

10:57:32.513 [main] INFO  LINSTOR/Controller - SYSTEM - ErrorReporter DB first time init.
10:57:32.515 [main] INFO  LINSTOR/Controller - SYSTEM - Log directory set to: '/var/log/linstor-controller'
10:57:32.533 [Main] INFO  LINSTOR/Controller - SYSTEM - Loading API classes started.
10:57:32.800 [Main] INFO  LINSTOR/Controller - SYSTEM - API classes loading finished: 267ms
10:57:32.800 [Main] INFO  LINSTOR/Controller - SYSTEM - Dependency injection started.
10:57:32.807 [Main] INFO  LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule"
10:57:32.807 [Main] INFO  LINSTOR/Controller - SYSTEM - Extension module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule" is not installed
10:57:33.297 [Main] INFO  LINSTOR/Controller - SYSTEM - Dependency injection finished: 497ms
10:57:33.436 [Main] INFO  LINSTOR/Controller - SYSTEM - Initializing authentication subsystem
10:57:33.617 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'TimerEventService' of type TimerEventService
10:57:33.617 [Main] INFO  LINSTOR/Controller - SYSTEM - Initializing the k8s crd database connector
10:57:34.577 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'K8sCrdDatabaseService' of type K8sCrdDatabaseService
10:57:34.582 [Main] INFO  LINSTOR/Controller - SYSTEM - Loading security objects
10:57:34.630 [Main] INFO  LINSTOR/Controller - SYSTEM - Current security level is NO_SECURITY
10:57:35.070 [Main] INFO  LINSTOR/Controller - SYSTEM - Core objects load from database is in progress
10:57:35.941 [Main] ERROR LINSTOR/Controller - SYSTEM - Attempt to create an instance of class com.linbit.linstor.core.objects.ResourceDefinition with a null com.linbit.linstor.core.objects.ResourceGroup reference [Report number 625D441C-00000-000000]

10:57:35.942 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutdown in progress
10:57:35.943 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'TaskScheduleService' of type TaskScheduleService
10:57:35.943 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'TaskScheduleService' to complete shutdown
10:57:35.943 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'K8sCrdDatabaseService' of type K8sCrdDatabaseService
10:57:35.945 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'K8sCrdDatabaseService' to complete shutdown
10:57:35.945 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'TimerEventService' of type TimerEventService
10:57:35.945 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'TimerEventService' to complete shutdown
10:57:35.945 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutdown complete
ERROR REPORT 625D441C-00000-000000

============================================================

Application:                        LINBIT? LINSTOR
Module:                             Controller
Version:                            1.18.0
Build ID:                           648ab925644f53039239c5aec366a11f046f5977
Build time:                         2022-03-28T18:25:25+00:00
Error time:                         2022-04-18 10:57:35
Node:                               linstor-controller-fcb49b597-z8ht2

============================================================

Reported error:
===============

Category:                           Error
Class name:                         ImplementationError
Class canonical name:               com.linbit.ImplementationError
Generated at:                       Method 'ctorNotNull', Source file 'ErrorCheck.java', Line #20

Error message:                      Attempt to create an instance of class com.linbit.linstor.core.objects.ResourceDefinition with a null com.linbit.linstor.core.objects.ResourceGroup reference

Call backtrace:

    Method                                   Native Class:Line number
    ctorNotNull                              N      com.linbit.ErrorCheck:20
    <init>                                   N      com.linbit.linstor.core.objects.ResourceDefinition:134
    load                                     N      com.linbit.linstor.core.objects.ResourceDefinitionDbDriver:204
    load                                     N      com.linbit.linstor.core.objects.ResourceDefinitionDbDriver:53
    loadAll                                  N      com.linbit.linstor.dbdrivers.k8s.crd.K8sCrdEngine:175
    loadAll                                  N      com.linbit.linstor.dbdrivers.AbsDatabaseDriver:100
    loadAll                                  N      com.linbit.linstor.dbdrivers.DatabaseLoader:314
    loadCoreObjects                          N      com.linbit.linstor.core.DbDataInitializer:176
    initialize                               N      com.linbit.linstor.core.DbDataInitializer:108
    startSystemServices                      N      com.linbit.linstor.core.ApplicationLifecycleManager:87
    start                                    N      com.linbit.linstor.core.Controller:347
    main                                     N      com.linbit.linstor.core.Controller:585

Caused by:
==========

Category:                           RuntimeException
Class name:                         NullPointerException
Class canonical name:               java.lang.NullPointerException
Generated at:                       Method 'ctorNotNull', Source file 'ErrorCheck.java', Line #20

Call backtrace:

    Method                                   Native Class:Line number
    ctorNotNull                              N      com.linbit.ErrorCheck:20
    <init>                                   N      com.linbit.linstor.core.objects.ResourceDefinition:134
    load                                     N      com.linbit.linstor.core.objects.ResourceDefinitionDbDriver:204
    load                                     N      com.linbit.linstor.core.objects.ResourceDefinitionDbDriver:53
    loadAll                                  N      com.linbit.linstor.dbdrivers.k8s.crd.K8sCrdEngine:175
    loadAll                                  N      com.linbit.linstor.dbdrivers.AbsDatabaseDriver:100
    loadAll                                  N      com.linbit.linstor.dbdrivers.DatabaseLoader:314
    loadCoreObjects                          N      com.linbit.linstor.core.DbDataInitializer:176
    initialize                               N      com.linbit.linstor.core.DbDataInitializer:108
    startSystemServices                      N      com.linbit.linstor.core.ApplicationLifecycleManager:87
    start                                    N      com.linbit.linstor.core.Controller:347
    main                                     N      com.linbit.linstor.core.Controller:585

END OF ERROR REPORT.
redtex commented 2 years ago

Absolutely the same, with only one exception, that we using older version of LINSTOR:

LINSTOR, Module Controller
Version:            1.17.0 (7e646d83dbbadf1ec066e1bc8b29ae018aff1f66)
Build time:         2021-12-09T07:27:52+00:00
Java Version:       11
Java VM:            Debian, Version 11.0.14+9-post-Debian-1deb10u1
Operating system:   Linux, Version 4.18.0-80.el8.x86_64
Environment:        amd64, 1 processors, 5216 MiB memory reserved for allocations

LINSTOR backend is etcd 3.4.15 To resolve this issue I had to restore by hand deleted etcd keys:

/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/ALLOWED_PROVIDER_LIST
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/DESCRIPTION
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/DO_NOT_PLACE_WITH_RSC_LIST
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/LAYER_STACK
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/NODE_NAME_LIST
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/POOL_NAME
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/POOL_NAME_DISKLESS
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/REPLICAS_ON_DIFFERENT
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/REPLICAS_ON_SAME
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/REPLICA_COUNT
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/RESOURCE_GROUP_DSP_NAME
/LINSTOR/RESOURCE_GROUPS/<STORAGE-CLASS-NAME>/UUID
/LINSTOR/SEC_ACL_MAP//resourcegroups/<STORAGE-CLASS-NAME>:PUBLIC/ACCESS_TYPE
/LINSTOR/SEC_OBJECT_PROTECTION//resourcegroups/<STORAGE-CLASS-NAME>/CREATOR_IDENTITY_NAME
/LINSTOR/SEC_OBJECT_PROTECTION//resourcegroups/<STORAGE-CLASS-NAME>/OWNER_ROLE_NAME
/LINSTOR/SEC_OBJECT_PROTECTION//resourcegroups/<STORAGE-CLASS-NAME>/SECURITY_TYPE_NAME
/LINSTOR/VOLUME_GROUPS/<STORAGE-CLASS-NAME>:0/FLAGS
/LINSTOR/VOLUME_GROUPS/<STORAGE-CLASS-NAME>:0/UUID
rp- commented 2 years ago

@WanzenBug could this be related to any of the "recent" CRD fixes?

WanzenBug commented 2 years ago

Possibly, though then I'm not sure how the same issue could apply to the etcd backend.

kvaps commented 1 year ago

Hi, I just faced this issue again:

trace log:

LINSTOR, Module Controller
Version:            1.20.0 (9c6f7fad48521899f7a99c564b1d33aeacfdbfa8)
Build time:         2022-11-07T16:37:38+00:00
Java Version:       11
Java VM:            Ubuntu, Version 11.0.16+8-post-Ubuntu-0ubuntu122.04
Operating system:   Linux, Version 5.15.0-25-generic
Environment:        amd64, 1 processors, 15524 MiB memory reserved for allocations

System components initialization in progress

Loading configuration file "/etc/linstor/linstor.toml"
12:54:58.573 [main] INFO  LINSTOR/Controller - SYSTEM - ErrorReporter DB first time init.
12:54:58.575 [main] INFO  LINSTOR/Controller - SYSTEM - Log directory set to: '/var/log/linstor-controller'
12:54:58.592 [main] INFO  LINSTOR/Controller - SYSTEM - Database type is Kubernetes-CRD
12:54:58.593 [Main] INFO  LINSTOR/Controller - SYSTEM - Loading API classes started.
12:54:58.913 [Main] INFO  LINSTOR/Controller - SYSTEM - API classes loading finished: 320ms
12:54:58.913 [Main] INFO  LINSTOR/Controller - SYSTEM - Dependency injection started.
12:54:58.920 [Main] INFO  LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.modularcrypto.FipsCryptoModule"
12:54:58.920 [Main] INFO  LINSTOR/Controller - SYSTEM - Extension module "com.linbit.linstor.modularcrypto.FipsCryptoModule" is not installed
12:54:58.920 [Main] INFO  LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.modularcrypto.JclCryptoModule"
12:54:58.925 [Main] DEBUG LINSTOR/Controller - SYSTEM - Constructing instance of module "com.linbit.linstor.modularcrypto.JclCryptoModule" with default constructor
12:54:58.925 [Main] INFO  LINSTOR/Controller - SYSTEM - Dynamic load of extension module "com.linbit.linstor.modularcrypto.JclCryptoModule" was successful
12:54:58.925 [Main] INFO  LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule"
12:54:58.925 [Main] INFO  LINSTOR/Controller - SYSTEM - Extension module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule" is not installed
12:54:59.484 [Main] INFO  LINSTOR/Controller - SYSTEM - Dependency injection finished: 571ms
12:54:59.485 [Main] INFO  LINSTOR/Controller - SYSTEM - Cryptography provider: Using default cryptography module
12:54:59.638 [Main] INFO  LINSTOR/Controller - SYSTEM - Initializing authentication subsystem
12:54:59.831 [Main] DEBUG LINSTOR/Controller - SYSTEM - SpaceTrackingService: No instance available to add as a system service
12:54:59.832 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'TimerEventService' of type TimerEventService
12:54:59.832 [Main] INFO  LINSTOR/Controller - SYSTEM - Initializing the k8s crd database connector
12:54:59.833 [Main] INFO  LINSTOR/Controller - SYSTEM - Kubernetes-CRD connection URL is "k8s"
12:55:00.721 [Main] TRACE LINSTOR/Controller - SYSTEM - Found database version 9
12:55:00.850 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'K8sCrdDatabaseService' of type K8sCrdDatabaseService
12:55:00.854 [Main] INFO  LINSTOR/Controller - SYSTEM - Loading security objects
12:55:01.236 [Main] INFO  LINSTOR/Controller - SYSTEM - Current security level is NO_SECURITY
12:55:01.236 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/nodesMap)
12:55:01.288 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/nodesMap)
12:55:01.343 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/nodesMap)
12:55:01.344 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/nodesMap)
12:55:01.344 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/rscDfnMap)
12:55:01.344 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/rscDfnMap)
12:55:01.344 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/rscDfnMap)
12:55:01.344 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/rscDfnMap)
12:55:01.344 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/rscGrpMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/rscGrpMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/rscGrpMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/rscGrpMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/storPoolMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/storPoolMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/storPoolMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/storPoolMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/freeSpaceMgrMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/freeSpaceMgrMap)
12:55:01.345 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/freeSpaceMgrMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/freeSpaceMgrMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/keyValueStoreMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/keyValueStoreMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/keyValueStoreMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/keyValueStoreMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/externalFileMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/externalFileMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/externalFileMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/externalFileMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/remoteMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/remoteMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/remoteMap)
12:55:01.346 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/remoteMap)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/scheduleMap)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/scheduleMap)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/scheduleMap)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/scheduleMap)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/conf)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/conf)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/conf)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/conf)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/sys/controller/shutdown)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/sys/controller/shutdown)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/sys/controller/shutdown)
12:55:01.347 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/sys/controller/shutdown)
12:55:01.379 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/CTRLCFG)
12:55:01.471 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=STLTCFG)
12:55:01.472 [Main] INFO  LINSTOR/Controller - SYSTEM - Core objects load from database is in progress
12:55:01.472 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading all RESOURCE_GROUPSs
12:55:01.530 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/resourcegroups/DFLTRSCGRP)
12:55:01.530 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/resourcegroups/DFLTRSCGRP)
12:55:01.530 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/resourcegroups/DFLTRSCGRP)
12:55:01.531 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/resourcegroups/DFLTRSCGRP)
12:55:01.531 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/resourcegroups/DFLTRSCGRP)
12:55:01.532 [Main] TRACE LINSTOR/Controller - SYSTEM - Loaded 1 RESOURCE_GROUPSs
12:55:01.533 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading all VOLUME_GROUPSs
12:55:01.589 [Main] TRACE LINSTOR/Controller - SYSTEM - Loaded 0 VOLUME_GROUPSs
12:55:01.590 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading all NODESs
12:55:01.643 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKM02)
12:55:01.644 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKM02)
12:55:01.644 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKM02)
12:55:01.644 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKM02)
12:55:01.644 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKM02)
12:55:01.645 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKW02)
12:55:01.645 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKW02)
12:55:01.645 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKW02)
12:55:01.645 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKW02)
12:55:01.645 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKW02)
12:55:01.645 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKF01)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKF01)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKF01)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKF01)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKF01)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKS03)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKS03)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKS03)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKS03)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKS03)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/LINSTOR-CONTROLLER-785B9B87F9-QRKN6)
12:55:01.646 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/LINSTOR-CONTROLLER-785B9B87F9-QRKN6)
12:55:01.647 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/LINSTOR-CONTROLLER-785B9B87F9-QRKN6)
12:55:01.647 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/LINSTOR-CONTROLLER-785B9B87F9-QRKN6)
12:55:01.647 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/LINSTOR-CONTROLLER-785B9B87F9-QRKN6)
12:55:01.647 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKS02)
12:55:01.647 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKS02)
12:55:01.647 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKS02)
12:55:01.647 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKS02)
12:55:01.647 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKS02)
12:55:01.647 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKW01)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKW01)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKW01)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKW01)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKW01)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKF02)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKF02)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKF02)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKF02)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKF02)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKM03)
12:55:01.648 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKM03)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKM03)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKM03)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKM03)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKS01)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKS01)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKS01)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKS01)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKS01)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/LINSTOR-CONTROLLER-785B9B87F9-GW28V)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/LINSTOR-CONTROLLER-785B9B87F9-GW28V)
12:55:01.649 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/LINSTOR-CONTROLLER-785B9B87F9-GW28V)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/LINSTOR-CONTROLLER-785B9B87F9-GW28V)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/LINSTOR-CONTROLLER-785B9B87F9-GW28V)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKM01)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKM01)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKM01)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKM01)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKM01)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/nodes/SPB99-DEV-DKW03)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/nodes/SPB99-DEV-DKW03)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/nodes/SPB99-DEV-DKW03)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/nodes/SPB99-DEV-DKW03)
12:55:01.650 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading properties for instance (InstanceName=/nodes/SPB99-DEV-DKW03)
12:55:01.651 [Main] TRACE LINSTOR/Controller - SYSTEM - Loaded 13 NODESs
12:55:01.651 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading all RESOURCE_DEFINITIONSs
12:55:01.706 [Main] TRACE LINSTOR/Controller - SYSTEM - Loading ObjectProtection (ObjProtPath=/resourcedefinitions/PVC-BE6B4C5E-AD49-435A-B06D-8CAA69D5E61C)
12:55:01.706 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection instance created. (ObjProtPath=/resourcedefinitions/PVC-BE6B4C5E-AD49-435A-B06D-8CAA69D5E61C)
12:55:01.706 [Main] TRACE LINSTOR/Controller - SYSTEM - AccessControl entries restored (ObjProtPath=/resourcedefinitions/PVC-BE6B4C5E-AD49-435A-B06D-8CAA69D5E61C)
12:55:01.706 [Main] TRACE LINSTOR/Controller - SYSTEM - ObjectProtection loaded (ObjProtPath=/resourcedefinitions/PVC-BE6B4C5E-AD49-435A-B06D-8CAA69D5E61C)
12:55:01.718 [Main] ERROR LINSTOR/Controller - SYSTEM - Attempt to create an instance of class com.linbit.linstor.core.objects.ResourceDefinition with a null com.linbit.linstor.core.objects.ResourceGroup reference [Report number 63738C22-00000-000000]

12:55:01.719 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutdown in progress
12:55:01.720 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'EbsStatusPoll' of type EbsStatusPoll
12:55:01.720 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'EbsStatusPoll' to complete shutdown
12:55:01.720 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'ScheduleBackupService' of type ScheduleBackupService
12:55:01.720 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'ScheduleBackupService' to complete shutdown
12:55:01.720 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'TaskScheduleService' of type TaskScheduleService
12:55:01.720 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'TaskScheduleService' to complete shutdown
12:55:01.720 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'K8sCrdDatabaseService' of type K8sCrdDatabaseService
12:55:01.721 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'K8sCrdDatabaseService' to complete shutdown
12:55:01.721 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'TimerEventService' of type TimerEventService
12:55:01.721 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'TimerEventService' to complete shutdown
12:55:01.721 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutdown complete

error report:

ERROR REPORT 63738C22-00000-000000

============================================================

Application:                        LINBIT? LINSTOR
Module:                             Controller
Version:                            1.20.0
Build ID:                           9c6f7fad48521899f7a99c564b1d33aeacfdbfa8
Build time:                         2022-11-07T16:37:38+00:00
Error time:                         2022-11-15 12:55:01
Node:                               linstor-controller-7dcdfd7cb8-vmkrb

============================================================

Reported error:
===============

Category:                           Error
Class name:                         ImplementationError
Class canonical name:               com.linbit.ImplementationError
Generated at:                       Method 'ctorNotNull', Source file 'ErrorCheck.java', Line #20

Error message:                      Attempt to create an instance of class com.linbit.linstor.core.objects.ResourceDefinition with a null com.linbit.linstor.core.objects.ResourceGroup reference

Call backtrace:

    Method                                   Native Class:Line number
    ctorNotNull                              N      com.linbit.ErrorCheck:20
    <init>                                   N      com.linbit.linstor.core.objects.ResourceDefinition:134
    load                                     N      com.linbit.linstor.core.objects.ResourceDefinitionDbDriver:204
    load                                     N      com.linbit.linstor.core.objects.ResourceDefinitionDbDriver:53
    loadAll                                  N      com.linbit.linstor.dbdrivers.k8s.crd.K8sCrdEngine:175
    loadAll                                  N      com.linbit.linstor.dbdrivers.AbsDatabaseDriver:102
    loadAll                                  N      com.linbit.linstor.dbdrivers.DatabaseLoader:330
    loadCoreObjects                          N      com.linbit.linstor.core.DbDataInitializer:176
    initialize                               N      com.linbit.linstor.core.DbDataInitializer:108
    startSystemServices                      N      com.linbit.linstor.core.ApplicationLifecycleManager:87
    start                                    N      com.linbit.linstor.core.Controller:360
    main                                     N      com.linbit.linstor.core.Controller:608

Caused by:
==========

Category:                           RuntimeException
Class name:                         NullPointerException
Class canonical name:               java.lang.NullPointerException
Generated at:                       Method 'ctorNotNull', Source file 'ErrorCheck.java', Line #20

Call backtrace:

    Method                                   Native Class:Line number
    ctorNotNull                              N      com.linbit.ErrorCheck:20
    <init>                                   N      com.linbit.linstor.core.objects.ResourceDefinition:134
    load                                     N      com.linbit.linstor.core.objects.ResourceDefinitionDbDriver:204
    load                                     N      com.linbit.linstor.core.objects.ResourceDefinitionDbDriver:53
    loadAll                                  N      com.linbit.linstor.dbdrivers.k8s.crd.K8sCrdEngine:175
    loadAll                                  N      com.linbit.linstor.dbdrivers.AbsDatabaseDriver:102
    loadAll                                  N      com.linbit.linstor.dbdrivers.DatabaseLoader:330
    loadCoreObjects                          N      com.linbit.linstor.core.DbDataInitializer:176
    initialize                               N      com.linbit.linstor.core.DbDataInitializer:108
    startSystemServices                      N      com.linbit.linstor.core.ApplicationLifecycleManager:87
    start                                    N      com.linbit.linstor.core.Controller:360
    main                                     N      com.linbit.linstor.core.Controller:608

END OF ERROR REPORT.
kvaps commented 1 year ago

Workaroud is to replace missing resource groups to default one:

rgs=$(kubectl get resourcegroups.internal.linstor.linbit.com -o custom-columns=NAME:.spec.resource_group_name --no-headers)
kubectl get resourcedefinitions.internal.linstor.linbit.com -o custom-columns=NAME:.metadata.name,RG:.spec.resource_group_name --no-headers | \
  while read rd rg; do
    if ! echo "$rgs" | grep -q "$rg"; then
      (
        set -x
        kubectl patch resourcedefinitions.internal.linstor.linbit.com "$rd" --type=merge -p '{"spec":{"resource_group_name":"DFLTRSCGRP"}}'
      )
    fi
  done
WanzenBug commented 1 year ago

Very strange, I wonder if there is a situation in LINSTOR where we allow deletion of a resource group at the same time as creating a resource definition from it. I'm wondering because it looks like this also happened (happens?) with ETCD, so it may not be related to DB specific code, instead just a check at the wrong time in LINSTOR.

It could be that the CSI driver triggers this issue more frequently, because it tries to delete a resource group during PV deletion (with the assumption that it will fail if resource definitions remain)

duckhawk commented 9 months ago

I encountered a similar situation in my cluster. The Linstor controller refused to start due to a corrupted database. This happened after a spontaneous restart of the controller, and I couldn't find any other reasons for it. Here is the controller log:

root@linstor-controller-677d7b45d5-sjwn9:/# /usr/bin/piraeus-entry.sh startController
Importing keystore /tmp/tmp.2zp9HsgOUU to /etc/linstor/ssl/keystore.jks...
Entry for alias linstor successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled
Certificate was added to keystore
Importing keystore /tmp/tmp.iDuEyryIbX to /etc/linstor/https/keystore.jks...
Entry for alias linstor successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled
Certificate was added to keystore
LINSTOR, Module Controller
Version:            1.24.2 (adb19ca96a07039401023410c1ea116f09929295)
Build time:         2023-10-05T13:14:50+00:00
Java Version:       11
Java VM:            Ubuntu, Version 11.0.20.1+1-post-Ubuntu-0ubuntu122.04
Operating system:   Linux, Version 5.15.0-83-generic
Environment:        amd64, 4 processors, 2990 MiB memory reserved for allocations

System components initialization in progress

Loading configuration file "/etc/linstor/linstor.toml"
09:35:33.542 [main] INFO  LINSTOR/Controller - SYSTEM - ErrorReporter DB first time init.
09:35:33.545 [main] INFO  LINSTOR/Controller - SYSTEM - Log directory set to: '/var/log/linstor-controller'
09:35:33.618 [main] INFO  LINSTOR/Controller - SYSTEM - Database type is Kubernetes-CRD
09:35:33.619 [Main] INFO  LINSTOR/Controller - SYSTEM - Loading API classes started.
09:35:34.275 [Main] INFO  LINSTOR/Controller - SYSTEM - API classes loading finished: 655ms
09:35:34.275 [Main] INFO  LINSTOR/Controller - SYSTEM - Dependency injection started.
09:35:34.295 [Main] INFO  LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.modularcrypto.FipsCryptoModule"
09:35:34.296 [Main] INFO  LINSTOR/Controller - SYSTEM - Extension module "com.linbit.linstor.modularcrypto.FipsCryptoModule" is not installed
09:35:34.297 [Main] INFO  LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.modularcrypto.JclCryptoModule"
09:35:34.307 [Main] INFO  LINSTOR/Controller - SYSTEM - Dynamic load of extension module "com.linbit.linstor.modularcrypto.JclCryptoModule" was successful
09:35:34.308 [Main] INFO  LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule"
09:35:34.309 [Main] INFO  LINSTOR/Controller - SYSTEM - Extension module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule" is not installed
09:35:35.603 [Main] INFO  LINSTOR/Controller - SYSTEM - Dependency injection finished: 1328ms
09:35:35.604 [Main] INFO  LINSTOR/Controller - SYSTEM - Cryptography provider: Using default cryptography module
09:35:36.053 [Main] INFO  LINSTOR/Controller - SYSTEM - Initializing authentication subsystem
09:35:36.445 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'TimerEventService' of type TimerEventService
09:35:36.446 [Main] INFO  LINSTOR/Controller - SYSTEM - Initializing the k8s crd database connector
09:35:36.447 [Main] INFO  LINSTOR/Controller - SYSTEM - Kubernetes-CRD connection URL is "k8s"
09:35:38.424 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'K8sCrdDatabaseService' of type K8sCrdDatabaseService
09:35:38.437 [Main] INFO  LINSTOR/Controller - SYSTEM - Security objects load from database is in progress
09:35:39.015 [Main] INFO  LINSTOR/Controller - SYSTEM - Security objects load from database completed
09:35:39.016 [Main] INFO  LINSTOR/Controller - SYSTEM - Core objects load from database is in progress
09:35:39.765 [Main] ERROR LINSTOR/Controller - SYSTEM - Unknown error during loading data from DB [Report number 651FD4E4-00000-000000]

09:35:39.767 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutdown in progress
09:35:39.771 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'EbsStatusPoll' of type EbsStatusPoll
09:35:39.772 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'EbsStatusPoll' to complete shutdown
09:35:39.773 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'ScheduleBackupService' of type ScheduleBackupService
09:35:39.774 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'ScheduleBackupService' to complete shutdown
09:35:39.775 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'TaskScheduleService' of type TaskScheduleService
09:35:39.775 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'TaskScheduleService' to complete shutdown
09:35:39.776 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'K8sCrdDatabaseService' of type K8sCrdDatabaseService
09:35:39.787 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'K8sCrdDatabaseService' to complete shutdown
09:35:39.789 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutting down service instance 'TimerEventService' of type TimerEventService
09:35:39.789 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Waiting for service instance 'TimerEventService' to complete shutdown
09:35:39.789 [Thread-2] INFO  LINSTOR/Controller - SYSTEM - Shutdown complete

Error report:

root@linstor-controller-677d7b45d5-sjwn9:/# linstor error-reports 651FD4E4-00000-000000
usage: linstor error-reports [-h] {delete, list, show} ...
linstor error-reports: error: argument {delete, list, show}: invalid choice: '651FD4E4-00000-000000' (choose from 'list', 'l', 'show', 's', 'delete', 'd')
root@linstor-controller-677d7b45d5-sjwn9:/# linstor error-reports show 651FD4E4-00000-000000
^C
linstor: Client exiting (received SIGINT)
root@linstor-controller-677d7b45d5-sjwn9:/# cd /var/log
root@linstor-controller-677d7b45d5-sjwn9:/var/log# cd linstor-controller/
root@linstor-controller-677d7b45d5-sjwn9:/var/log/linstor-controller# ls -l
total 32
-rw-r--r-- 1 root root  3511 Oct  6 09:35 ErrorReport-651FD4E4-00000-000000.log
-rw-r--r-- 1 root root 20480 Oct  6 09:35 error-report.mv.db
-rw-r--r-- 1 root root  4575 Oct  6 09:35 linstor-Controller.log
root@linstor-controller-677d7b45d5-sjwn9:/var/log/linstor-controller# cat ErrorReport-651FD4E4-00000-000000.log 
ERROR REPORT 651FD4E4-00000-000000

============================================================

Application:                        LINBIT? LINSTOR
Module:                             Controller
Version:                            1.24.2
Build ID:                           adb19ca96a07039401023410c1ea116f09929295
Build time:                         2023-10-05T13:14:50+00:00
Error time:                         2023-10-06 09:35:39
Node:                               linstor-controller-677d7b45d5-sjwn9

============================================================

Reported error:
===============

Category:                           Error
Class name:                         ImplementationError
Class canonical name:               com.linbit.ImplementationError
Generated at:                       Method 'loadCoreObjects', Source file 'DatabaseLoader.java', Line #680

Error message:                      Unknown error during loading data from DB

Call backtrace:

    Method                                   Native Class:Line number
    loadCoreObjects                          N      com.linbit.linstor.dbdrivers.DatabaseLoader:680
    loadCoreObjects                          N      com.linbit.linstor.core.DbDataInitializer:169
    initialize                               N      com.linbit.linstor.core.DbDataInitializer:101
    startSystemServices                      N      com.linbit.linstor.core.ApplicationLifecycleManager:87
    start                                    N      com.linbit.linstor.core.Controller:361
    main                                     N      com.linbit.linstor.core.Controller:609

Caused by:
==========

Description:
    ObjProt (/resourcegroups/SC-CD955795-445C-5C0E-ABF5-ECC0F14F06E7) not found!

Category:                           LinStorException
Class name:                         DatabaseException
Class canonical name:               com.linbit.linstor.dbdrivers.DatabaseException
Generated at:                       Method 'getInstance', Source file 'ObjectProtectionFactory.java', Line #89

Error message:                      ObjProt (/resourcegroups/SC-CD955795-445C-5C0E-ABF5-ECC0F14F06E7) not found!

ErrorContext: 

Call backtrace:

    Method                                   Native Class:Line number
    getInstance                              N      com.linbit.linstor.security.ObjectProtectionFactory:89
    getObjectProtection                      N      com.linbit.linstor.dbdrivers.AbsDatabaseDriver:271
    load                                     N      com.linbit.linstor.core.objects.ResourceGroupDbDriver:302
    load                                     N      com.linbit.linstor.core.objects.ResourceGroupDbDriver:59
    loadAll                                  N      com.linbit.linstor.dbdrivers.k8s.crd.K8sCrdEngine:237
    loadAll                                  N      com.linbit.linstor.dbdrivers.AbsDatabaseDriver:170
    loadCoreObjects                          N      com.linbit.linstor.dbdrivers.DatabaseLoader:334
    loadCoreObjects                          N      com.linbit.linstor.core.DbDataInitializer:169
    initialize                               N      com.linbit.linstor.core.DbDataInitializer:101
    startSystemServices                      N      com.linbit.linstor.core.ApplicationLifecycleManager:87
    start                                    N      com.linbit.linstor.core.Controller:361
    main                                     N      com.linbit.linstor.core.Controller:609

END OF ERROR REPORT.

Meanwhile, the RG SC-CD955795-445C-5C0E-ABF5-ECC0F14F06E7 existed among the resources. Before making any changes, I saved all Linstor resources in yaml format and can provide them upon request. For now, I decided to temporarily transfer all resources associated with this RG to DFLTGRP and then change it to another group. The controller wouldn't start until this RG was completely cleared.