search

LINBIT / linstor-server

High Performance Software-Defined Block Storage for container, cloud and virtualisation. Fully integrated with Docker, Kubernetes, Openstack, Proxmox etc.
https://docs.linbit.com/docs/linstor-guide/
GNU General Public License v3.0
981 stars 76 forks source link

Controller start failed after update v1.27.1 v1.28.0 #414

Open andlf opened 2 months ago

andlf commented 2 months ago

Controller start failed after update v1.27.1 to v1.28.0 [Main] ERROR LINSTOR/Controller - SYSTEM - Unknown error during loading data from DB [Report number 66C312CF-00000-000000] 

ERROR REPORT 66C34C75-00000-000000

============================================================

Application:                        LINBIT? LINSTOR
Module:                             Controller
Version:                            1.28.0
Build ID:                           959382f7b4fb9436fefdd21dfa262e90318edaed
Build time:                         2024-07-11T10:21:06+00:00
Error time:                         2024-08-19 13:45:31
Node:                               linstor-controller-84f6f64678-rnkwm
Thread:                             Main

============================================================

Reported error:
===============

Category:                           Error
Class name:                         ImplementationError
Class canonical name:               com.linbit.ImplementationError
Generated at:                       Method 'loadCoreObjects', Source file 'DatabaseLoader.java', Line #680

Error message:                      Unknown error during loading data from DB

Call backtrace:

    Method                                   Native Class:Line number
    loadCoreObjects                          N      com.linbit.linstor.dbdrivers.DatabaseLoader:680
    loadCoreObjects                          N      com.linbit.linstor.core.DbDataInitializer:169
    initialize                               N      com.linbit.linstor.core.DbDataInitializer:101
    startSystemServices                      N      com.linbit.linstor.core.ApplicationLifecycleManager:87
    start                                    N      com.linbit.linstor.core.Controller:374
    main                                     N      com.linbit.linstor.core.Controller:625

Caused by:
==========

Category:                           LinStorException
Class name:                         DatabaseException
Class canonical name:               com.linbit.linstor.dbdrivers.DatabaseException
Generated at:                       Method 'getInstance', Source file 'ObjectProtectionFactory.java', Line #89

Error message:                      ObjProt (/resourcedefinitions/PVC-59149AEB-651C-459B-8AF5-91A99E4A7BB3) not found!

ErrorContext:

Call backtrace:

    Method                                   Native Class:Line number
    getInstance                              N      com.linbit.linstor.security.ObjectProtectionFactory:89
    getObjectProtection                      N      com.linbit.linstor.dbdrivers.AbsDatabaseDriver:288
    load                                     N      com.linbit.linstor.core.objects.ResourceDefinitionDbDriver:192
    load                                     N      com.linbit.linstor.core.objects.ResourceDefinitionDbDriver:54
    loadAll                                  N      com.linbit.linstor.dbdrivers.k8s.crd.K8sCrdEngine:238
    loadAll                                  N      com.linbit.linstor.dbdrivers.AbsDatabaseDriver:180
    loadCoreObjects                          N      com.linbit.linstor.dbdrivers.DatabaseLoader:354
    loadCoreObjects                          N      com.linbit.linstor.core.DbDataInitializer:169
    initialize                               N      com.linbit.linstor.core.DbDataInitializer:101
    startSystemServices                      N      com.linbit.linstor.core.ApplicationLifecycleManager:87
    start                                    N      com.linbit.linstor.core.Controller:374
    main                                     N      com.linbit.linstor.core.Controller:625
rp- commented 2 months ago

Is this an older installation? I mean had there been upgrades before? From the error it looks like an object protection entry is missing for a resource definition, but I'm not aware of any recent issues with that.

andlf commented 2 months ago

Is this an older installation? I mean had there been upgrades before? From the error it looks like an object protection entry is missing for a resource definition, but I'm not aware of any recent issues with that.

It is first upgrade, cluster is 2 week old, Cozystack testing

rp- commented 2 months ago

You can try for now to add secaclmap and secobjectprotection if they are missing for the mentioned resource. Like it was done in: https://github.com/LINBIT/linstor-server/issues/415

ghernadi commented 2 months ago

You don't happen to have a backup from your pre-upgrade database? If you have, it could help us to reproduce (+ understand + fix) the bug. If you have a backup from your v1.27.1 database, please send a dump of it to my email address (see my profile here).

andlf commented 2 months ago

Unfortunately, i don't made backup. I fix this issue by removing some resources with PVC-59149AEB-651C-459B-8AF5-91A99E4A7BB3 , found in backup after upgrade. ./linstor/resourcedefinitions.internal.linstor.linbit.com.yaml: resource_dsp_name: pvc-59149aeb-651c-459b-8af5-91a99e4a7bb3 ./linstor/resourcedefinitions.internal.linstor.linbit.com.yaml: resource_name: PVC-59149AEB-651C-459B-8AF5-91A99E4A7BB3 ./linstor/layerdrbdresourcedefinitions.internal.linstor.linbit.com.yaml: resource_name: PVC-59149AEB-651C-459B-8AF5-91A99E4A7BB3 ./linstor/propscontainers.internal.linstor.linbit.com.yaml: props_instance: /RESOURCEDEFINITIONS/PVC-59149AEB-651C-459B-8AF5-91A99E4A7BB3