search

LINBIT / linstor-server

High Performance Software-Defined Block Storage for container, cloud and virtualisation. Fully integrated with Docker, Kubernetes, Openstack, Proxmox etc.
https://docs.linbit.com/docs/linstor-guide/
GNU General Public License v3.0
984 stars 76 forks source link

ETCD: linstor-proxmox can't connect after migrating from H2 to ETCD #366

Open ggzengel opened 1 year ago

ggzengel commented 1 year ago

Linstor and DRBD is fully working after migration (export-db/import-db) except proxmox can't access the linstor-controller.

pvestatd[7861]: could not connect to any LINSTOR controller at /usr/share/perl5/PVE/Storage/Custom/LINSTORPlugin.pm line 214

It looks like REST-API does not respond:

# wget 172.19.1.4:3370/health
--2023-08-08 17:04:22--  http://172.19.1.4:3370/health
Connecting to 172.19.1.4:3370... connected.
HTTP request sent, awaiting response... 500 Internal Server Error
2023-08-08 17:04:22 ERROR 500: Internal Server Error.
2023-08-08T17:04:09.953648+00:00 px4 Controller[1694726]: LINSTOR, Module Controller
2023-08-08T17:04:09.997696+00:00 px4 Controller[1694726]: Version:            1.24.0 (326986a4d1834d970e03f26594f3198af6ad47a6)
2023-08-08T17:04:09.997798+00:00 px4 Controller[1694726]: Build time:         2023-08-07T11:51:43+00:00
2023-08-08T17:04:09.997821+00:00 px4 Controller[1694726]: Java Version:       17
2023-08-08T17:04:09.997840+00:00 px4 Controller[1694726]: Java VM:            Debian, Version 17.0.8+7-Debian-1deb12u1
2023-08-08T17:04:09.997859+00:00 px4 Controller[1694726]: Operating system:   Linux, Version 6.2.16-6-pve
2023-08-08T17:04:09.997881+00:00 px4 Controller[1694726]: Environment:        amd64, 16 processors, 30688 MiB memory reserved for allocations
2023-08-08T17:04:09.997913+00:00 px4 Controller[1694726]: System components initialization in progress
2023-08-08T17:04:10.008121+00:00 px4 Controller[1694726]: WARNING: A terminally deprecated method in java.lang.System has been called
2023-08-08T17:04:10.008242+00:00 px4 Controller[1694726]: WARNING: System::setSecurityManager has been called by com.linbit.linstor.core.Controller (file:/usr/share/linstor-server/lib/controller-1.24.0.jar)
2023-08-08T17:04:10.008277+00:00 px4 Controller[1694726]: WARNING: Please consider reporting this to the maintainers of com.linbit.linstor.core.Controller
2023-08-08T17:04:10.008296+00:00 px4 Controller[1694726]: WARNING: System::setSecurityManager will be removed in a future release
2023-08-08T17:04:10.130876+00:00 px4 Controller[1694726]: Loading configuration file "/etc/linstor/linstor.toml"
2023-08-08T17:04:10.362399+00:00 px4 Controller[1694726]: 17:04:10,232 |-INFO in ch.qos.logback.classic.LoggerContext[default] - This is logback-classic version 1.3.8
2023-08-08T17:04:10.362516+00:00 px4 Controller[1694726]: 17:04:10,238 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
2023-08-08T17:04:10.362552+00:00 px4 Controller[1694726]: 17:04:10,238 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/usr/share/linstor-server/lib/conf/logback.xml]
2023-08-08T17:04:10.362589+00:00 px4 Controller[1694726]: 17:04:10,337 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - Processing appender named [STDOUT]
2023-08-08T17:04:10.362622+00:00 px4 Controller[1694726]: 17:04:10,337 |-INFO in ch.qos.logback.core.model.processor.AppenderModelHandler - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
2023-08-08T17:04:10.362654+00:00 px4 Controller[1694726]: 17:04:10,343 |-INFO in ch.qos.logback.core.model.processor.ImplicitModelHandler - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
2023-08-08T17:04:10.362685+00:00 px4 Controller[1694726]: 17:04:10,358 |-WARN in ch.qos.logback.core.model.processor.AppenderModelHandler - Appender named [FILE] not referenced. Skipping further processing.
2023-08-08T17:04:10.362714+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [okhttp3] to OFF
2023-08-08T17:04:10.362739+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [org.apache.http] to INFO
2023-08-08T17:04:10.362777+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [cron] to INFO
2023-08-08T17:04:10.362807+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [LINSTOR/Controller] to INFO
2023-08-08T17:04:10.362839+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [LINSTOR/Controller] to false
2023-08-08T17:04:10.362870+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[LINSTOR/Controller]
2023-08-08T17:04:10.362902+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [LINSTOR/Satellite] to INFO
2023-08-08T17:04:10.362944+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [LINSTOR/Satellite] to false
2023-08-08T17:04:10.362975+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[LINSTOR/Satellite]
2023-08-08T17:04:10.363005+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting level of logger [LINSTOR/TESTS] to OFF
2023-08-08T17:04:10.363033+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.LoggerModelHandler - Setting additivity of logger [LINSTOR/TESTS] to false
2023-08-08T17:04:10.363061+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[LINSTOR/TESTS]
2023-08-08T17:04:10.363089+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.classic.model.processor.RootLoggerModelHandler - Setting level of ROOT logger to INFO
2023-08-08T17:04:10.363119+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.core.model.processor.AppenderRefModelHandler - Attaching appender named [STDOUT] to Logger[ROOT]
2023-08-08T17:04:10.363150+00:00 px4 Controller[1694726]: 17:04:10,359 |-INFO in ch.qos.logback.core.model.processor.DefaultProcessor@1b2abca6 - End of configuration.
2023-08-08T17:04:10.363180+00:00 px4 Controller[1694726]: 17:04:10,360 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@6392827e - Registering current configuration as safe fallback point
2023-08-08T17:04:10.656103+00:00 px4 Controller[1694726]: 17:04:10.655 [main] INFO  LINSTOR/Controller - SYSTEM - ErrorReporter DB version 1 found.
2023-08-08T17:04:10.656621+00:00 px4 Controller[1694726]: 17:04:10.656 [main] INFO  LINSTOR/Controller - SYSTEM - Log directory set to: '/var/log/linstor-controller'
2023-08-08T17:04:10.682661+00:00 px4 Controller[1694726]: 17:04:10.682 [main] INFO  LINSTOR/Controller - SYSTEM - Database type is etcd
2023-08-08T17:04:10.682881+00:00 px4 Controller[1694726]: 17:04:10.682 [Main] INFO  LINSTOR/Controller - SYSTEM - Loading API classes started.
2023-08-08T17:04:11.126970+00:00 px4 Controller[1694726]: 17:04:11.126 [Main] INFO  LINSTOR/Controller - SYSTEM - API classes loading finished: 443ms
2023-08-08T17:04:11.127123+00:00 px4 Controller[1694726]: 17:04:11.126 [Main] INFO  LINSTOR/Controller - SYSTEM - Dependency injection started.
2023-08-08T17:04:11.146791+00:00 px4 Controller[1694726]: 17:04:11.146 [Main] INFO  LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.modularcrypto.FipsCryptoModule"
2023-08-08T17:04:11.147267+00:00 px4 Controller[1694726]: 17:04:11.147 [Main] INFO  LINSTOR/Controller - SYSTEM - Extension module "com.linbit.linstor.modularcrypto.FipsCryptoModule" is not installed
2023-08-08T17:04:11.147358+00:00 px4 Controller[1694726]: 17:04:11.147 [Main] INFO  LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.modularcrypto.JclCryptoModule"
2023-08-08T17:04:11.166204+00:00 px4 Controller[1694726]: 17:04:11.165 [Main] INFO  LINSTOR/Controller - SYSTEM - Dynamic load of extension module "com.linbit.linstor.modularcrypto.JclCryptoModule" was successful
2023-08-08T17:04:11.166459+00:00 px4 Controller[1694726]: 17:04:11.166 [Main] INFO  LINSTOR/Controller - SYSTEM - Attempting dynamic load of extension module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule"
2023-08-08T17:04:11.167412+00:00 px4 Controller[1694726]: 17:04:11.167 [Main] INFO  LINSTOR/Controller - SYSTEM - Dynamic load of extension module "com.linbit.linstor.spacetracking.ControllerSpaceTrackingModule" was successful
2023-08-08T17:04:12.160229+00:00 px4 Controller[1694726]: 17:04:12.159 [Main] INFO  LINSTOR/Controller - SYSTEM - Dependency injection finished: 1033ms
2023-08-08T17:04:12.160389+00:00 px4 Controller[1694726]: 17:04:12.160 [Main] INFO  LINSTOR/Controller - SYSTEM - Cryptography provider: Using default cryptography module
2023-08-08T17:04:12.457393+00:00 px4 Controller[1694726]: 17:04:12.457 [Main] INFO  LINSTOR/Controller - SYSTEM - Initializing authentication subsystem
2023-08-08T17:04:12.707377+00:00 px4 Controller[1694726]: 17:04:12.707 [Main] INFO  LINSTOR/Controller - SYSTEM - SpaceTrackingService: Instance added as a system service
2023-08-08T17:04:12.708368+00:00 px4 Controller[1694726]: 17:04:12.708 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'TimerEventService' of type TimerEventService
2023-08-08T17:04:12.708716+00:00 px4 Controller[1694726]: 17:04:12.708 [Main] INFO  LINSTOR/Controller - SYSTEM - Initializing the etcd database
2023-08-08T17:04:12.708792+00:00 px4 Controller[1694726]: 17:04:12.708 [Main] INFO  LINSTOR/Controller - SYSTEM - etcd connection URL is "etcd://px1.cc.private:2379,px2.cc.private:2379,px3.cc.private:2379,px4.cc.private:2379"
2023-08-08T17:04:13.344203+00:00 px4 Controller[1694726]: 17:04:13.343 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'ETCDDatabaseService' of type ETCDDatabaseService
2023-08-08T17:04:13.348404+00:00 px4 Controller[1694726]: 17:04:13.348 [Main] INFO  LINSTOR/Controller - SYSTEM - Security objects load from database is in progress
2023-08-08T17:04:13.504371+00:00 px4 Controller[1694726]: 17:04:13.504 [Main] INFO  LINSTOR/Controller - SYSTEM - Security objects load from database completed
2023-08-08T17:04:13.504515+00:00 px4 Controller[1694726]: 17:04:13.504 [Main] INFO  LINSTOR/Controller - SYSTEM - Core objects load from database is in progress
2023-08-08T17:04:13.741625+00:00 px4 Controller[1694726]: 17:04:13.741 [Main] INFO  LINSTOR/Controller - SYSTEM - Core objects load from database completed
2023-08-08T17:04:13.830935+00:00 px4 Controller[1694726]: 17:04:13.830 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'TaskScheduleService' of type TaskScheduleService
2023-08-08T17:04:13.833775+00:00 px4 Controller[1694726]: 17:04:13.833 [Main] INFO  LINSTOR/Controller - SYSTEM - Initializing network communications services
2023-08-08T17:04:13.834306+00:00 px4 Controller[1694726]: 17:04:13.834 [Main] WARN  LINSTOR/Controller - SYSTEM - The SSL network communication service 'DebugSslConnector' could not be started because the keyStore file (/etc/linstor/ssl/keystore.jks) is missing
2023-08-08T17:04:13.839026+00:00 px4 Controller[1694726]: 17:04:13.838 [Main] INFO  LINSTOR/Controller - SYSTEM - Created network communication service 'PlainConnector'
2023-08-08T17:04:13.839247+00:00 px4 Controller[1694726]: 17:04:13.839 [Main] WARN  LINSTOR/Controller - SYSTEM - The SSL network communication service 'SslConnector' could not be started because the keyStore file (/etc/linstor/ssl/keystore.jks) is missing
2023-08-08T17:04:13.839307+00:00 px4 Controller[1694726]: 17:04:13.839 [Main] INFO  LINSTOR/Controller - SYSTEM - Created network communication service 'SslConnector'
2023-08-08T17:04:13.904847+00:00 px4 Controller[1694726]: 17:04:13.904 [Main] INFO  LINSTOR/Controller - SYSTEM - Reconnecting to previously known nodes
2023-08-08T17:04:13.922775+00:00 px4 Controller[1694726]: 17:04:13.922 [Main] INFO  LINSTOR/Controller - SYSTEM - Reconnect requests sent
2023-08-08T17:04:13.923410+00:00 px4 Controller[1694726]: 17:04:13.923 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'SpaceTrackingService' of type SpaceTrackingService
2023-08-08T17:04:13.923742+00:00 px4 Controller[1694726]: 17:04:13.923 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'ScheduleBackupService' of type ScheduleBackupService
2023-08-08T17:04:13.924750+00:00 px4 Controller[1694726]: 17:04:13.924 [Main] INFO  LINSTOR/Controller - SYSTEM - Starting service instance 'EbsStatusPoll' of type EbsStatusPoll
2023-08-08T17:04:14.061903+00:00 px4 Controller[1694726]: 17:04:14.061 [SpaceTrackingService] ERROR LINSTOR/Controller - SYSTEM - Uncaught exception in j [Report number 64D2758A-00000-000000]
2023-08-08T17:04:14.223450+00:00 px4 Controller[1694726]: Aug 08, 2023 5:04:14 PM org.glassfish.jersey.server.wadl.WadlFeature configure
2023-08-08T17:04:14.223568+00:00 px4 Controller[1694726]: WARNING: JAX-B API not found . WADL feature is disabled.
2023-08-08T17:04:14.740898+00:00 px4 Controller[1694726]: 17:04:14.740 [Main] ERROR LINSTOR/Controller - SYSTEM - Unsupported address type [Report number 64D2758A-00000-000001]
2023-08-08T17:04:14.741076+00:00 px4 Controller[1694726]: 17:04:14.740 [Main] INFO  LINSTOR/Controller - SYSTEM - Trying to start grizzly http server on fallback ipv4: 0.0.0.0
2023-08-08T17:04:14.761325+00:00 px4 Controller[1694726]: Aug 08, 2023 5:04:14 PM org.glassfish.jersey.server.wadl.WadlFeature configure
2023-08-08T17:04:14.761417+00:00 px4 Controller[1694726]: WARNING: JAX-B API not found . WADL feature is disabled.
2023-08-08T17:04:14.961809+00:00 px4 Controller[1694726]: Aug 08, 2023 5:04:14 PM org.glassfish.grizzly.http.server.NetworkListener start
2023-08-08T17:04:14.961957+00:00 px4 Controller[1694726]: INFO: Started listener bound to [0.0.0.0:3370]
2023-08-08T17:04:14.970258+00:00 px4 Controller[1694726]: Aug 08, 2023 5:04:14 PM org.glassfish.grizzly.http.server.HttpServer start
2023-08-08T17:04:14.970501+00:00 px4 Controller[1694726]: INFO: [HttpServer-1] Started.
2023-08-08T17:04:14.973940+00:00 px4 Controller[1694726]: 17:04:14.973 [Main] INFO  LINSTOR/Controller - SYSTEM - Controller initialized

I use:

# apt list | grep linstor

linstor-client/unknown,now 1.19.0-1 all [installed]
linstor-common/unknown,now 1.24.0-1 all [installed,automatic]
linstor-controller/unknown,now 1.24.0-1 all [installed]
linstor-proxmox/unknown,now 7.0.1-1 all [installed]
linstor-satellite/unknown,now 1.24.0-1 all [installed]
python-linstor/unknown,now 1.19.0-1 all [installed,automatic]

# lsb_release -a

No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:    12
Codename:   bookworm
ggzengel commented 1 year ago

After deleting /LINSTOR/SPACE_HISTORY/ and setting /LINSTOR/TRACKING_DATE/ENTRY_DATE the REST-API works.

etcdctl --user=root:$pwd del /LINSTOR/SPACE_HISTORY/ --prefix etcdctl --user=root:$pwd put /LINSTOR/TRACKING_DATE/ENTRY_DATE 2023-08-08

this is related to: https://github.com/LINBIT/linstor-server/issues/365