search

eu-nebulous / monitoring

Mozilla Public License 2.0
0 stars 0 forks source link

EMS not starting on application cluster on PROD environment #39

Open robert-sanfeliu opened 4 days ago

robert-sanfeliu commented 4 days ago

During a deployment of an application on PROD environment, EMS on the application cluster fails with: 

2024-11-25T11:11:10.893Z  INFO 30 --- [           main] o.s.boot.SpringApplication               :

 ███████╗███╗   ███╗███████╗     ██████╗ ██████╗ ███╗   ██╗████████╗██████╗  ██████╗ ██╗
 ██╔════╝████╗ ████║██╔════╝    ██╔════╝██╔═══██╗████╗  ██║╚══██╔══╝██╔══██╗██╔═══██╗██║
 █████╗  ██╔████╔██║███████╗    ██║     ██║   ██║██╔██╗ ██║   ██║   ██████╔╝██║   ██║██║
 ██╔══╝  ██║╚██╔╝██║╚════██║    ██║     ██║   ██║██║╚██╗██║   ██║   ██╔══██╗██║   ██║██║
 ███████╗██║ ╚═╝ ██║███████║    ╚██████╗╚██████╔╝██║ ╚████║   ██║   ██║  ██║╚██████╔╝███████╗
 ╚══════╝╚═╝     ╚═╝╚══════╝     ╚═════╝ ╚═════╝ ╚═╝  ╚═══╝   ╚═╝   ╚═╝  ╚═╝ ╚═════╝ ╚══════╝

 :: EMS Control ::                        (7.0.0-SNAPSHOT)
 :: Spring Boot ::                        (v3.2.10)
 :: Java (TM)   ::                        (21.0.3)
 :: Build Num.  :: 0271dfe3a763e996e433c7465e0cce23527132dc
 :: Build Date  :: 2024-10-04 18:01:15.455+0300
 :: SCM Branch  :: master,   at Repos.: https://gitlab.com/nebulous-project/ems-main.git
 :: Image Tag   :: ems-server:2024-oct
 :: Description :: *****  Preparing EMS for NebulOus  *****
                  ==========  EMS Nebulous Docker Image details  ==========
 :: Build Num.  :: 3e6cb1ebfb382d4e29104d50e87afafc4d9026cc
 :: Build Date  :: 2024-10-04 18:01:45.419+0300
 :: SCM Branch  :: master,   at Repos.: https://gitlab.com/nebulous-project/ems-nebulous.git
 :: Image Tag   :: ems-server:2024-oct-nebulous
 :: Description :: EMS Nebulous Docker Image is based on the EMS Core Docker Image

2024-11-25T11:11:11.132Z  INFO 30 --- [           main] g.i.imu.ems.util.NetUtilPostProcessor    : NetUtilPostProcessor:  NetUtilPropertySource registered (deferred log)
2024-11-25T11:11:11.138Z  INFO 30 --- [           main] g.i.i.e.c.ControlServiceApplication      : Starting ControlServiceApplication using Java 21.0.3 with PID 30 (/opt/ems-server/BOOT-INF/classes started by emsuser in /opt/ems-server)
2024-11-25T11:11:11.140Z  INFO 30 --- [           main] g.i.i.e.c.ControlServiceApplication      : No active profile set, falling back to 1 default profile: "default"
2024-11-25T11:11:17.190Z  INFO 30 --- [           main] faultConfiguringBeanFactoryPostProcessor : No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created.
2024-11-25T11:11:17.205Z  INFO 30 --- [           main] faultConfiguringBeanFactoryPostProcessor : No bean named 'integrationHeaderChannelRegistry' has been explicitly defined. Therefore, a default DefaultHeaderChannelRegistry will be created.
2024-11-25T11:11:18.733Z  WARN 30 --- [           main] gr.iccs.imu.ems.util.NetUtil             : NetUtil.getPublicIpAddress(): No Public IP address or connectivity problems exist
2024-11-25T11:11:19.162Z  INFO 30 --- [           main] gr.iccs.imu.ems.util.PasswordUtil        : PasswordUtil: Initialized default Password Encoder: gr.iccs.imu.ems.util.password.AsterisksPasswordEncoder
2024-11-25T11:11:21.846Z  INFO 30 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 8111 (https)
2024-11-25T11:11:21.901Z  INFO 30 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2024-11-25T11:11:21.902Z  INFO 30 --- [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.30]
2024-11-25T11:11:22.110Z  INFO 30 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2024-11-25T11:11:22.114Z  INFO 30 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 10185 ms
2024-11-25T11:11:23.352Z  INFO 30 --- [           main] g.i.i.e.b.c.s.SelfHealingManagerImpl     : Self-Healing Manager initialized
2024-11-25T11:11:23.356Z  INFO 30 --- [           main] g.i.i.e.b.c.s.SelfHealingManagerImpl     : Recovery context: RecoveryContext(variablesMap={BAGUETTE_CLIENT_BASE_DIR=~/baguette-client})
2024-11-25T11:11:23.578Z  INFO 30 --- [           main] g.i.i.e.baguette.server.BaguetteServer   : BaguetteServer: Generated new username/password: username=user-8522a3b9-ecde-46fc-9f54-7d6b80c84681, password=********
2024-11-25T11:11:24.865Z  INFO 30 --- [           main] eu.nebulous.ems.EmsNebulousProperties    : EmsNebulousProperties(applicationId=b1b63ef2-2fe1-4905-902d-191167a3555d, appId=null, appUid=null, appUuid=null, emsServerPodUid=8f98de99-e377-45a9-87b2-62bd7caef2c7, emsServerPodNamespace=default, appPodLabel=app)
2024-11-25T11:11:24.879Z  INFO 30 --- [           main] sEmsClientDeploymentPostProcessingPlugin : K8sEmsClientDeploymentPostProcessingPlugin: Application Id (from Env.): b1b63ef2-2fe1-4905-902d-191167a3555d
2024-11-25T11:11:26.602Z  INFO 30 --- [           main] o.apache.activemq.broker.BrokerService   : Loaded the Bouncy Castle security provider at position: -1
2024-11-25T11:11:27.211Z  INFO 30 --- [           main] o.apache.activemq.broker.BrokerService   : Using Persistence Adapter: MemoryPersistenceAdapter
2024-11-25T11:11:27.225Z  INFO 30 --- [           main] o.apache.activemq.broker.BrokerService   : Starting Persistence Adapter: MemoryPersistenceAdapter
2024-11-25T11:11:27.227Z  INFO 30 --- [           main] o.apache.activemq.broker.BrokerService   : Starting Temp Data Store
2024-11-25T11:11:27.228Z  INFO 30 --- [           main] o.apache.activemq.broker.BrokerService   : Starting Job Scheduler Store
2024-11-25T11:11:27.228Z  INFO 30 --- [           main] o.apache.activemq.broker.BrokerService   : Persistence Adapter successfully started
2024-11-25T11:11:27.425Z  INFO 30 --- [           main] o.apache.activemq.broker.BrokerService   : Apache ActiveMQ 6.1.3 (broker, ID:ems-ems-server-5fd97d59f7-n7njw-44753-1732533087269-0:1) is starting
2024-11-25T11:11:27.452Z  INFO 30 --- [           main] o.a.a.t.TransportServerThreadSupport     : Listening for connections at: ssl://ems-ems-server-5fd97d59f7-n7njw:61617?transport.daemon=true&transport.trace=false&transport.useKeepAlive=true&transport.useInactivityMonitor=false&transport.needClientAuth=false&transport.verifyHostName=true&transport.connectionTimeout=0&transport.keepAlive=true
2024-11-25T11:11:27.466Z  INFO 30 --- [           main] o.a.activemq.broker.TransportConnector   : Connector ssl://ems-ems-server-5fd97d59f7-n7njw:61617?transport.daemon=true&transport.trace=false&transport.useKeepAlive=true&transport.useInactivityMonitor=false&transport.needClientAuth=false&transport.verifyHostName=true&transport.connectionTimeout=0&transport.keepAlive=true started
2024-11-25T11:11:27.468Z  INFO 30 --- [           main] o.a.a.t.TransportServerThreadSupport     : Listening for connections at: tcp://ems-ems-server-5fd97d59f7-n7njw:61616?transport.daemon=true&transport.trace=false&transport.useKeepAlive=true&transport.useInactivityMonitor=false&transport.needClientAuth=false&transport.verifyHostName=true&transport.connectionTimeout=0&transport.keepAlive=true
2024-11-25T11:11:27.470Z  INFO 30 --- [           main] o.a.activemq.broker.TransportConnector   : Connector tcp://ems-ems-server-5fd97d59f7-n7njw:61616?transport.daemon=true&transport.trace=false&transport.useKeepAlive=true&transport.useInactivityMonitor=false&transport.needClientAuth=false&transport.verifyHostName=true&transport.connectionTimeout=0&transport.keepAlive=true started
2024-11-25T11:11:27.472Z  INFO 30 --- [           main] o.a.a.t.TransportServerThreadSupport     : Listening for connections at: stomp://ems-ems-server-5fd97d59f7-n7njw:61610?transport.daemon=true&transport.trace=false&transport.useKeepAlive=true&transport.useInactivityMonitor=false&transport.needClientAuth=false&transport.verifyHostName=true&transport.connectionTimeout=0&transport.keepAlive=true
2024-11-25T11:11:27.473Z  INFO 30 --- [           main] o.a.activemq.broker.TransportConnector   : Connector stomp://ems-ems-server-5fd97d59f7-n7njw:61610?transport.daemon=true&transport.trace=false&transport.useKeepAlive=true&transport.useInactivityMonitor=false&transport.needClientAuth=false&transport.verifyHostName=true&transport.connectionTimeout=0&transport.keepAlive=true started
2024-11-25T11:11:27.476Z  INFO 30 --- [           main] o.a.a.t.TransportServerThreadSupport     : Listening for connections at: amqp://ems-ems-server-5fd97d59f7-n7njw:5672?transport.daemon=true&transport.trace=false&transport.useKeepAlive=true&transport.useInactivityMonitor=false&transport.needClientAuth=false&transport.verifyHostName=true&transport.connectionTimeout=0&transport.keepAlive=true
2024-11-25T11:11:27.478Z  INFO 30 --- [           main] o.a.activemq.broker.TransportConnector   : Connector amqp://ems-ems-server-5fd97d59f7-n7njw:5672?transport.daemon=true&transport.trace=false&transport.useKeepAlive=true&transport.useInactivityMonitor=false&transport.needClientAuth=false&transport.verifyHostName=true&transport.connectionTimeout=0&transport.keepAlive=true started
2024-11-25T11:11:27.478Z  INFO 30 --- [           main] o.apache.activemq.broker.BrokerService   : Apache ActiveMQ 6.1.3 (broker, ID:ems-ems-server-5fd97d59f7-n7njw-44753-1732533087269-0:1) started
2024-11-25T11:11:27.478Z  INFO 30 --- [           main] o.apache.activemq.broker.BrokerService   : For help or more information please see: http://activemq.apache.org
2024-11-25T11:11:27.516Z  INFO 30 --- [       Thread-5] .i.i.e.b.b.i.MessageForwarderInterceptor : MessageQueueProcessor: Starts processing message queue and forward messages
2024-11-25T11:11:27.732Z  INFO 30 --- [           main] c.e.e.c.service.EPServiceProviderImpl    : Initializing engine URI 'default' version 7.1.0
2024-11-25T11:11:28.467Z  INFO 30 --- [           main] e.n.ems.translate.NebulousEmsTranslator  : NebulousEmsTranslator: initialized
2024-11-25T11:11:28.558Z  INFO 30 --- [           main] g.i.i.e.c.c.ControlServiceCoordinator    : ControlServiceCoordinator.initTranslator(): Effective translator: eu.nebulous.ems.translate.NebulousEmsTranslator
2024-11-25T11:11:28.560Z  INFO 30 --- [           main] eu.nebulous.ems.service.MvvService       : MvvService: initialized
2024-11-25T11:11:28.561Z  INFO 30 --- [           main] g.i.i.e.baguette.server.BaguetteServer   : BaguetteServer.startServer(): Starting SSH server...
2024-11-25T11:11:28.565Z  INFO 30 --- [           main] gr.iccs.imu.ems.baguette.server.Sshd     : ** SSH server **
2024-11-25T11:11:28.567Z  INFO 30 --- [           main] gr.iccs.imu.ems.baguette.server.Sshd     : SSH server: Public IP address: 10.0.0.1
2024-11-25T11:11:28.567Z  INFO 30 --- [           main] gr.iccs.imu.ems.baguette.server.Sshd     : SSH server:  Starting on port: 2222
2024-11-25T11:11:28.567Z  INFO 30 --- [           main] gr.iccs.imu.ems.baguette.server.Sshd     : SSH server:   Server key file: /opt/ems-server/config/hostkey.pem
2024-11-25T11:11:28.669Z  INFO 30 --- [           main] s.c.u.s.e.EdDSASecurityProviderRegistrar : getOrCreateProvider(EdDSA) created instance of net.i2p.crypto.eddsa.EdDSASecurityProvider
2024-11-25T11:11:28.898Z  INFO 30 --- [           main] o.a.s.s.k.SimpleGeneratorHostKeyProvider : generateKeyPair(EC) generating host key=nistp521
2024-11-25T11:11:29.238Z  INFO 30 --- [           main] gr.iccs.imu.ems.baguette.server.Sshd     : SSH server: Ready
2024-11-25T11:11:29.242Z  INFO 30 --- [           main] g.i.i.e.baguette.server.BaguetteServer   : BaguetteServer.startServer(): Starting SSH server... done
2024-11-25T11:11:29.243Z  INFO 30 --- [       Thread-6] gr.iccs.imu.ems.baguette.server.Sshd     : --> Heartbeat: Started: period=60000ms
2024-11-25T11:11:29.377Z  INFO 30 --- [           main] g.iccs.imu.ems.control.util.TopicBeacon  : Topic Beacon started: init-delay=60000ms, delay=60000ms
2024-11-25T11:11:29.389Z  INFO 30 --- [           main] g.i.imu.ems.common.plugin.PluginManager  : PluginManager: Started
2024-11-25T11:11:29.794Z  INFO 30 --- [           main] g.iccs.imu.ems.brokercep.EventForwarder  : EventForwarder: Starting event publish/forward worker
2024-11-25T11:11:29.842Z  INFO 30 --- [           main] gr.iccs.imu.ems.brokercep.StatsPrinter   : BCEP Statistics Printer disabled
2024-11-25T11:11:30.055Z  INFO 30 --- [           main] g.i.i.e.c.misc.SystemResourceMonitor     : SystemResourceMonitor started
2024-11-25T11:11:30.325Z  INFO 30 --- [           main] g.i.i.e.c.c.AbstractEndpointCollector    : Collectors::ServerPrometheusCollector: New Allowed Topics: null -- Topics Map: null
2024-11-25T11:11:30.516Z  INFO 30 --- [           main] .c.i.ControlServiceInfoEndpointExtension : Info endpoint is enabled and exposed. Added EMS info extension.
2024-11-25T11:11:30.547Z  INFO 30 --- [           main] g.i.i.e.c.c.AbstractEndpointCollector    : Collectors::ServerNetdataCollector: New Allowed Topics: null -- Topics Map: null
2024-11-25T11:11:30.702Z  INFO 30 --- [           main] eu.nebulous.ems.k8s.K8sPodWatcher        : K8sPodWatcher: Enabled  (running every PT1M, init-delay=PT30S)
2024-11-25T11:11:30.719Z  INFO 30 --- [           main] eu.nebulous.ems.boot.EmsBootProperties   : EmsBootProperties(enabled=false, processorPeriod=500, applicationIdPropertyName=application, modelsDir=models, modelsIndexFile=models/index.json, dslTopic=eu.nebulouscloud.ui.dsl.generic, optimiserMetricsTopic=eu.nebulouscloud.optimiser.controller.metric_list, modelsTopic=eu.nebulouscloud.ui.dsl.metric_model, modelsResponseTopic=eu.nebulouscloud.ui.dsl.metric_model.reply, emsBootTopic=eu.nebulouscloud.ems.boot, emsBootResponseTopic=eu.nebulouscloud.ems.boot.reply, validateModels=true, storeInvalidModels=true, reportInvalidModels=true)
2024-11-25T11:11:30.751Z  INFO 30 --- [           main] eu.nebulous.ems.boot.ModelsService       : ModelsService is disabled because EMS Boot is disabled
2024-11-25T11:11:30.754Z  INFO 30 --- [           main] e.n.ems.boot.NebulousEventsService       : EMS Boot is disabled
2024-11-25T11:11:30.764Z  INFO 30 --- [           main] e.n.e.s.EmsBootInitializerProperties     : EmsBootInitializerProperties(enabled=true, initialWait=PT1S, retryPeriod=PT30S)
2024-11-25T11:11:30.778Z  INFO 30 --- [           main] e.n.e.s.ExternalBrokerListenerService    : ExternalBrokerListenerService: Application Id: b1b63ef2-2fe1-4905-902d-191167a3555d
2024-11-25T11:11:31.558Z  INFO 30 --- [           main] e.n.e.s.ExternalBrokerListenerService    : ExternalBrokerListenerService: created subscribers
2024-11-25T11:11:31.563Z  INFO 30 --- [           main] e.n.e.s.ExternalBrokerListenerService    : ExternalBrokerListenerService: created publishers
2024-11-25T11:11:31.846Z  INFO 30 --- [           main] eu.nebulouscloud.exn.Connector           : Starting connector...
Signaling EMS to exit
2024-11-25T11:11:32.158Z  INFO 30 --- [           main] eu.nebulouscloud.exn.core.Context        : Registering 2 consumers
2024-11-25T11:11:32.871Z  INFO 30 --- [:1): I/O Thread] o.a.q.p.client.impl.ClientConnection     : Connection ID:a5638fc5-3abc-4222-9875-30f420772c0a:1:1 connected to server: 158.37.63.86:32755
2024-11-25T11:11:33.195Z  INFO 30 --- [pool-6-thread-2] eu.nebulouscloud.exn.core.Manager        : Linking consumer topic://eu.nebulouscloud.monitoring.commands
2024-11-25T11:11:33.466Z  INFO 30 --- [pool-6-thread-3] eu.nebulouscloud.exn.core.Manager        : Linking consumer topic://eu.nebulouscloud.optimiser.solver.solution
2024-11-25T11:11:33.468Z  INFO 30 --- [pool-6-thread-3] eu.nebulouscloud.exn.core.Manager        :       for application b1b63ef2-2fe1-4905-902d-191167a3555d
2024-11-25T11:11:33.468Z  INFO 30 --- [           main] eu.nebulouscloud.exn.core.Context        : Registering 1 publishers
2024-11-25T11:11:33.485Z  INFO 30 --- [           main] e.n.e.s.AbstractExternalBrokerService    : AbstractExternalBrokerService: Connected to broker
2024-11-25T11:11:33.485Z  INFO 30 --- [           main] e.n.e.s.ExternalBrokerListenerService    : ExternalBrokerListenerService: Initialized listeners and publishers
2024-11-25T11:11:33.499Z  INFO 30 --- [           main] e.n.ems.translate.analyze.AnalysisUtils  : AnalysisUtils: useCompositeNames: false
2024-11-25T11:11:35.730Z  INFO 30 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 2 endpoint(s) beneath base path '/actuator'
2024-11-25T11:11:35.886Z  INFO 30 --- [           main] o.s.s.web.DefaultSecurityFilterChain     : Will secure any request with [org.springframework.security.web.session.DisableEncodeUrlFilter@2eaaef66, org.springframework.security.web.session.ForceEagerSessionCreationFilter@6fab250b, org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter@3a8dd899, org.springframework.security.web.context.SecurityContextHolderFilter@491cceb4, org.springframework.security.web.header.HeaderWriterFilter@77cac3b, org.springframework.security.web.authentication.logout.LogoutFilter@62509b77, org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter@400c0d26, gr.iccs.imu.ems.control.webconf.WebSecurityConfig$$Lambda/0x00007c50bcbb3de0@3da068e8, gr.iccs.imu.ems.control.webconf.WebSecurityConfig$$Lambda/0x00007c50bcbb4020@448861e5, gr.iccs.imu.ems.control.webconf.WebSecurityConfig$$Lambda/0x00007c50bcbb4260@72bd8702, gr.iccs.imu.ems.control.webconf.WebSecurityConfig$$Lambda/0x00007c50bcbb44a0@2620d0dc, org.springframework.security.web.savedrequest.RequestCacheAwareFilter@504ed038, org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter@3fb8e614, org.springframework.security.web.authentication.AnonymousAuthenticationFilter@18f90bd8, org.springframework.security.web.session.SessionManagementFilter@74f23ecc, org.springframework.security.web.access.ExceptionTranslationFilter@35984172, org.springframework.security.web.access.intercept.AuthorizationFilter@34ec33f4]
2024-11-25T11:11:36.186Z  WARN 30 --- [           main] ion$DefaultTemplateResolverConfiguration : Cannot find template location: classpath:/templates/ (please add some templates, check your Thymeleaf configuration, or set spring.thymeleaf.check-template-location=false)
2024-11-25T11:11:36.667Z  WARN 30 --- [           main] .b.a.g.t.GroovyTemplateAutoConfiguration : Cannot find template location: classpath:/templates/ (please add some templates, check your Groovy configuration, or set spring.groovy.template.check-template-location=false)
2024-11-25T11:11:37.501Z  INFO 30 --- [           main] o.s.i.endpoint.EventDrivenConsumer       : Adding {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2024-11-25T11:11:37.502Z  INFO 30 --- [           main] o.s.i.channel.PublishSubscribeChannel    : Channel 'application.errorChannel' has 1 subscriber(s).
2024-11-25T11:11:37.502Z  INFO 30 --- [           main] o.s.i.endpoint.EventDrivenConsumer       : started bean '_org.springframework.integration.errorLogger'
2024-11-25T11:11:37.734Z  INFO 30 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port 8111 (https) with context path ''
2024-11-25T11:11:37.739Z  INFO 30 --- [           main] i.i.e.b.c.i.h.AbstractInstallationHelper : Server PEM certificate stored in file: /opt/ems-server/config/server.pem
2024-11-25T11:11:37.741Z  INFO 30 --- [           main] i.i.e.b.c.i.h.AbstractInstallationHelper : Server PEM certificate stored in file: /opt/ems-server/config/server.pem
2024-11-25T11:11:37.790Z  INFO 30 --- [ionShutdownHook] g.i.imu.ems.brokercep.BrokerCepConsumer  : BrokerCepConsumer is shutting down
2024-11-25T11:11:37.793Z  INFO 30 --- [ionShutdownHook] g.i.i.e.b.broker.BrokerAdvisoryWatcher   : BrokerAdvisoryWatcher is shutting down
2024-11-25T11:11:37.811Z  INFO 30 --- [           main] g.i.i.e.c.ControlServiceApplication      : Started ControlServiceApplication in 28.253 seconds (process running for 30.145)
2024-11-25T11:11:37.823Z  INFO 30 --- [ionShutdownHook] o.s.i.endpoint.EventDrivenConsumer       : Removing {logging-channel-adapter:_org.springframework.integration.errorLogger} as a subscriber to the 'errorChannel' channel
2024-11-25T11:11:37.823Z  INFO 30 --- [ionShutdownHook] o.s.i.channel.PublishSubscribeChannel    : Channel 'application.errorChannel' has 0 subscriber(s).
2024-11-25T11:11:37.825Z  INFO 30 --- [ionShutdownHook] o.s.i.endpoint.EventDrivenConsumer       : stopped bean '_org.springframework.integration.errorLogger'
2024-11-25T11:11:37.844Z  INFO 30 --- [           main] .s.b.a.l.ConditionEvaluationReportLogger :

Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.
2024-11-25T11:11:37.864Z  INFO 30 --- [ionShutdownHook] o.apache.activemq.broker.BrokerService   : Apache ActiveMQ 6.1.3 (broker, ID:ems-ems-server-5fd97d59f7-n7njw-44753-1732533087269-0:1) is shutting down
2024-11-25T11:11:37.870Z  INFO 30 --- [ionShutdownHook] o.a.activemq.broker.TransportConnector   : Connector ssl://ems-ems-server-5fd97d59f7-n7njw:61617?transport.daemon=true&transport.trace=false&transport.useKeepAlive=true&transport.useInactivityMonitor=false&transport.needClientAuth=false&transport.verifyHostName=true&transport.connectionTimeout=0&transport.keepAlive=true stopped
2024-11-25T11:11:37.870Z  INFO 30 --- [.keepAlive=true] o.a.a.transport.tcp.TcpTransportServer   : socketQueue interrupted - stopping
2024-11-25T11:11:37.871Z  INFO 30 --- [.keepAlive=true] o.a.activemq.broker.TransportConnector   : Could not accept connection during shutdown  : null (null)
2024-11-25T11:11:37.876Z  INFO 30 --- [.keepAlive=true] o.a.a.transport.tcp.TcpTransportServer   : socketQueue interrupted - stopping
2024-11-25T11:11:37.879Z  INFO 30 --- [.keepAlive=true] o.a.activemq.broker.TransportConnector   : Could not accept connection during shutdown  : null (null)
2024-11-25T11:11:37.889Z  WARN 30 --- [0.1:35376@61616] o.a.a.b.TransportConnection.Transport    : Transport Connection to: tcp://127.0.0.1:35376 failed: Unexpected error occurred: org.apache.activemq.broker.BrokerStoppedException: Broker BrokerService[broker] is being stopped
2024-11-25T11:11:37.890Z ERROR 30 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'applicationAvailability': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:220)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:205)
        at org.springframework.context.event.AbstractApplicationEventMulticaster.retrieveApplicationListeners(AbstractApplicationEventMulticaster.java:265)
        at org.springframework.context.event.AbstractApplicationEventMulticaster.getApplicationListeners(AbstractApplicationEventMulticaster.java:222)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:145)
        at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:452)
        at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:385)
        at org.springframework.boot.availability.AvailabilityChangeEvent.publish(AvailabilityChangeEvent.java:92)
        at org.springframework.boot.availability.AvailabilityChangeEvent.publish(AvailabilityChangeEvent.java:78)
        at org.springframework.boot.context.event.EventPublishingRunListener.started(EventPublishingRunListener.java:104)
        at org.springframework.boot.SpringApplicationRunListeners.lambda$started$5(SpringApplicationRunListeners.java:76)
        at java.base/java.lang.Iterable.forEach(Unknown Source)
        at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118)
        at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112)
        at org.springframework.boot.SpringApplicationRunListeners.started(SpringApplicationRunListeners.java:76)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:341)
        at gr.iccs.imu.ems.control.ControlServiceApplication.main(ControlServiceApplication.java:70)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:102)
        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:64)
        at org.springframework.boot.loader.launch.PropertiesLauncher.main(PropertiesLauncher.java:580)

2024-11-25T11:11:37.909Z  INFO 30 --- [ionShutdownHook] o.a.activemq.broker.TransportConnector   : Connector tcp://ems-ems-server-5fd97d59f7-n7njw:61616?transport.daemon=true&transport.trace=false&transport.useKeepAlive=true&transport.useInactivityMonitor=false&transport.needClientAuth=false&transport.verifyHostName=true&transport.connectionTimeout=0&transport.keepAlive=true stopped
2024-11-25T11:11:37.912Z  INFO 30 --- [ionShutdownHook] o.a.activemq.broker.TransportConnector   : Connector stomp://ems-ems-server-5fd97d59f7-n7njw:61610?transport.daemon=true&transport.trace=false&transport.useKeepAlive=true&transport.useInactivityMonitor=false&transport.needClientAuth=false&transport.verifyHostName=true&transport.connectionTimeout=0&transport.keepAlive=true stopped
2024-11-25T11:11:37.914Z  INFO 30 --- [ionShutdownHook] o.a.activemq.broker.TransportConnector   : Connector amqp://ems-ems-server-5fd97d59f7-n7njw:5672?transport.daemon=true&transport.trace=false&transport.useKeepAlive=true&transport.useInactivityMonitor=false&transport.needClientAuth=false&transport.verifyHostName=true&transport.connectionTimeout=0&transport.keepAlive=true stopped
2024-11-25T11:11:37.925Z  INFO 30 --- [ionShutdownHook] o.apache.activemq.broker.BrokerService   : Apache ActiveMQ 6.1.3 (broker, ID:ems-ems-server-5fd97d59f7-n7njw-44753-1732533087269-0:1) uptime 10.746 seconds
2024-11-25T11:11:37.928Z  INFO 30 --- [ionShutdownHook] o.apache.activemq.broker.BrokerService   : Apache ActiveMQ 6.1.3 (broker, ID:ems-ems-server-5fd97d59f7-n7njw-44753-1732533087269-0:1) is shutdown
2024-11-25T11:11:37.950Z  INFO 30 --- [0.1:61616@35356] o.a.a.jms.pool.PooledConnectionFactory   : Expiring connection ActiveMQConnection {id=ID:ems-ems-server-5fd97d59f7-n7njw-44753-1732533087269-4:1,clientId=ID:ems-ems-server-5fd97d59f7-n7njw-44753-1732533087269-3:1,started=false} on IOException: null
robert-sanfeliu commented 4 days ago

Manually deleting the "ems-server" pod and starting it again solved the issue. Also, I tried another deployment on the same env and worked OK out of the box