ELB failed to put instances in service within 300 seconds

[dbeja]

I've been using Boxfuse for a while but now when I try to deploy some changes it always fails with the following error:

[WARNING] WARNING: Run failed: Time out: ELB failed to put instances in service within 300 seconds
=> check the instance logs
=> ensure your application responds with an HTTP 200 at /health on port 443
=> ensure the healthcheck configuration (healthcheck.port, healthcheck.path, healthcheck.timeout) is correct

I didn't change anything related to ELB.

[axelfontaine]

What do your application logs say?

[dbeja]

I'm using with a Spring Boot application and Maven plugin.

This was the log of the last "start". It also seems to be always starting.

[axelfontaine]

Nothing has changed on our end. So this sounds like a case where you have to crack open the debugger and figure out what bean/external service exactly is preventing your application from starting correctly.

[dbeja]

If I run locally using Spring Boot it runs fine but when I try to mvn package to then use boxfuse manually to deploy I get this error:

2020-10-14 15:48:29.617  WARN 56266 --- [           main] o.s.w.c.s.GenericWebApplicationContext   : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'springSecurityFilterChain' defined in class path resource [org/springframework/security/config/annotation/web/configuration/WebSecurityConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.servlet.Filter]: Factory method 'springSecurityFilterChain' threw exception; nested exception is java.lang.RuntimeException: javax.xml.bind.JAXBException
 - with linked exception:
[java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory]
2020-10-14 15:48:29.618  INFO 56266 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Shutting down ExecutorService 'brokerChannelExecutor'
2020-10-14 15:48:29.618  INFO 56266 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Shutting down ExecutorService 'messageBrokerTaskScheduler'
2020-10-14 15:48:29.620  INFO 56266 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Shutting down ExecutorService 'clientOutboundChannelExecutor'
2020-10-14 15:48:29.620  INFO 56266 --- [           main] o.s.s.concurrent.ThreadPoolTaskExecutor  : Shutting down ExecutorService 'clientInboundChannelExecutor'
2020-10-14 15:48:29.622  INFO 56266 --- [           main] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2020-10-14 15:48:29.635  INFO 56266 --- [           main] utoConfigurationReportLoggingInitializer : 

Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enabled.
2020-10-14 15:48:29.642 ERROR 56266 --- [           main] o.s.boot.SpringApplication               : Application startup failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'springSecurityFilterChain' defined in class path resource [org/springframework/security/config/annotation/web/configuration/WebSecurityConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.servlet.Filter]: Factory method 'springSecurityFilterChain' threw exception; nested exception is java.lang.RuntimeException: javax.xml.bind.JAXBException
 - with linked exception:
[java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599) ~[spring-beans-4.3.4.RELEASE.jar:4.3.4.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1128) ~[spring-beans-4.3.4.RELEASE.jar:4.3.4.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1022) ~[spring-beans-4.3.4.RELEASE.jar:4.3.4.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:512) ~[spring-beans-4.3.4.RELEASE.jar:4.3.4.RELEASE]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) ~[spring-beans-4.3.4.RELEASE.jar:4.3.4.RELEASE]

I'm trying new to figure out if this is the cause and why it's happening now.

[dbeja]

I added some dependencies needed for more recent Java versions and it seemed to give me a bit more feedback on the logs.

The process seems to go fine but then I get this error on the logs: [INFO] ELB: i-XXXXXXXXX [OutOfService] => Instance has failed at least the UnhealthyThreshold number of health checks consecutively. Instance deregistration currently in progress.

I checked on EC2 and that instance is terminated. It's another one that is running. And I've also noticed that I have there a lot of terminated instances for the same app today.

[dbeja]

Ok, found the issue. For some reason, the health endpoints started to return DOWN and ELB depends on that status.

I fixed the health status and it's now working back again.