Allow the embedded web server to be shut down gracefully

[LutzStrobel]

We are using spring-boot and spring-cloud to realize a micro service archtecture. During load tests we are facing lots of errors such as "entitymanager closed and others" if we shut the micro service down during load. We are wondering if there is an option to configure the embedded container to shut its service connector down and waits for an empty request queue before shutting down the complete application context.

If there is no such option, I did not find any, then it would be great to extend the shutdownhook of spring to respect such requirements.

[wilkinsona]

We currently stop the application context and then stop the container. We did try to reverse this ordering but it had some unwanted side effects so we're stuck with it for now at least. That's the bad news.

The good news is that you can actually get a graceful shutdown yourself if you're happy to get your hands a bit dirty. The gist is that you need to pause Tomcat's connector and then wait for its thread pool to shutdown before allowing the destruction of the application context to proceed. It looks something like this:

package com.example;

import java.util.concurrent.Executor;
import java.util.concurrent.TimeUnit;

import org.apache.catalina.connector.Connector;
import org.apache.tomcat.util.threads.ThreadPoolExecutor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.context.embedded.ConfigurableEmbeddedServletContainer;
import org.springframework.boot.context.embedded.EmbeddedServletContainerCustomizer;
import org.springframework.boot.context.embedded.tomcat.TomcatConnectorCustomizer;
import org.springframework.boot.context.embedded.tomcat.TomcatEmbeddedServletContainerFactory;
import org.springframework.context.ApplicationListener;
import org.springframework.context.annotation.Bean;
import org.springframework.context.event.ContextClosedEvent;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@RestController
public class Gh4657Application {

    public static void main(String[] args) {
        SpringApplication.run(Gh4657Application.class, args);
    }

    @RequestMapping("/pause")
    public String pause() throws InterruptedException {
        Thread.sleep(10000);
        return "Pause complete";
    }

    @Bean
    public GracefulShutdown gracefulShutdown() {
        return new GracefulShutdown();
    }

    @Bean
    public EmbeddedServletContainerCustomizer tomcatCustomizer() {
        return new EmbeddedServletContainerCustomizer() {

            @Override
            public void customize(ConfigurableEmbeddedServletContainer container) {
                if (container instanceof TomcatEmbeddedServletContainerFactory) {
                    ((TomcatEmbeddedServletContainerFactory) container)
                            .addConnectorCustomizers(gracefulShutdown());
                }

            }
        };
    }

    private static class GracefulShutdown implements TomcatConnectorCustomizer,
            ApplicationListener<ContextClosedEvent> {

        private static final Logger log = LoggerFactory.getLogger(GracefulShutdown.class);

        private volatile Connector connector;

        @Override
        public void customize(Connector connector) {
            this.connector = connector;
        }

        @Override
        public void onApplicationEvent(ContextClosedEvent event) {
            this.connector.pause();
            Executor executor = this.connector.getProtocolHandler().getExecutor();
            if (executor instanceof ThreadPoolExecutor) {
                try {
                    ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) executor;
                    threadPoolExecutor.shutdown();
                    if (!threadPoolExecutor.awaitTermination(30, TimeUnit.SECONDS)) {
                        log.warn("Tomcat thread pool did not shut down gracefully within "
                                + "30 seconds. Proceeding with forceful shutdown");
                    }
                }
                catch (InterruptedException ex) {
                    Thread.currentThread().interrupt();
                }
            }
        }

    }

}

I think it makes sense to provide behaviour like this out of the box, or at least an option to enable it. However, that'll require a bit more work as it'll need to work with all of the embedded containers that we support (Jetty, Tomcat, and Undertow), cope with multiple connectors (HTTP and HTTPS, for example) and we'll need to think about the configuration options, if any, that we'd want to offer: switching it on or off, configuring the period that we wait for the thread pool to shutdown, etc.

[LutzStrobel]

Thank you for your advice. We currently simply shut down the embedded container and the waiting a short time before closing the application context.

What do you thing, will it be possible in future to shut down a spring boot application more gracefully?

[wilkinsona]

What do you thing, will it be possible in future to shut down a spring boot application more gracefully?

Yes. As I said above "I think it makes sense to provide behaviour like this out of the box, or at least an option to enable it".

I'm going to re-open this issue as we'll use it to track the possible enhancement.

[tkvangorder]

+1 on this request. We ran into a similar problem when load testing and dropping a node from the test. @wilkinsona in your example, I was thinking of using an implementation of smartlifecylce so I can insure the connector is shutdown first. You said you ran into issues shutting down tomcat first?

[bohrqiu]

I think the right order is:

1. pause the io endpoint

   web container just pasue(deny new request come in),and RPC framework need notify client don't send new request and wait current request be proccessed

2. wait the request to be processed and response client
3. close the io endpoint
4. close spring service
5. close log system

so we do it as follows:

    @Override
    public void finished(ConfigurableApplicationContext context, Throwable exception) {
        if (exception == null) {
            //install UncaughtExceptionHandler
            UncaughtExceptionHandlerWrapper.install();
            //when system startup ,register shutdown hooks to clean resouces.
            Runtime.getRuntime().addShutdownHook(new Thread() {
                @Override
                public void run() {
                    //run all shutdown hooks before spring container avoid service dependency
                    ShutdownHooks.shutdownAll();
                    //close spring container
                    context.close();
                    shutdownLogSystem();
                }
            });
            //log startup info
            LoggerFactory.getLogger(YijiApplicationRunListener.class).info("启动成功: http://127.0.0.1:{}",
            context.getEnvironment().getProperty(Apps.HTTP_PORT));
        } else {
            ShutdownHooks.shutdownAll();
            shutdownLogSystem();
        }
    }

[tkvangorder]

This issue is actually a bit more complicated.

I know that SmartLifeCycle can be used to set the order in which beans are notified of life cycle events. However, there is no generalized way for a service to know it should startup/shutdown before another service.

Consider the following:

A sprint boot application is running an embedded servlet container and is also producing/consuming JMS messages.

On the close event, the servlet container really needs to pause the connector first, process its remain working (any connections that have already been establish.)

We need to insure this is the FIRST thing that happens, prior to the JMS infrastructure shutting down because the work done inside tomcat may rely on JMS.

The JMS infrastructure has a similar requirement: it must stop listening for messages and chew through any remaining messages it has already accepted.

I can certainly implement a SmartLifeCycle class that sets the phase "very high"....and I could even create two instances, one for embedded servlet container and one for JMS and insure the correct order.

But in the spirit of Spring Boot, if I add a tomcat container, its my expectation that when the application shuts down, it will gracefully stop accepting connections, process remaining work, and exit.

It would be helpful if there was a mechanism to allow ordering to be expressed relative to other services "JMS must start before tomcat", "tomcat must close before JMS". This would be similar to the AutoConfigureBefore/AutoConfigureAfter annotations that are used in Spring boot.

One way to approach this might be to create an enum for the generalized services (This is not ideal, but I can't think of another way without introducing artificial, compile time dependencies.):

EMBEDDED_CONTAINER JMS DISCOVERY_CLIENT . . The annotations could leverage the enums to order the life cycle listeners.

For now, its up to the developer to explicitly define the shutdown order of services via "SmartLifeCycle" instances...which can get a bit messy and seems like extra work for something that should really work out of the box.

[wilkinsona]

@tkvangorder You don't need to use SmartLifecycleto gracefully shut down Tomcat as shown in my comment above. It happens in response to a ContextClosedEvent which is fired at the very beginning of the context's close processing before any of the beans in the context are processed.

Beyond this, Spring Framework already has support for closing things down in the correct order. For example you can implement DisposableBean. When the container disposes of a bean, it will dispose of anything that depends on it first.

[thefallentree]

@wilkinsona is the order of shutdown anyway guaranteed ? Is spring boot always going to shutdown application context first? we are trying to implement what you said, but was not sure if this will change in the future.

[wilkinsona]

Is spring boot always going to shutdown application context first?

Yes. When you're using an embedded container, it's the application context being shut down that triggers the shutdown of the container.

[thefallentree]

I want to report back on how to setup this correctly using Jetty and springboot , use this class instead of EmbeddedServletAutoconfiguration and you are set.

public class HttpConfig {
    private static final Logger log = LoggerFactory.getLogger(HttpConfig.class);

    private static volatile Server server;

    // Jetty HTTP Server
    //
    // see [1] on how to implement graceful shutdown in springboot.
    // Note that since use jetty, we need to use server.stop(), also StatisticsHandler must be
    // configured, for jetty graceful shutdown to work.
    //
    // [1]: https://github.com/spring-projects/spring-boot/issues/4657
    @Bean
    @Autowired
    public JettyEmbeddedServletContainerFactory jettyEmbeddedServletContainerFactory(HttpSetting httpSetting) {
        JettyEmbeddedServletContainerFactory factory = new JettyEmbeddedServletContainerFactory();

        factory.setPort(httpSetting.getPort());
        log.info("Jetty configured on port: " + httpSetting.getPort());

        factory.addServerCustomizers(new JettyServerCustomizer() {
            @Override
            public void customize(Server server1) {
                server = server1;
                log.info("Jetty version: " + server.getVersion());

                // Configure shutdown wait time.
                if (httpSetting.getShutdownWaitTime() > 0) {
                    // Add StatsticsHandler, in order for graceful shutdown to work.
                    StatisticsHandler handler = new StatisticsHandler();
                    handler.setHandler(server.getHandler());
                    server.setHandler(handler);

                    log.info("Shutdown wait time: " + httpSetting.getShutdownWaitTime() + "s");
                    server.setStopTimeout(httpSetting.getShutdownWaitTime());

                    // We will stop it through JettyGracefulShutdown class.
                    server.setStopAtShutdown(false);
                }
            }
        });
        return factory;
    }

    @Bean
    public JettyGracefulShutdown jettyGracefulShutdown() { return new JettyGracefulShutdown(); }

    // Springboot closes application context before everything.
    private static class JettyGracefulShutdown implements ApplicationListener<ContextClosedEvent>{
        private static final Logger log = LoggerFactory.getLogger(JettyGracefulShutdown.class);

        @Override
        public void onApplicationEvent(ContextClosedEvent event) {
            if (server == null) {
                log.error("Jetty server variable is null, this should not happen!");
                return;
            }
            log.info("Entering shutdown for Jetty.");
            if (!(server.getHandler() instanceof StatisticsHandler)) {
                log.error("Root handler is not StatisticsHandler, graceful shutdown may not work at all!");
            } else {
                log.info("Active requests: " + ((StatisticsHandler) server.getHandler()).getRequestsActive());
            }
            try {
                long begin = System.currentTimeMillis();
                server.stop();
                log.info("Shutdown took " + (System.currentTimeMillis() - begin) + " ms.");
            } catch (Exception e) {
                log.error("Fail to shutdown gracefully.", e);
            }
        }
    }
}

[clanie]

The tomcat version didn't work as-is for me because this.connector.getProtocolHandler().getExecutor() returned null. The protocolHandler was a Http11NioProtocol. To make it work I let the customize(Connector connector) method also set an Executor on the connectors ProtocolHandler.

[ktracer-isp]

@clanie: Can you post the code you did for that? We're running into a similar problem and I'd like to see a working solution if you have one.

[czyoshida-naoko]

I found this article. http://martin.podval.eu/2015/06/java-docker-spring-boot-and-signals.html So I want to do exec java -jar even when I install my app as init.d service. How can I pass exec to a launch.script? or How can I use a customized launch.script than embedded launch.script? Is there any solutions?

[robocide]

Hi,

forgive me if i am mistaken. but the the question stated the request to "waits for an empty request queue before shutting "

isnt it simpler... to just count the number of requests (From Filter) ? and then when the number is 0 just signal for grceful shutdown ?

Thanks!

[wilkinsona]

isnt it simpler... to just count the number of requests (From Filter) ? and then when the number is 0 just signal for grceful shutdown ?

No, not really. You need to stop accepting new requests, wait for any active requests to complete, and then gracefully shut everything down. If you just wait for the number of active requests to reach 0 there's then a race condition between new requests being accepted and the shutdown completing.

[robocide]

@wilkinsona thanks for your response ! ... i would like to regard.

You said:

1. "You need to stop accepting new requests," -> My Solution below handles it.
2. "wait for any active requests to complete," -> My Solution below handles it.

Solution:

public class LimitFilter implements Filter {
    public static AtomicBoolean isRunning = true;
    public int count;
    private Object lock = new Object();

    public void doFilter(ServletRequest request, ServletResponse response,
            FilterChain chain) throws IOException, ServletException {
        try {           
            if (isRunning) {
                // let the request through and process as usual
                 synchronized (lock) {                
                      count++;
                 }
                try{
                chain.doFilter(request, response); 
                }finaly{
                       synchronized (lock) {                
                          count--;
                 }
                }
            } else {
               //return server unavailable http status 503 
            }
        } finally {
            synchronized (lock) {                
                   if(count==0 && !isRunning)
                       AppSingleton.signalSafeShutdown();
            }           
        }
    }

now from shutdown hook:

@PreDestroy public void cleanup(){ LimitFilter.isRunning=false; AppSingleton.waitSafeShutdown(); }

Description: ok now the Filter's isRunning can be controlled from listening to contextCloseEvent ..... now when the contextClosedEvent is fired we will do 2 things: 1] set LimitFilter.isRunning=false in order not recieve new requests. 2] wait for the current counter to be 0 in order to wait for current requests to finish.

i would be happy if someone sees any problem with this kind of approach rather than trying complex solution of reverting the shutdown order and handling container specific complex code.... ?

thanks. Avishay.

[wilkinsona]

@avishayhirsh Thanks for your efforts, but you are over-simplifying the problem.

Firstly, your solution does not stop accepting requests, instead it accepts them and then responds with a 503. We need to pause the container's connectors so that they stop allowing clients to connect at all. Secondly, your mechanism for counting active requests means that all requests briefly become single threaded. That will have an unacceptable impact on performance.

[robocide]

@wilkinsona "Thanks for your efforts, but you are over-simplifying the problem."

Well sometimes simple approach is good enough and currently i think allot of developers can live with the points you raise here which i will regard to.

"Firstly, your solution does not stop accepting requests, instead it accepts them and then responds with a 503. We need to pause the container's connectors so that they stop allowing clients to connect at all. "

That's correct , but actually its fine by me to do it like this way , but thanks for point this out !

"Secondly, your mechanism for counting active requests means that all requests become single threaded. That will have an unacceptable impact on performance."

Can you explain why ? the requests processing is not singleThreaded , only the counter inc+dec is....

@wilkinsona thanks again for having this converstion :) what do you think ?

[corentin59]

Hello,

This is a proof of concept : spring-boot-graceful-shutdown

Support for tomcat and undertow... and works with actuator.

[dfjones]

Hi All,

I stumbled onto this issue and saw some of the workarounds here and came up with one that seems a bit simpler than others had mentioned, so I’m curious if there are any downsides to this approach. Thanks for the feedback.

private final List<EmbeddedServletContainer> embeddedContainers = new ArrayList<>();

@EventListener
public synchronized void onContainerInitialized(EmbeddedServletContainerInitializedEvent event) {
  embeddedContainers.add(event.getEmbeddedServletContainer());
}

@EventListener
public synchronized void onApplicationContextClose(ContextClosedEvent contextClosedEvent) {
  for (EmbeddedServletContainer container : embeddedContainers) {
    container.stop();
  }
}

Regarding a proper fix for the issue, is there any update on if we should expect this to land in the Boot 2.0 release (or sooner)?

Also, I feel like it took some luck to find this thread and I was a bit surprised that clean shutdown is not being handled properly for servlet containers. I think it would be great if the Boot documentation could point out this issue and some of the workarounds.

[wilkinsona]

@dfjones The doesn't give inflight requests a grace period to complete and isn't much different to what already happens when the container is stopped as part of the context closing.

Regarding a proper fix for the issue, is there any update on if we should expect this to land in the Boot 2.0 release (or sooner)?

As shown by its milestone, the issue is currently targeted at 2.0.

[dfjones]

@wilkinsona Would a grace period for inflight requests be dependent on the servlet container impl? We are using Jetty. Taking a look at what they do when stop is called it seems like we are getting the desired graceful shutdown behavior (Jetty appears to be waiting for inflight requests to finish after closing the network connector).

For a little more detail on the issue we were seeing, without a workaround like what I posted, it appears that the container was not being stopped until after the context was already closed. It manifested in exceptions like the following resulting in HTTP 500s returned:

javax.servlet.ServletException: A MultiException has 1 exceptions.  They are:
1. org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'resource': Singleton bean creation not allowed while the singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)

    at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:489)
    at org.glassfish.jersey.servlet.ServletContainer.serviceImpl(ServletContainer.java:408)
    at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:583)
    at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:524)
    at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:461)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)

[MarcoGnecchi]

How should I go about gracefully shutdown a spring boot app with multiple connectors like the one implemented here? The shutdown of the embedded tomcat works fine (the HTTPS connector), but if I set up the additional connector in this way:

@Bean public EmbeddedServletContainerFactory servletContainer() {
Connector connector = new Connector("org.apache.coyote.http11.Http11NioProtocol");
connector.setPort(httpPort);
TomcatEmbeddedServletContainerFactory tomcat = new TomcatEmbeddedServletContainerFactory();
tomcat.addAdditionalTomcatConnectors(connector);
tomcat.addConnectorCustomizers(gracefulShutdown());
return tomcat;
} 

Spring never calls customize on this connector. Is there another way of doing this?

[wilkinsona]

@MarcoGnecchi You'd need to change GracefulShutdown so that it's aware of your additional connector in addition to the one that'll be passed into its customize method.

[rdramana85]

@wilkinsona Can you suggest an option for gracefulshutdown using Undertow. I see that there is a GracefulShutdownHandler but not sure how to add it to a Spring boot app.

[wilkinsona]

@rdramana85 You'd need to use an UndertowBuilderCustomzer and Undertow's standard API for handler configuration to add it to a Spring Boot app.

[yochentt]

@wilkinsona Is it possible to shutdown Tomcat gracefully in the case Spring Application run failure?

[wilkinsona]

I don't think that's necessary. If the app fails to start there will be no in-flight requests as the containers connectors aren't started till the app is running.

[nithril]

Is this still planned for 2.0.0 as 2.0.0.M1 is out?

The undertow graceful shutdown returns 503. Tomcat Endpoint#Pause stops to accept and process connection but the socket is not closed and the connections are stalled waiting to be accepted. Thus if the application shutdown takes some time, these connections/requests are stalled until tomcat is stopped and then refused. Actually Endpoint#Pause does not seems viable.

Returning 503 seems a common approach handled by most if not all http proxy.

[wilkinsona]

Is this still planned for 2.0.0 as 2.0.0.M1 is out?

Yes. The issue is still assigned to the 2.0.0 milestone.

Actually Endpoint#Pause does not seems viable.

Why do you think that it isn't viable?

When Tomcat's StandardService is stopped it pauses the connectors, stops the underlying Engine (which will wait for requests to complete), and then stops the connectors. Tomcat and Undertow may take different approaches, but, given Tomcat's vast adoption, I think it's unlikely that its approach is not viable.

[nithril]

Optimal may be more appropriate than viable.

When Tomcat's StandardService is stopped it pauses the connectors

This is the main concern. For what I have understand after testing and reading the tomcat codes when the http connector is paused, new connections/requests are stalled until tomcat is stopped and then refused. It may be an issue if the application shutdown takes some time, even for 5s.

Maybe do you plan to improve the Tomcat behaviour?

[wilkinsona]

Maybe do you plan to improve the Tomcat behaviour?

No, I don't think so. I think we should defer to the Tomcat team's expertise as it's highly likely that they know more about shutting Tomcat down gracefully and how that should behave than we do. If you believe Tomcat can/should be improved, then please raise the matter on the Tomcat Users mailing list.

[nithril]

I agree.

Could you please share how you planned to implement this ticket?

[wilkinsona]

Could you please share how you planned to implement this ticket?

Nothing's been planned beyond what's been sketched out by various people in this issue.

[nithril]

In your opinion is it something that make sense:

• At startup a GracefulShutdownFilter servlet filter is registered (like undertow one) which keeps track of active requests
• The context shutdown is triggered
• The GracefulShutdownFilter filter returns http status 503 for all new requests
• When all accepted requests before the shutdown are all completed, the GracefulShutdownFilter#shutdownComplete boolean is set to true
• As late as possible and only if GracefulShutdownFilter#shutdownComplete is true, tomcat is stopped.

This solution has already been proposed with the following comment :

Firstly, your solution does not stop accepting requests, instead it accepts them and then responds with a 503. We need to pause the container's connectors so that they stop allowing clients to connect at all.

Is it always your opinion taking into account Connector#pause does not immediatly stops accepting new requests (ie. with a connection refused) but it stalls them until the connector is closed?

[stevenschlansker]

For what it's worth, we are using the workaround for Jetty in our application now to get the ability to "stop" the connector accepting new requests, while allowing the existing ones to finish. If this feature is resolved with a filter that 503s back requests, we're not interested and would continue using the horrible workaround instead. Our service routing relies on the ability of our applications to cease accepting connections and having it return 503 instead would be a step backwards for us.

Maybe this isn't viable with Tomcat, but Jetty supports it out of the box, and the Tomcat-centric filter described above sounds like a strict step backwards from Jetty's behavior.

[wilkinsona]

@stevenschlansker Historically, our approach with this sort of thing has been to use whatever each container's native way of handling things has been. For example, access logging uses whatever Jetty, Tomcat, and Undertow provide (with slightly different configuration options and functionality) rather than us rolling our own solution across all three containers. I am almost certain that we'll follow the same approach here.

[wilkinsona]

Is it always your opinion taking into account Connector#pause does not immediatly stops accepting new requests (ie. with a connection refused) but it stalls them until the connector is closed?

@nithril As I touched on in my comment above this one, it's my opinion that the best option is for us to build upon each container's existing support. Right now, for Tomcat, that means that we would pause the connector as this is what standalone Tomcat does.

[nithril]

@wilkinsona Ok I understand, thanks for your feedback.

@stevenschlansker The filter I wrote about was only for Tomcat

[renannprado]

@thefallentree solution is good, however sometimes I get this exception while testing the graceful shutdown with a pending connection:

    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:835) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.resolvePreparedArguments(ConstructorResolver.java:785) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:415) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1173) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1067) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:220) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveNamedBean(DefaultListableBeanFactory.java:1018) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:345) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:340) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.cloud.sleuth.instrument.web.TraceHandlerInterceptor.getErrorController(TraceHandlerInterceptor.java:194) ~[spring-cloud-sleuth-core-1.2.1.RELEASE.jar:1.2.1.RELEASE]
    at org.springframework.cloud.sleuth.instrument.web.TraceHandlerInterceptor.isErrorControllerRelated(TraceHandlerInterceptor.java:83) ~[spring-cloud-sleuth-core-1.2.1.RELEASE.jar:1.2.1.RELEASE]
    at org.springframework.cloud.sleuth.instrument.web.TraceHandlerInterceptor.afterCompletion(TraceHandlerInterceptor.java:132) ~[spring-cloud-sleuth-core-1.2.1.RELEASE.jar:1.2.1.RELEASE]
    at org.springframework.web.servlet.HandlerExecutionChain.triggerAfterCompletion(HandlerExecutionChain.java:170) ~[spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.processDispatchResult(DispatcherServlet.java:1055) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:980) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [javax.servlet-api-3.1.0.jar:3.1.0]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:841) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:206) [websocket-server-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:110) [spring-boot-actuator-1.5.3.RELEASE.jar:1.5.3.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
    at br.combbbbb.aaaaa.backend.filter.AuthenticationGate.doFilter(AuthenticationGate.java:40) [classes/:na]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
    at br.combbbbb.aaaaa.backend.filter.AuthenticationFilter.doFilter(AuthenticationFilter.java:41) [classes/:na]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
    at br.combbbbb.aaaaa.backend.filter.dev.SwaggerQueryStringAuthenticationFilter.doFilter(SwaggerQueryStringAuthenticationFilter.java:38) [classes/:na]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:96) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.springframework.cloud.sleuth.instrument.web.TraceFilter.doFilter(TraceFilter.java:153) [spring-cloud-sleuth-core-1.2.1.RELEASE.jar:1.2.1.RELEASE]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
    at br.combbbbb.aaaaa.backend.service.RequestLogging.doFilterInternal(RequestLogging.java:44) [classes/:na]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) [spring-web-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
    at io.prometheus.client.filter.MetricsFilter.doFilter(MetricsFilter.java:172) [simpleclient_servlet-0.0.22.jar:na]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
    at br.combbbbb.aaaaa.backend.filter.HttpStatusMetricFilter.doFilter(HttpStatusMetricFilter.java:41) [classes/:na]
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1621) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:541) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:481) [jetty-servlet-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.server.Server.handle(Server.java:564) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:317) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-server-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) [jetty-io-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) [jetty-io-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) [jetty-io-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:128) [jetty-util-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.util.thread.Invocable$InvocableExecutor.invoke(Invocable.java:222) [jetty-util-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:294) [jetty-util-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126) [jetty-util-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672) ~[jetty-util-9.4.4.v20170414.jar:9.4.4.v20170414]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590) ~[jetty-util-9.4.4.v20170414.jar:9.4.4.v20170414]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.cache.config.internalCacheAdvisor' defined in class path resource [org/springframework/cache/annotation/ProxyCachingConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cache.interceptor.BeanFactoryCacheOperationSourceAdvisor]: Factory method 'cacheAdvisor' threw exception; nested exception is java.lang.NullPointerException
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1173) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1067) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.aop.framework.autoproxy.BeanFactoryAdvisorRetrievalHelper.findAdvisorBeans(BeanFactoryAdvisorRetrievalHelper.java:92) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.aop.framework.autoproxy.AbstractAdvisorAutoProxyCreator.findCandidateAdvisors(AbstractAdvisorAutoProxyCreator.java:102) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.aop.aspectj.annotation.AnnotationAwareAspectJAutoProxyCreator.findCandidateAdvisors(AnnotationAwareAspectJAutoProxyCreator.java:88) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.aop.aspectj.autoproxy.AspectJAwareAdvisorAutoProxyCreator.shouldSkip(AspectJAwareAdvisorAutoProxyCreator.java:103) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.aop.framework.autoproxy.AbstractAutoProxyCreator.wrapIfNecessary(AbstractAutoProxyCreator.java:340) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.aop.framework.autoproxy.AbstractAutoProxyCreator.postProcessAfterInitialization(AbstractAutoProxyCreator.java:298) ~[spring-aop-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsAfterInitialization(AbstractAutowireCapableBeanFactory.java:423) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1633) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    ... 100 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cache.interceptor.BeanFactoryCacheOperationSourceAdvisor]: Factory method 'cacheAdvisor' threw exception; nested exception is java.lang.NullPointerException
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    ... 117 common frames omitted
Caused by: java.lang.NullPointerException: null
    at org.springframework.cache.annotation.ProxyCachingConfiguration.cacheAdvisor(ProxyCachingConfiguration.java:47) ~[spring-context-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.cache.annotation.ProxyCachingConfiguration$$EnhancerBySpringCGLIB$$300d9692.CGLIB$cacheAdvisor$0(<generated>) ~[spring-context-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.cache.annotation.ProxyCachingConfiguration$$EnhancerBySpringCGLIB$$300d9692$$FastClassBySpringCGLIB$$4af7b287.invoke(<generated>) ~[spring-context-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) ~[spring-core-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:358) ~[spring-context-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at org.springframework.cache.annotation.ProxyCachingConfiguration$$EnhancerBySpringCGLIB$$300d9692.cacheAdvisor(<generated>) ~[spring-context-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
    at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162) ~[spring-beans-4.3.8.RELEASE.jar:4.3.8.RELEASE]
    ... 118 common frames omitted

2017-06-13 23:38:17.821  INFO [aaaaa_backend,,,] 6267 --- [tp1364765175-20] b.c.u.z.backend.service.RequestLogging   : After request [method=GET;uri=/api/v1/sleep?UOLSER=1;status=500;client=127.0.0.1;took=5087]
2017-06-13 23:38:17.826  INFO [aaaaa_backend,,,] 6267 --- [      Thread-22] o.e.jetty.server.AbstractConnector       : Stopped ServerConnector@2e446168{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2017-06-13 23:38:17.826  INFO [aaaaa_backend,,,] 6267 --- [      Thread-22] org.eclipse.jetty.server.session         : Stopped scavenging
Disconnected from the target VM, address: '127.0.0.1:41102', transport: 'socket'
2017-06-13 23:38:17.830  INFO [aaaaa_backend,,,] 6267 --- [      Thread-22] o.e.j.s.h.ContextHandler.application     : Destroying Spring FrameworkServlet 'dispatcherServlet'
2017-06-13 23:38:17.831  INFO [aaaaa_backend,,,] 6267 --- [      Thread-22] o.e.jetty.server.handler.ContextHandler  : Stopped o.s.b.c.e.j.JettyEmbeddedWebAppContext@1dd17c0c{/,[file:///tmp/jetty-docbase.1191171853553345825.8080/, jar:file:/home/joaop/.m2/repository/io/springfox/springfox-swagger-ui/2.6.1/springfox-swagger-ui-2.6.1.jar!/META-INF/resources],UNAVAILABLE}```

[jorgheymans]

if this ever gets implemented, please consider the case for a delayed graceful shutdown with a hook linking back to the application. My usecase: i want to shutdown one node of a clustered spring-boot app but first signal to the load balancer in front of it that this particular node should no longer be used. So i would need to delay any shutdown actions until i am sure that the LB will no longer route requests to this node.

[penglongli]

Uh..i have some ideas. If we want shutdown the server gracefully, we should do the following:

• Return the end signal, but the server hasn't stop actually. If this can do, the clustered will think this node is CLOSED;
• Reject all requests;
• Wait active requests all completed;
• Set the maximum Expiration-Time for the server to be shutdown.

These are my understanding, may not be comprehensive. Hope these help!

[mlausch1963]

@jorgheymans Seconded. It gets even more complicated. For example prometheus.io (monitoring) scrapes an HTTP endpoint in the application to get monitoring data. Shutting down the connector, will also prevent the monitoring system to get the metrics and statistics. In Openshift, a service running in a container signals the infrastructure (like loadbalancer) its readyness to accept requests by returning a 200 OK response to a configurable URL. Returning something else like 500 makes the infrastructure forward requests to this instance. In this scenario it makes sense to shut down the readyness endpoint or make it return != 200, but let everything else running until the input queue is empty or a max grace period expires.

I understand that this implementation will not be standard spring boot behaviour, but there should be hooks in the shutdown machinery to allow something like this to be implemented

[hbrands]

I would really like to see a solution in version 2.0, because it's an important production feature. Please consider rescheduling to a 2.0 milestone release to get feedback from users in time.

[manuelwallrapp]

We use Openshift to manage our containers and wanted a zero downtime deployment. So we implemented a Springboot Actuator Healthcheck. To Listen to the Sigterm Signal we implemented our own Springboot Starter. Once the JVM sends the Shutdownevent, the Shutdownlistener set the Healthcheck to false and some configured seconds later initiate the Springcontext Shutdown. We testet this graceful shutdown behavior hook with Openshift. Details you find here. The Jar is published in the public sonatype maven repo.

https://github.com/SchweizerischeBundesbahnen/springboot-graceful-shutdown

[ghost]

I had the same issues with OpenShift. Thank you all for sharing your code examples and ideas. I prefer the approach of @nithril to return HTTP Code 503 during shutdown process. It's much faster than waiting for the OpenShift health check to become unhealthy. In my oppinion OpenShift should add an option to remove the pod from service before sending the SIGTERM. But currently shutting down the HTTP Connector is the recommended solution: https://docs.openshift.com/container-platform/3.6/dev_guide/deployments/advanced_deployment_strategies.html#graceful-Termination

Considering this I created an autoconfigured SpringBoot-Application module with:

1. Application-Context initializer and ShutdownHandler
2. HTTPFilter to count the requests and return 503 when shutting down.

Please find the sources here: https://github.com/ramato-procon/springboot-graceful-terminator

It's not yet production ready, but I would appreciate your review/comment.

[robocide]

@ramato-procon i sugguested the same approach :) (under avishay username) is it the idea same as you implemented ?

[ghost]

@robocide Yes it is the same idea. The filter counts each running request (but with an AtomicFieldUpdater, which is better in performance than synchronize). It returns 503, when the shutdown process has been started: https://github.com/ramato-procon/springboot-graceful-terminator/blob/master/src/main/java/de/ra/springboot/graceful/GracefulHttpFilter.java

The shutdown process is started by the GracefulShutdownHook, which first inits the shutdown process in the Filter and waits until all running requests have been processed. In the next step the http connector will be stopped. Afterwards the ApplicationContext will be closed: https://github.com/ramato-procon/springboot-graceful-terminator/blob/master/src/main/java/de/ra/springboot/graceful/GracefulShutdownHook.java

[rabbitgyk]

This is a troublesome thing, waiting for shutdown.

[wilkinsona]

Having discussed this with a couple of people at Spring IO last week, it occurred to me that this may be better handled as a Spring Framework feature or at least building upon a Spring Framework feature. There's nothing Boot-specific about allowing Spring-managed components to be shut down gracefully and the concept being part of the Framework would allow other projects in the portfolio to provide graceful shutdown integration without creating a circular dependency between those projects and Boot. @snicoll is going to raise it with the Framework team.