Support graceful shutdown of connection pool (@RefreshScope spring boot)

[amarkanth1984]

We are in the process of moving our application from Spring MVC to Spring Boot inorder to leverage the Spring consul features (one of the main ones being dynamic config reload without the need to restart the application). The suggested way is to use "@RefreshScope" annotation on "dataSource" bean (i.e. http://cloud.spring.io/spring-cloud-static/docs/1.0.x/spring-cloud.html#_refresh_scope) and when i try it using Spring Boot 1.4.1 and HikariCp-2.4.7, I see the following which is good but NOT the behaviour i expected since i had a transaction in progress when the pool got closed. Is there any plan to support graceful shutdown? (it could be as simple as waiting for the current active connections to go idle or wait for some specific amount of time ... lets say 30 secs or configurable)

Looking at shutdown() method, looks like you force close all connections and shutdown the pool when bean destroy method (i.e. close() method is called).

/* * Shutdown the pool, closing all idle connections and aborting or closing * active connections. / public final synchronized void shutdown() throws InterruptedException

2016-10-06 20:34:41.810 INFO 28596 --- [ool-65-thread-1] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@260b673b: startup date [Thu Oct 06 20:34:41 PDT 2016]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@29570865 2016-10-06 20:34:41.826 INFO 28596 --- [ool-65-thread-1] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring 2016-10-06 20:34:41.871 INFO 28596 --- [ool-65-thread-1] o.s.boot.SpringApplication : Started application in 2.837 seconds (JVM running for 245.173) 2016-10-06 20:34:41.871 INFO 28596 --- [ool-65-thread-1] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@260b673b: startup date [Thu Oct 06 20:34:41 PDT 2016]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@29570865 2016-10-06 20:34:41.873 INFO 28596 --- [ool-65-thread-1] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@29570865: startup date [Thu Oct 06 20:34:39 PDT 2016]; root of context hierarchy 2016-10-06 20:34:50.071 INFO 28596 --- [ool-65-thread-1] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Close initiated... 2016-10-06 20:34:50.086 INFO 28596 --- [ool-65-thread-1] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Closed. 2016-10-06 20:34:50.088 INFO 28596 --- [ool-65-thread-1] o.s.c.e.event.RefreshEventListener : Refresh keys changed: [jdbcConnectString] 2016-10-06 20:34:52.171 INFO 28596 --- [ ServiceMonitor] com.zaxxer.hikari.HikariDataSource : HikariPool-2 - Started.

2016-10-06 20:34:52.190 INFO 28596 --- [ ServiceMonitor] com.zaxxer.hikari.pool.PoolBase : HikariPool-2 - Driver does not support get/set network timeout for connections. (Method org.postgresql.jdbc.PgConnection.getNetworkTimeout() is not yet implemented.)

Thanks, Amar

[nitincchauhan]

As mentioned in @RefreshScope, For reloading of important properties of configuration (user name, password, url etc), you need to shutdown existing and create NEW pool.

[amarkanth1984]

I don't understand your comment Nitin. I am asking about graceful shutdown of connection pool rather than force shutdown done in close() method of HikariPool class. Force shutdown will result in transaction failures which we don't want.

[amarkanth1984]

Looking at https://github.com/brettwooldridge/HikariCP/issues/228, I could set the data source destroy method to softEvictConnections. Will play around with it and update here if it works.

[amarkanth1984]

@brettwooldridge

[jamisonhyatt]

Did you find a way to use softEvictConnections?

My problem is that I want access to the softEvictConnections without JMX, since this will be part of a circuit breaker.

I feel like there's a couple ways to solve this. Let us construct a HikariDataSource from a HikariPool, or adding a method to HikariDataSource which calls HikariPool's softEvictConnections method. This would at least allow us to write our own graceful shutdown methods.

[martinufo]

:+1: for the public softEvictConnections method. @jamisonhyatt So far we've been using it like this:

HikariDataSource dataSource = ...
((HikariPoolMXBean) PropertyAccessorFactory.forDirectFieldAccess(dataSource)
    .getPropertyValue("pool")).softEvictConnections();

PropertyAccessorFactory is from Spring, but you should be able to use standard java reflection (I've not tested this):

Field pool = HikariDataSource.class.getDeclaredField("pool");
pool.setAccessible(true);
((HikariPoolMXBean) pool.get(dataSource)).softEvictConnections();

[brettwooldridge]

We'll look into adding a getHikariPoolMXBean() method to HikariDataSource.

[jamisonhyatt]

@martinufo That worked, thanks.

@brettwooldridge that would be handy. For now, I have tied pool startup with a check to ensure I can reflect on HikariDataSource properly at application startup.

[brettwooldridge]

@jamisonhyatt @martinufo With the new code, a "graceful" shutdown would go something like this:

HikariPoolMXBean poolBean = dataSource.getHikariPoolMXBean();
while (poolBean.getActiveConnections() > 0 && /* some timeout is not exceeded */ ) {
  poolBean.softEvictConnections();
  // sleep a little
}
dataSource.close();

[devdef007]

Are there any application properties to achieve the same ?