Closed jorgheymans closed 5 years ago
cc @openzipkin/armeria in case y'all have insight into this. This started after the switcheroo and maybe there's an easy explanation/setting somewhere
Confirming with java -jar zipkin.jar --logging.level.root=TRACE
that the armeriaServer
bean indeed is taking a long time to destroy:
2019-03-19 09:14:25.531 TRACE 28697 --- [ Thread-5] o.s.b.f.s.DisposableBeanAdapter : Invoking destroy method 'close' on bean with name 'cassandraCluster' 2019-03-19 09:14:25.555 TRACE 28697 --- [ Thread-5] o.s.b.f.s.DisposableBeanAdapter : Invoking destroy method 'close' on bean with name 'armeriaServer' 2019-03-19 09:14:32.784 TRACE 28697 --- [ Thread-5] o.s.b.f.s.DisposableBeanAdapter : Invoking destroy() on bean with name 'defaultValidator' 2019-03-19 09:14:32.784 TRACE 28697 --- [ Thread-5] o.s.b.f.s.DefaultListableBeanFactory : Retrieved dependent beans for bean 'prometheusMeterRegistry': [metrics, zipkin2.server.internal.MetricsHealthController, metricsEndpoint, zipkin2.autoconfigure.prometheus.ZipkinPrometheusMetricsAutoConfiguration, metricsRestTemplateCustomizer] 2019-03-19 09:14:32.784 TRACE 28697 --- [ Thread-5] o.s.b.f.s.DefaultListableBeanFactory : Retrieved dependent beans for bean 'metrics': [zipkin2.server.internal.ZipkinHttpCollector]
Is graceful shutdown enabled? ~(It's disabled if not configured.)~
It looks like graceful shutdown might be enabled by default
I think this is because graceful shutdowns is a good default for normal servers, though since tracing is often best effort maybe should be changed in zipkin server
Zipkin does not appear to configure graceful shutdown for the ArmeriaServer, so it would rely on the defaults.
@anuraaga you mean gracefulShutdownQuietPeriodMillis
rather ?https://github.com/line/armeria/blob/4412f9984047509423cdad7f6ae78a499a3f7d5e/spring/boot-autoconfigure/src/main/java/com/linecorp/armeria/spring/ArmeriaSettings.java#L181
The quiet period is how long the server will wait for requests to go away. The timeout is the maximum amount of time it will wait - with a load balancer configured for the health point, the quiet period will generally be enough for shutdown but without that the server will always wait until the timeout of 40s.
To disable graceful shutdown on zipkin server, we should probably disable both settings - though @adriancole should make the call as enabling graceful by default can also reduce gotchas in default configurations.
So before we switched to armeria, we didn't have such a long quiet period. So in this case, I think we should disable the settings by default as folks wouldn't have relied on them formerly and the first time experience isn't great with them on either.
On Thu, Mar 21, 2019 at 10:24 PM Anuraag Agrawal notifications@github.com wrote:
The quiet period is how long the server will wait for requests to go away. The timeout is the maximum amount of time it will wait - with a load balancer configured for the health point, the quiet period will generally be enough for shutdown but without that the server will always wait until the timeout of 40s.
To disable graceful shutdown on zipkin server, we should probably disable both settings - though @adriancole https://github.com/adriancole should make the call as enabling graceful by default can also reduce gotchas in default configurations.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openzipkin/zipkin/issues/2454#issuecomment-475249888, or mute the thread https://github.com/notifications/unsubscribe-auth/AAD616t86Z4SL7lIxeUVXlOMXwqFN_T9ks5vY5YogaJpZM4b7dNG .
java -jar zipkin.jar
takes very long to shutdown (max i have seen is half a minute). Seeing this with the latest version an in-memory storage. A cold unused instance takes about 5 seconds to stop upon CTRL-C. This is on linux using jdk8.