KafkaBinderMetrics prevent JVM shutdown due to not shutting down scheduler thread pool!

[tschaper69]

We have built a Spring Cloud Stream Kafka application using Spring Cloud Stream 3.0.13.RELEASE that allows to be shutdown by invoking the actuator 'shutdown' endpoint. This does not work due to a thread pool maintained by class org.springframework.cloud.stream.binder.kafka.KafkaBinderMetrics. The threads in this pool are regular (non-daemon) threads. On invocation of the 'shutdown' endpoint the thread pool is not shutdown. So the threads in it keep on running preventing the JVM process from terminating. I made a thread dump after invoking the 'shutdown' endpoint. The 'evil' threads are those with name **pool-4-thread-***: thread_dump.txt By debugging the application I found out that they belong to the ScheduledExecutorService instance held in member variable 'scheduler' of the KafkaBinderMetrics class.

[tschaper69]

In order to make it work I excluded the io.micrometer:micrometer-core dependency. But this is a high price to pay for it means that the application provides no metrics at all.

[sobychacko]

@tschaper69 Thanks for this report. The binder version you are using (3.0.13.RELESE) is an older release (released in 2019) and is already out of OSS support and soon to be out of any commercial support also. See this support timeline for details. I suggest you upgrade to the latest GA if possible (3.2.2). I have a feeling that this problem you are reporting may be present in the latest release as well after looking at the code, but if you can confirm that after upgrading, that would be great.

[garyrussell]

Yes; there was a change in Micrometer (1.7, IIRC) that allows us to unregister the meters.

[tschaper69]

@sobychacko : I'm sorry but we are currently not in the position to upgrade. The behaviour is easy to reproduce so you should be able to verify by yourself if it is still existent in the latest version.

[garyrussell]

Unfortunately, the older micrometer (pre 1.6.11) used by that version provides no mechanism to unregister the metrics.

https://github.com/micrometer-metrics/micrometer/commit/603264632e3be5336902f65d42abbf2ed6769d45

There is nothing that this project can do about it.

You might be able to override the micrometer version to a newer one, but, as @sobychacko said, that version of SCSt is no longer supported.

The issue there https://github.com/micrometer-metrics/micrometer/issues/2018 says it was resolved in 1.7.0, but it looks like they back-ported it to 1.6.11.

[tschaper69]

@garyrussell : So upgrading to a newer Kafka Binder (and the depending Micrometer) version will definitely fix the problem?

[garyrussell]

Yes.

[tschaper69]

@garyrussell : Thank you for providing the providing the information.