Murmur does not shut down when SIGTERM

[Silmerias]

murmur-static_x86-1.3.0 Debian stretch 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u1 (2019-09-20) x86_64 GNU/Linux

When sent SIGTERM by systemd we see murmur closing, but the process will never shut down before systemd sigterm timeout kill it with SIGKILL.

2019-10-01 00:05:06.968 Caught SIGTERM, exiting 2019-10-01 00:05:06.968 Killing running servers

During the process the CPU is 100% overheated and strace show murmur trying to poll at 24000 messages/seconds.

poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}]) poll([{fd=12, events=POLLIN}], 1, -1) = 1 ([{fd=12, revents=POLLIN}])

[davidebeatrici]

Could you attach GDB to Murmur and check the backtrace after SIGTERM is sent, please?

[Silmerias]

Attaching to process 16043 [New LWP 16044] [New LWP 16045] [New LWP 16046] [New LWP 16047] 0xf778fc89 in __kernel_vsyscall () (gdb) backtrace

0 0xf778fc89 in __kernel_vsyscall ()

1 0x00a7b693 in ?? ()

2 0xf66e4d09 in ?? ()

3 0xf66e4f00 in ?? ()

4 0xf66e500d in ?? ()

5 0xf66e532b in ?? ()

6 0xf66e563b in ?? ()

7 0xf66eb331 in ?? ()

8 0xf66eb5da in ?? ()

9 0xf66e9c54 in ?? ()

10 0xf66e9d75 in ?? ()

11 0xf6718a5c in ?? ()

12 0xf6fa2042 in ?? ()

13 0x080b010a in ?? ()

14 0x0808375c in ?? ()

15 0x0808dfb1 in ?? ()

16 0x0810bc39 in ?? ()

17 0x0807c314 in ?? ()

18 0x009c5edc in ?? ()

19 0x0805f1a1 in ?? ()

[mkrsn]

Same problem here. Any News on that?

version: 1.3.4.x86_static Linux 5.10.23-gentoo #1 SMP Mon Mar 15 12:07:21 CET 2021 x86_64 GNU/Linux

After a SIGTERM:

...
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
poll([{fd=12, events=POLLIN}], 1, -1)   = 1 ([{fd=12, revents=POLLIN}])
...

fd12 is the file descriptor to the MariaDB Socket.

backtrace

Attaching to process 9207
[New LWP 9209]
[New LWP 9210]
[New LWP 9211]
[New LWP 9212]
[New LWP 9220]
0xf7f68549 in __kernel_vsyscall ()
(gdb) backtrace
#0  0xf7f68549 in __kernel_vsyscall ()
#1  0x00a7b693 in ?? ()
#2  0xf6ebbd09 in ?? ()
#3  0xf6ebbf00 in ?? ()
#4  0xf6ebc00d in ?? ()
#5  0xf6ebc32b in ?? ()
#6  0xf6ebc63b in ?? ()
#7  0xf6ec2331 in ?? ()
#8  0xf6ec25da in ?? ()
#9  0xf6ec0c54 in ?? ()
#10 0xf6ec0d75 in ?? ()
#11 0xf6eefa5c in ?? ()
#12 0xf7779042 in ?? ()
#13 0x080d231a in ?? ()
#14 0x08083a8c in ?? ()
#15 0x08173dcb in ?? ()
#16 0x08173e26 in ?? ()
#17 0x081bcb8c in ?? ()
#18 0xf7398c69 in ?? ()
#19 0xf73998aa in ?? ()
#20 0x081b92a3 in ?? ()
#21 0x0810f08d in ?? ()
#22 0x0807c490 in ?? ()
#23 0x009c5edc in ?? ()
#24 0x0805f201 in ?? ()

[Krzmbrzl]

Afaik nobody was able to track the issue down yet. In fact I am not even sure somebody found the time to really try yet :thinking: