search

FirebirdSQL / firebird

Firebird server, client and tools
https://www.firebirdsql.org/
1.24k stars 212 forks source link

Firebird leaves TCP connections #7674

Open tomaszdubiel18 opened 1 year ago

tomaszdubiel18 commented 1 year ago

Hello. I've been told to create an issue here. Firebird 3.0.10. The problem is present on Windows Server 2022, Firebird 3.0.10 SuperServer and on Ubuntu 22.04.2 Firebird 3.0.10 ClassicServer. On Ubuntu the problem is serious. Many connections remain in ESTABLISHED state, even though there are no entries from that IP in MON$ATTACHMENTS, and every two days RAM is finished. On Windows it's not that serious, but the problem is present as well. tcp windows

AlexPeshkoff commented 1 year ago

Re UBUNTU. Please provide stack traces (all threads) for any one process which keeps undesired connection in ESTABLISHED state.

tomaszdubiel18 commented 1 year ago

When I ran: root@stream:/opt/firebird# strace -s 99 -ffp 639871 I got this. Is this what you want? Sorry for the delay, but problem temporarily stopped occuring, but now came back and it is again very problematic: strace: Process 639871 attached with 7 threads [pid 639937] futex(0x7fa216f699d8, FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...> [pid 639913] futex(0x7fa216f7cd8c, FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...> [pid 639876] futex(0x7fa216101130, FUTEX_WAIT_BITSET|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...> [pid 639873] futex(0x7fa21d166350, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY <unfinished ...> [pid 639872] restart_syscall(<... resuming interrupted read ...> <unfinished ...> [pid 639874] restart_syscall(<... resuming interrupted read ...> <unfinished ...> [pid 639871] restart_syscall(<... resuming interrupted read ...> <unfinished ...> [pid 639872] <... restart_syscall resumed>) = -1 ETIMEDOUT (Connection timed out) [pid 639872] futex(0x7fa21ce46d40, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1694525515, tv_nsec=641920000}, FUTEX_BITSET_MATCH_ANY <unfinished ...> [pid 639871] <... restart_syscall resumed>) = 0 [pid 639871] poll([{fd=1, events=POLLIN}], 1, 60000

tomaszdubiel18 commented 1 year ago

When I ran gdb command, I got this: Please take a look: Thread 7 (Thread 0x7fa1d4be7640 (LWP 639937) "firebird"):

0 __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7fa216f699d8) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, clockid=0, expected=0, futex_word=0x7fa216f699d8) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fa216f699d8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abs time@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139

3 0x00007fa21d23aac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fa216f69988, cond=0x7fa216f699b0) at ./nptl/pthread_cond_wait.c:5 03

4 ___pthread_cond_wait (cond=0x7fa216f699b0, mutex=0x7fa216f69988) at ./nptl/pthread_cond_wait.c:627

5 0x00007fa216a873c0 in ?? () from /opt/firebird/plugins/libEngine12.so

6 0x00007fa21676d11b in ?? () from /opt/firebird/plugins/libEngine12.so

7 0x00007fa21676ee13 in ?? () from /opt/firebird/plugins/libEngine12.so

8 0x00007fa216a6f6f5 in ?? () from /opt/firebird/plugins/libEngine12.so

9 0x00007fa21d23bb43 in start_thread (arg=) at ./nptl/pthread_create.c:442

10 0x00007fa21d2cda00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x7fa2097ff640 (LWP 639913) "firebird"):

0 __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7fa216f7cd8c) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, clockid=0, expected=0, futex_word=0x7fa216f7cd8c) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fa216f7cd8c, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abs time@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139

3 0x00007fa21d23aac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fa216f7cd38, cond=0x7fa216f7cd60) at ./nptl/pthread_cond_wait.c:5 03

4 ___pthread_cond_wait (cond=0x7fa216f7cd60, mutex=0x7fa216f7cd38) at ./nptl/pthread_cond_wait.c:627

5 0x00007fa216a873c0 in ?? () from /opt/firebird/plugins/libEngine12.so

6 0x00007fa2169fcc28 in ?? () from /opt/firebird/plugins/libEngine12.so

7 0x00007fa2169fdf83 in ?? () from /opt/firebird/plugins/libEngine12.so

8 0x00007fa216a6f6f5 in ?? () from /opt/firebird/plugins/libEngine12.so

9 0x00007fa21d23bb43 in start_thread (arg=) at ./nptl/pthread_create.c:442

10 0x00007fa21d2cda00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7fa210bff640 (LWP 639876) "firebird"):

0 __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7fa216101130) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, clockid=0, expected=0, futex_word=0x7fa216101130) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fa216101130, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abs time@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139

3 0x00007fa21d23aac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fa2161010e0, cond=0x7fa216101108) at ./nptl/pthread_cond_wait.c:5 03

4 ___pthread_cond_wait (cond=0x7fa216101108, mutex=0x7fa2161010e0) at ./nptl/pthread_cond_wait.c:627

5 0x00007fa216a873c0 in ?? () from /opt/firebird/plugins/libEngine12.so

6 0x00007fa2166e2be9 in ?? () from /opt/firebird/plugins/libEngine12.so

--Type for more, q to quit, c to continue without paging--c

7 0x00007fa2166df733 in ?? () from /opt/firebird/plugins/libEngine12.so

8 0x00007fa216a6f6f5 in ?? () from /opt/firebird/plugins/libEngine12.so

9 0x00007fa21d23bb43 in start_thread (arg=) at ./nptl/pthread_create.c:442

10 0x00007fa21d2cda00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7fa2177fe640 (LWP 639874) "firebird"):

0 __futex_abstimed_wait_common64 (private=, cancel=true, abstime=0x7fa2177fdd10, op=393, expected=0, futex_word=0x7fa21d1622f0) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=, abstime=0x7fa2177fdd10, clockid=0, expected=0, futex_word=0x7fa21d1622f0) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fa21d1622f0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7fa2177fdd10, private=) at ./nptl/futex-internal.c:139

3 0x00007fa21d243a30 in do_futex_wait (sem=sem@entry=0x7fa21d1622f0, abstime=abstime@entry=0x7fa2177fdd10, clockid=0) at ./nptl/sem_waitcommon.c:111

4 0x00007fa21d243ad3 in __new_sem_wait_slow64 (sem=0x7fa21d1622f0, abstime=0x7fa2177fdd10, clockid=0) at ./nptl/sem_waitcommon.c:183

5 0x00007fa21d734340 in ?? () from /opt/firebird/bin/../lib/libfbclient.so.2

6 0x00007fa21d6437d5 in ?? () from /opt/firebird/bin/../lib/libfbclient.so.2

7 0x00007fa21d6ff9f5 in ?? () from /opt/firebird/bin/../lib/libfbclient.so.2

8 0x00007fa21d23bb43 in start_thread (arg=) at ./nptl/pthread_create.c:442

9 0x00007fa21d2cda00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7fa217fff640 (LWP 639873) "firebird"):

0 __futex_abstimed_wait_common64 (private=, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7fa21d166350) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=, abstime=0x0, clockid=0, expected=0, futex_word=0x7fa21d166350) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fa21d166350, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=) at ./nptl/futex-internal.c:139

3 0x00007fa21d243c5f in do_futex_wait (sem=sem@entry=0x7fa21d166350, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111

4 0x00007fa21d243cf8 in __new_sem_wait_slow64 (sem=0x7fa21d166350, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183

5 0x00007fa21d734252 in ?? () from /opt/firebird/bin/../lib/libfbclient.so.2

6 0x00007fa21d68c715 in ?? () from /opt/firebird/bin/../lib/libfbclient.so.2

7 0x00007fa21d6ff9f5 in ?? () from /opt/firebird/bin/../lib/libfbclient.so.2

8 0x00007fa21d23bb43 in start_thread (arg=) at ./nptl/pthread_create.c:442

9 0x00007fa21d2cda00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7fa21ce47640 (LWP 639872) "firebird"):

0 __futex_abstimed_wait_common64 (private=, cancel=true, abstime=0x7fa21ce46c30, op=393, expected=0, futex_word=0x7fa21ce46d40) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=, abstime=0x7fa21ce46c30, clockid=0, expected=0, futex_word=0x7fa21ce46d40) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fa21ce46d40, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7fa21ce46c30, private=) at ./nptl/futex-internal.c:139

3 0x00007fa21d243a30 in do_futex_wait (sem=sem@entry=0x7fa21ce46d40, abstime=abstime@entry=0x7fa21ce46c30, clockid=0) at ./nptl/sem_waitcommon.c:111

4 0x00007fa21d243ad3 in __new_sem_wait_slow64 (sem=0x7fa21ce46d40, abstime=0x7fa21ce46c30, clockid=0) at ./nptl/sem_waitcommon.c:183

5 0x00000000004827e0 in ?? ()

6 0x000000000042f68e in ?? ()

7 0x00000000004442ae in ?? ()

8 0x0000000000458e85 in threadStart ()

9 0x00007fa21d23bb43 in start_thread (arg=) at ./nptl/pthread_create.c:442

10 0x00007fa21d2cda00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7fa21d173c00 (LWP 639871) "firebird"):

0 0x00007fa21d2bfd7f in __GI___poll (fds=0x7fa21d1419b8, nfds=1, timeout=60000) at ../sysdeps/unix/sysv/linux/poll.c:29

1 0x000000000041abf1 in ?? ()

2 0x0000000000417134 in ?? ()

3 0x0000000000436823 in ?? ()

4 0x0000000000446a2a in ?? ()

5 0x00007fa21d1d0d90 in __libc_start_call_main (main=main@entry=0x446750, argc=argc@entry=1, argv=argv@entry=0x7ffcfa1f9a58) at ../sysdeps/nptl/libc_start_call_main.h:58

6 0x00007fa21d1d0e40 in __libc_start_main_impl (main=0x446750, argc=1, argv=0x7ffcfa1f9a58, init=, fini=, rtld_fini=, stack_end=0x7ffcfa1f9a48) at ../csu/libc-start.c:392

7 0x0000000000412599 in ?? ()

8 0x00007ffcfa1f9a48 in ?? ()

9 0x000000000000001c in ?? ()

10 0x0000000000000001 in ?? ()

11 0x00007ffcfa1f9ed2 in ?? ()

12 0x0000000000000000 in ?? ()

AlexPeshkoff commented 1 year ago

Sorry, forgot to mention that you should install debugInfo for your FB version.

tomaszdubiel18 commented 1 year ago

I hope I did everything ok. Is this useful information for you? Please take a look. We are looking for a solution to this problem.

Thread 7 (Thread 0x7fa1d3777640 (LWP 782759) "firebird"):

0 __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7fa216f69728) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, clockid=0, expected=0, futex_word=0x7fa216f69728) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fa216f69728, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139

3 0x00007fa21d23aac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fa216f696d8, cond=0x7fa216f69700) at ./nptl/pthread_cond_wait.c:503

4 ___pthread_cond_wait (cond=0x7fa216f69700, mutex=0x7fa216f696d8) at ./nptl/pthread_cond_wait.c:627

5 0x00007fa216a873c0 in Firebird::SharedMemoryBase::eventWait (this=, event=0x7fa216f696d0, value=2, micro_seconds=) at /usr/home/fireb ird/3.0.10/src/common/isc_sync.cpp:852

6 0x00007fa21676d11b in Jrd::EventManager::watcher_thread (this=0x7fa21c3ee4b0) at /usr/home/firebird/3.0.10/src/jrd/event.cpp:1409

7 0x00007fa21676ee13 in ThreadFinishSync<Jrd::EventManager*>::internalRun (this=this@entry=0x7fa21c3ee550) at /usr/home/firebird/3.0.10/src/jrd/../common/ThreadStart .h:189

8 ThreadFinishSync<Jrd::EventManager*>::internalRun (arg=arg@entry=0x7fa21c3ee550) at /usr/home/firebird/3.0.10/src/jrd/../common/ThreadStart.h:181

9 0x00007fa216a6f6f5 in (anonymous namespace)::ThreadArgs::run (this=) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:78

10 (anonymous namespace)::threadStart (arg=0x7fa21c3edb30) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:94

11 0x00007fa21d23bb43 in start_thread (arg=) at ./nptl/pthread_create.c:442

12 0x00007fa21d2cda00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x7fa2097ff640 (LWP 782758) "firebird"):

0 __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7fa216f7cdcc) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, clockid=0, expected=0, futex_word=0x7fa216f7cdcc) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fa216f7cdcc, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139

3 0x00007fa21d23aac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fa216f7cd78, cond=0x7fa216f7cda0) at ./nptl/pthread_cond_wait.c:503

4 ___pthread_cond_wait (cond=0x7fa216f7cda0, mutex=0x7fa216f7cd78) at ./nptl/pthread_cond_wait.c:627

5 0x00007fa216a873c0 in Firebird::SharedMemoryBase::eventWait (this=, event=0x7fa216f7cd70, value=2, micro_seconds=) at /usr/home/fireb ird/3.0.10/src/common/isc_sync.cpp:852

6 0x00007fa2169fcc28 in Jrd::LockManager::blocking_action_thread (this=0x7fa21c3edb90) at /usr/home/firebird/3.0.10/src/lock/lock.cpp:1563

7 0x00007fa2169fdf83 in ThreadFinishSync<Jrd::LockManager*>::internalRun (this=this@entry=0x7fa21c3edc28) at /usr/home/firebird/3.0.10/src/lock/../lock/../common/Thr eadStart.h:189

8 ThreadFinishSync<Jrd::LockManager*>::internalRun (arg=arg@entry=0x7fa21c3edc28) at /usr/home/firebird/3.0.10/src/lock/../lock/../common/ThreadStart.h:181

9 0x00007fa216a6f6f5 in (anonymous namespace)::ThreadArgs::run (this=) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:78

10 (anonymous namespace)::threadStart (arg=0x7fa21c3ece30) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:94

11 0x00007fa21d23bb43 in start_thread (arg=) at ./nptl/pthread_create.c:442

12 0x00007fa21d2cda00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7fa210bff640 (LWP 782756) "firebird"):

0 __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7fa216101a90) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, clockid=0, expected=0, futex_word=0x7fa216101a90) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fa216101a90, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139

3 0x00007fa21d23aac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7fa216101a40, cond=0x7fa216101a68) at ./nptl/pthread_cond_wait.c:503

4 ___pthread_cond_wait (cond=0x7fa216101a68, mutex=0x7fa216101a40) at ./nptl/pthread_cond_wait.c:627

--Type for more, q to quit, c to continue without paging--c

5 0x00007fa216a873c0 in Firebird::SharedMemoryBase::eventWait (this=, event=0x7fa216101a38, value=1, micro_seconds=) at /usr/home/fireb ird/3.0.10/src/common/isc_sync.cpp:852

6 0x00007fa2166e2be9 in (anonymous namespace)::MappingIpc::clearDeliveryThread (this=0x7fa21c618b20) at /usr/home/firebird/3.0.10/src/jrd/Mapping.cpp:835

7 (anonymous namespace)::MappingIpc::clearDelivery (mapping=0x7fa21c618b20) at /usr/home/firebird/3.0.10/src/jrd/Mapping.cpp:912

8 0x00007fa2166df733 in ThreadFinishSync<(anonymous namespace)::MappingIpc*>::internalRun (this=this@entry=0x7fa21c618b80) at /usr/home/firebird/3.0.10/src/jrd/../co mmon/ThreadStart.h:189

9 ThreadFinishSync<(anonymous namespace)::MappingIpc*>::internalRun (arg=arg@entry=0x7fa21c618b80) at /usr/home/firebird/3.0.10/src/jrd/../common/ThreadStart.h:181

10 0x00007fa216a6f6f5 in (anonymous namespace)::ThreadArgs::run (this=) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:78

11 (anonymous namespace)::threadStart (arg=0x7fa21c3ec640) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:94

12 0x00007fa21d23bb43 in start_thread (arg=) at ./nptl/pthread_create.c:442

13 0x00007fa21d2cda00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7fa2177fe640 (LWP 782754) "firebird"):

0 __futex_abstimed_wait_common64 (private=, cancel=true, abstime=0x7fa2177fdd10, op=393, expected=0, futex_word=0x7fa21d1622f0) at ./nptl/futex-intern al.c:57

1 __futex_abstimed_wait_common (cancel=true, private=, abstime=0x7fa2177fdd10, clockid=0, expected=0, futex_word=0x7fa21d1622f0) at ./nptl/futex-inter nal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fa21d1622f0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f a2177fdd10, private=) at ./nptl/futex-internal.c:139

3 0x00007fa21d243a30 in do_futex_wait (sem=sem@entry=0x7fa21d1622f0, abstime=abstime@entry=0x7fa2177fdd10, clockid=0) at ./nptl/sem_waitcommon.c:111

4 0x00007fa21d243ad3 in __new_sem_wait_slow64 (sem=0x7fa21d1622f0, abstime=0x7fa2177fdd10, clockid=0) at ./nptl/sem_waitcommon.c:183

5 0x00007fa21d734340 in Firebird::SignalSafeSemaphore::tryEnter (this=0x7fa21d1622f0, seconds=, milliseconds=) at /usr/home/firebird/3. 0.10/src/common/classes/semaphore.cpp:202

6 0x00007fa21d6437d5 in Why::(anonymous namespace)::TimerEntry::timeThread () at /usr/home/firebird/3.0.10/src/yvalve/MasterImplementation.cpp:298

7 0x00007fa21d6ff9f5 in (anonymous namespace)::ThreadArgs::run (this=) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:78

8 (anonymous namespace)::threadStart (arg=0x7fa21d168e50) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:94

9 0x00007fa21d23bb43 in start_thread (arg=) at ./nptl/pthread_create.c:442

10 0x00007fa21d2cda00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7fa217fff640 (LWP 782753) "firebird"):

0 __futex_abstimed_wait_common64 (private=, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7fa21d166350) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=, abstime=0x0, clockid=0, expected=0, futex_word=0x7fa21d166350) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fa21d166350, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=) at ./nptl/futex-internal.c:139

3 0x00007fa21d243c5f in do_futex_wait (sem=sem@entry=0x7fa21d166350, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111

4 0x00007fa21d243cf8 in __new_sem_wait_slow64 (sem=0x7fa21d166350, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183

5 0x00007fa21d734252 in Firebird::SignalSafeSemaphore::enter (this=0x7fa21d166350) at /usr/home/firebird/3.0.10/src/common/classes/semaphore.cpp:163

6 0x00007fa21d68c715 in (anonymous namespace)::shutdownThread () at /usr/home/firebird/3.0.10/src/yvalve/why.cpp:754

7 0x00007fa21d6ff9f5 in (anonymous namespace)::ThreadArgs::run (this=) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:78

8 (anonymous namespace)::threadStart (arg=0x7fa21d166390) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:94

9 0x00007fa21d23bb43 in start_thread (arg=) at ./nptl/pthread_create.c:442

10 0x00007fa21d2cda00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7fa21ce47640 (LWP 782752) "firebird"):

0 __futex_abstimed_wait_common64 (private=, cancel=true, abstime=0x7fa21ce46c30, op=393, expected=0, futex_word=0x7fa21ce46d40) at ./nptl/futex-intern al.c:57

1 __futex_abstimed_wait_common (cancel=true, private=, abstime=0x7fa21ce46c30, clockid=0, expected=0, futex_word=0x7fa21ce46d40) at ./nptl/futex-inter nal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7fa21ce46d40, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f a21ce46c30, private=) at ./nptl/futex-internal.c:139

3 0x00007fa21d243a30 in do_futex_wait (sem=sem@entry=0x7fa21ce46d40, abstime=abstime@entry=0x7fa21ce46c30, clockid=0) at ./nptl/sem_waitcommon.c:111

4 0x00007fa21d243ad3 in __new_sem_wait_slow64 (sem=0x7fa21ce46d40, abstime=0x7fa21ce46c30, clockid=0) at ./nptl/sem_waitcommon.c:183

5 0x00000000004827e0 in Firebird::SignalSafeSemaphore::tryEnter (this=this@entry=0x7fa21ce46d40, seconds=seconds@entry=60, milliseconds=60000, milliseconds@entry=0) at /usr/home/firebird/3.0.10/src/common/classes/semaphore.cpp:202

6 0x000000000042f68e in Worker::wait (this=this@entry=0x7fa21ce46d30, timeout=timeout@entry=60) at /usr/home/firebird/3.0.10/src/remote/server/server.cpp:6355

7 0x00000000004442ae in loopThread () at /usr/home/firebird/3.0.10/src/remote/server/server.cpp:6159

8 0x0000000000458e85 in (anonymous namespace)::ThreadArgs::run (this=) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:78

9 (anonymous namespace)::threadStart (arg=0x7fa21d142ee0) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:94

10 0x00007fa21d23bb43 in start_thread (arg=) at ./nptl/pthread_create.c:442

11 0x00007fa21d2cda00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7fa21d173c00 (LWP 782751) "firebird"):

0 0x00007fa21d2bfd7f in __GI___poll (fds=0x7fa21d1419b8, nfds=1, timeout=60000) at ../sysdeps/unix/sysv/linux/poll.c:29

1 0x000000000041abf1 in Select::select (this=0x7fa21d1419a0, timeout=) at /usr/home/firebird/3.0.10/src/remote/inet.cpp:455

2 0x0000000000417134 in select_wait (selct=0x7fa21d1419a0, main_port=0x7fa21ce4c6c0) at /usr/home/firebird/3.0.10/src/remote/inet.cpp:2382

3 select_multi (main_port=0x7fa21ce4c6c0, buffer=0x7fa21ce50d40 "", bufsize=8192, length=0x7ffcfa1f8fa0, port=...) at /usr/home/firebird/3.0.10/src/remote/inet.cpp:2 160

4 0x0000000000436823 in SRVR_multi_thread (main_port=0x7fa21ce4c6c0, flags=flags@entry=0) at /usr/home/firebird/3.0.10/src/remote/server/server.cpp:1560

5 0x0000000000446a2a in main (argc=, argv=) at /usr/home/firebird/3.0.10/src/remote/server/os/posix/inet_server.cpp:423

AlexPeshkoff commented 1 year ago

Yes, stack traces are OK. I see EventManager in one of threads. That means your process can deliver events back to client, that adds one connection required for it. I.e. ONE additional opened socket is normal when you work with events, and yes, sooner of all it's not shown in monitoring.

How many connections does this particular (or similar - I doubt this one is still alive) process have actually? I.e. please add netstat for a PID next time you send me stack traces.

tomaszdubiel18 commented 1 year ago

Hello. This time I send stack for a regular Firebird process listening on the main port 3050. This process PID is 1035711 Attaching to process 1035711 [New LWP 1035712] [New LWP 1035713] [New LWP 1035714] [New LWP 1035716] [New LWP 1035734] [New LWP 1035735] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00007f6c98bddd7f in __GI___poll (fds=0x7f6c993ec9b8, nfds=1, timeout=60000) at ../sysdeps/unix/sysv/linux/poll.c:29 29 ../sysdeps/unix/sysv/linux/poll.c: No such file or directory. (gdb) thread apply all bt

Thread 7 (Thread 0x7f6c4e3ab640 (LWP 1035735) "firebird"):

0 __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f6c993e19d8) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, clockid=0, expected=0, futex_word=0x7f6c993e19d8) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f6c993e19d8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=1 28) at ./nptl/futex-internal.c:139

3 0x00007f6c98b58ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f6c993e1988, cond=0x7f6c993e19b0) at ./nptl/pthread_cond_wait.c:503

4 ___pthread_cond_wait (cond=0x7f6c993e19b0, mutex=0x7f6c993e1988) at ./nptl/pthread_cond_wait.c:627

5 0x00007f6c96a873c0 in Firebird::SharedMemoryBase::eventWait (this=, event=0x7f6c993e1980, value=2, micro_seconds=) at /usr/home/firebird/3.0.10/src/common/is c_sync.cpp:852

6 0x00007f6c9676d11b in Jrd::EventManager::watcher_thread (this=0x7f6c96f774b0) at /usr/home/firebird/3.0.10/src/jrd/event.cpp:1409

7 0x00007f6c9676ee13 in ThreadFinishSync<Jrd::EventManager*>::internalRun (this=this@entry=0x7f6c96f77550) at /usr/home/firebird/3.0.10/src/jrd/../common/ThreadStart.h:189

8 ThreadFinishSync<Jrd::EventManager*>::internalRun (arg=arg@entry=0x7f6c96f77550) at /usr/home/firebird/3.0.10/src/jrd/../common/ThreadStart.h:181

9 0x00007f6c96a6f6f5 in (anonymous namespace)::ThreadArgs::run (this=) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:78

10 (anonymous namespace)::threadStart (arg=0x7f6c96f76b30) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:94

11 0x00007f6c98b59b43 in start_thread (arg=) at ./nptl/pthread_create.c:442

12 0x00007f6c98beba00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 6 (Thread 0x7f6c8d7ff640 (LWP 1035734) "firebird"):

0 __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f6c993e4664) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, clockid=0, expected=0, futex_word=0x7f6c993e4664) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f6c993e4664, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=1 28) at ./nptl/futex-internal.c:139

3 0x00007f6c98b58ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f6c993e4610, cond=0x7f6c993e4638) at ./nptl/pthread_cond_wait.c:503

4 ___pthread_cond_wait (cond=0x7f6c993e4638, mutex=0x7f6c993e4610) at ./nptl/pthread_cond_wait.c:627

5 0x00007f6c96a873c0 in Firebird::SharedMemoryBase::eventWait (this=, event=0x7f6c993e4608, value=2, micro_seconds=) at /usr/home/firebird/3.0.10/src/common/is c_sync.cpp:852

6 0x00007f6c969fcc28 in Jrd::LockManager::blocking_action_thread (this=0x7f6c96f76b90) at /usr/home/firebird/3.0.10/src/lock/lock.cpp:1563

7 0x00007f6c969fdf83 in ThreadFinishSync<Jrd::LockManager*>::internalRun (this=this@entry=0x7f6c96f76c28) at /usr/home/firebird/3.0.10/src/lock/../lock/../common/ThreadStart.h:189

8 ThreadFinishSync<Jrd::LockManager*>::internalRun (arg=arg@entry=0x7f6c96f76c28) at /usr/home/firebird/3.0.10/src/lock/../lock/../common/ThreadStart.h:181

9 0x00007f6c96a6f6f5 in (anonymous namespace)::ThreadArgs::run (this=) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:78

10 (anonymous namespace)::threadStart (arg=0x7f6c96f75e30) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:94

11 0x00007f6c98b59b43 in start_thread (arg=) at ./nptl/pthread_create.c:442

12 0x00007f6c98beba00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7f6c94aff640 (LWP 1035716) "firebird"):

0 __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f6c94b00960) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, clockid=0, expected=0, futex_word=0x7f6c94b00960) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f6c94b00960, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=1 28) at ./nptl/futex-internal.c:139

3 0x00007f6c98b58ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f6c94b00910, cond=0x7f6c94b00938) at ./nptl/pthread_cond_wait.c:503

4 ___pthread_cond_wait (cond=0x7f6c94b00938, mutex=0x7f6c94b00910) at ./nptl/pthread_cond_wait.c:627

5 0x00007f6c96a873c0 in Firebird::SharedMemoryBase::eventWait (this=, event=0x7f6c94b00908, value=1, micro_seconds=) at /usr/home/firebird/3.0.10/src/common/is c_sync.cpp:852

6 0x00007f6c966e2be9 in (anonymous namespace)::MappingIpc::clearDeliveryThread (this=0x7f6c96f9ab20) at /usr/home/firebird/3.0.10/src/jrd/Mapping.cpp:835

7 (anonymous namespace)::MappingIpc::clearDelivery (mapping=0x7f6c96f9ab20) at /usr/home/firebird/3.0.10/src/jrd/Mapping.cpp:912

--Type for more, q to quit, c to continue without paging--c

8 0x00007f6c966df733 in ThreadFinishSync<(anonymous namespace)::MappingIpc*>::internalRun (this=this@entry=0x7f6c96f9ab80) at /usr/home/firebird/3.0.10/src/jrd/../common/ThreadStart.h:189

9 ThreadFinishSync<(anonymous namespace)::MappingIpc*>::internalRun (arg=arg@entry=0x7f6c96f9ab80) at /usr/home/firebird/3.0.10/src/jrd/../common/ThreadStart.h:181

10 0x00007f6c96a6f6f5 in (anonymous namespace)::ThreadArgs::run (this=) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:78

11 (anonymous namespace)::threadStart (arg=0x7f6c96f75640) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:94

12 0x00007f6c98b59b43 in start_thread (arg=) at ./nptl/pthread_create.c:442

13 0x00007f6c98beba00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7f6c977c9640 (LWP 1035714) "firebird"):

0 __futex_abstimed_wait_common64 (private=, cancel=true, abstime=0x7f6c977c8d10, op=393, expected=0, futex_word=0x7f6c9940d2f0) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=, abstime=0x7f6c977c8d10, clockid=0, expected=0, futex_word=0x7f6c9940d2f0) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f6c9940d2f0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f6c977c8d10, private=) at ./nptl/futex-internal.c:139

3 0x00007f6c98b61a30 in do_futex_wait (sem=sem@entry=0x7f6c9940d2f0, abstime=abstime@entry=0x7f6c977c8d10, clockid=0) at ./nptl/sem_waitcommon.c:111

4 0x00007f6c98b61ad3 in __new_sem_wait_slow64 (sem=0x7f6c9940d2f0, abstime=0x7f6c977c8d10, clockid=0) at ./nptl/sem_waitcommon.c:183

5 0x00007f6c99134340 in Firebird::SignalSafeSemaphore::tryEnter (this=0x7f6c9940d2f0, seconds=, milliseconds=) at /usr/home/firebird/3.0.10/src/common/classes/ semaphore.cpp:202

6 0x00007f6c990437d5 in Why::(anonymous namespace)::TimerEntry::timeThread () at /usr/home/firebird/3.0.10/src/yvalve/MasterImplementation.cpp:298

7 0x00007f6c990ff9f5 in (anonymous namespace)::ThreadArgs::run (this=) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:78

8 (anonymous namespace)::threadStart (arg=0x7f6c99413e50) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:94

9 0x00007f6c98b59b43 in start_thread (arg=) at ./nptl/pthread_create.c:442

10 0x00007f6c98beba00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7f6c97fca640 (LWP 1035713) "firebird"):

0 __futex_abstimed_wait_common64 (private=, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f6c99411350) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=, abstime=0x0, clockid=0, expected=0, futex_word=0x7f6c99411350) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f6c99411350, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private= ) at ./nptl/futex-internal.c:139

3 0x00007f6c98b61c5f in do_futex_wait (sem=sem@entry=0x7f6c99411350, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111

4 0x00007f6c98b61cf8 in __new_sem_wait_slow64 (sem=0x7f6c99411350, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183

5 0x00007f6c99134252 in Firebird::SignalSafeSemaphore::enter (this=0x7f6c99411350) at /usr/home/firebird/3.0.10/src/common/classes/semaphore.cpp:163

6 0x00007f6c9908c715 in (anonymous namespace)::shutdownThread () at /usr/home/firebird/3.0.10/src/yvalve/why.cpp:754

7 0x00007f6c990ff9f5 in (anonymous namespace)::ThreadArgs::run (this=) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:78

8 (anonymous namespace)::threadStart (arg=0x7f6c99411390) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:94

9 0x00007f6c98b59b43 in start_thread (arg=) at ./nptl/pthread_create.c:442

10 0x00007f6c98beba00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7f6c987cb640 (LWP 1035712) "firebird"):

0 __futex_abstimed_wait_common64 (private=, cancel=true, abstime=0x7f6c987cac30, op=393, expected=0, futex_word=0x7f6c987cad40) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=, abstime=0x7f6c987cac30, clockid=0, expected=0, futex_word=0x7f6c987cad40) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f6c987cad40, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f6c987cac30, private=) at ./nptl/futex-internal.c:139

3 0x00007f6c98b61a30 in do_futex_wait (sem=sem@entry=0x7f6c987cad40, abstime=abstime@entry=0x7f6c987cac30, clockid=0) at ./nptl/sem_waitcommon.c:111

4 0x00007f6c98b61ad3 in __new_sem_wait_slow64 (sem=0x7f6c987cad40, abstime=0x7f6c987cac30, clockid=0) at ./nptl/sem_waitcommon.c:183

5 0x00000000004827e0 in Firebird::SignalSafeSemaphore::tryEnter (this=this@entry=0x7f6c987cad40, seconds=seconds@entry=60, milliseconds=60000, milliseconds@entry=0) at /usr/home/firebird/3. 0.10/src/common/classes/semaphore.cpp:202

6 0x000000000042f68e in Worker::wait (this=this@entry=0x7f6c987cad30, timeout=timeout@entry=60) at /usr/home/firebird/3.0.10/src/remote/server/server.cpp:6355

7 0x00000000004442ae in loopThread () at /usr/home/firebird/3.0.10/src/remote/server/server.cpp:6159

8 0x0000000000458e85 in (anonymous namespace)::ThreadArgs::run (this=) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:78

9 (anonymous namespace)::threadStart (arg=0x7f6c993edee0) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:94

10 0x00007f6c98b59b43 in start_thread (arg=) at ./nptl/pthread_create.c:442

11 0x00007f6c98beba00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7f6c9941ec00 (LWP 1035711) "firebird"):

0 0x00007f6c98bddd7f in __GI___poll (fds=0x7f6c993ec9b8, nfds=1, timeout=60000) at ../sysdeps/unix/sysv/linux/poll.c:29

1 0x000000000041abf1 in Select::select (this=0x7f6c993ec9a0, timeout=) at /usr/home/firebird/3.0.10/src/remote/inet.cpp:455

2 0x0000000000417134 in select_wait (selct=0x7f6c993ec9a0, main_port=0x7f6c987d06c0) at /usr/home/firebird/3.0.10/src/remote/inet.cpp:2382

3 select_multi (main_port=0x7f6c987d06c0, buffer=0x7f6c987d4d40 "", bufsize=8192, length=0x7ffdb3df2980, port=...) at /usr/home/firebird/3.0.10/src/remote/inet.cpp:2160

4 0x0000000000436823 in SRVR_multi_thread (main_port=0x7f6c987d06c0, flags=flags@entry=0) at /usr/home/firebird/3.0.10/src/remote/server/server.cpp:1560

5 0x0000000000446a2a in main (argc=, argv=) at /usr/home/firebird/3.0.10/src/remote/server/os/posix/inet_server.cpp:423

tomaszdubiel18 commented 1 year ago

Here is another example, with some netstat info added. Can you see now what is the reason of those processes established without flag "keepalive" remaining active?

root@stream:/home/stream# netstat -ano --timers | grep "xxx:60929" tcp6 0 0 xxx:3050 xxx:60929 ESTABLISHED off (0.00/0/0)

root@stream:/home/stream# lsof -i :3050 | grep "xxx:60929" firebird 1040351 firebird 1u IPv6 12399451 0t0 TCP stream:gds-db->xxx:60929 (ESTABLISHED)

Attaching to process 1040351 [New LWP 1040352] [New LWP 1040353] [New LWP 1040354] [New LWP 1040356] [New LWP 1040358] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00007f6c98bddd7f in __GI___poll (fds=0x7f6c993ec9b8, nfds=1, timeout=60000) at ../sysdeps/unix/sysv/linux/poll.c:29 29 ../sysdeps/unix/sysv/linux/poll.c: No such file or directory. (gdb) thread apply all bt

Thread 6 (Thread 0x7f6c8d7ff640 (LWP 1040358) "firebird"):

0 __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f6c993e49c8) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, clockid=0, expected=0, futex_word=0x7f6c993e49c8) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f6c993e49c8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139

3 0x00007f6c98b58ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f6c993e4978, cond=0x7f6c993e49a0) at ./nptl/pthread_cond_wait.c:503

4 ___pthread_cond_wait (cond=0x7f6c993e49a0, mutex=0x7f6c993e4978) at ./nptl/pthread_cond_wait.c:627

5 0x00007f6c96a873c0 in Firebird::SharedMemoryBase::eventWait (this=, event=0x7f6c993e4970, value=933, micro_seconds=) at /usr/home/firebird/3.0.10/src/common/isc_sync.cpp:852

6 0x00007f6c969fcc28 in Jrd::LockManager::blocking_action_thread (this=0x7f6c96f76b90) at /usr/home/firebird/3.0.10/src/lock/lock.cpp:1563

7 0x00007f6c969fdf83 in ThreadFinishSync<Jrd::LockManager*>::internalRun (this=this@entry=0x7f6c96f76c28) at /usr/home/firebird/3.0.10/src/lock/../lock/../common/ThreadStart.h:189

8 ThreadFinishSync<Jrd::LockManager*>::internalRun (arg=arg@entry=0x7f6c96f76c28) at /usr/home/firebird/3.0.10/src/lock/../lock/../common/ThreadStart.h:181

9 0x00007f6c96a6f6f5 in (anonymous namespace)::ThreadArgs::run (this=) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:78

10 (anonymous namespace)::threadStart (arg=0x7f6c96f75e30) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:94

11 0x00007f6c98b59b43 in start_thread (arg=) at ./nptl/pthread_create.c:442

12 0x00007f6c98beba00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 5 (Thread 0x7f6c94aff640 (LWP 1040356) "firebird"):

0 __futex_abstimed_wait_common64 (private=128, cancel=true, abstime=0x0, op=265, expected=0, futex_word=0x7f6c94b048a8) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=128, abstime=0x0, clockid=0, expected=0, futex_word=0x7f6c94b048a8) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f6c94b048a8, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=private@entry=128) at ./nptl/futex-internal.c:139

3 0x00007f6c98b58ac1 in __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0x7f6c94b04858, cond=0x7f6c94b04880) at ./nptl/pthread_cond_wait.c:503

4 ___pthread_cond_wait (cond=0x7f6c94b04880, mutex=0x7f6c94b04858) at ./nptl/pthread_cond_wait.c:627

5 0x00007f6c96a873c0 in Firebird::SharedMemoryBase::eventWait (this=, event=0x7f6c94b04850, value=1, micro_seconds=) at /usr/home/firebird/3.0.10/src/common/isc_sync.cpp:852

6 0x00007f6c966e2be9 in (anonymous namespace)::MappingIpc::clearDeliveryThread (this=0x7f6c96f9ab20) at /usr/home/firebird/3.0.10/src/jrd/Mapping.cpp:835

7 (anonymous namespace)::MappingIpc::clearDelivery (mapping=0x7f6c96f9ab20) at /usr/home/firebird/3.0.10/src/jrd/Mapping.cpp:912

8 0x00007f6c966df733 in ThreadFinishSync<(anonymous namespace)::MappingIpc*>::internalRun (this=this@entry=0x7f6c96f9ab80) at /usr/home/firebird/3.0.10/src/jrd/../common/ThreadStart.h:189

9 ThreadFinishSync<(anonymous namespace)::MappingIpc*>::internalRun (arg=arg@entry=0x7f6c96f9ab80) at /usr/home/firebird/3.0.10/src/jrd/../common/ThreadStart.h:181

10 0x00007f6c96a6f6f5 in (anonymous namespace)::ThreadArgs::run (this=) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:78

11 (anonymous namespace)::threadStart (arg=0x7f6c96f75640) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:94

12 0x00007f6c98b59b43 in start_thread (arg=) at ./nptl/pthread_create.c:442

13 0x00007f6c98beba00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 4 (Thread 0x7f6c977c9640 (LWP 1040354) "firebird"):

0 __futex_abstimed_wait_common64 (private=, cancel=true, abstime=0x7f6c977c8d10, op=393, expected=0, futex_word=0x7f6c9940d2f0) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=, abstime=0x7f6c977c8d10, clockid=0, expected=0, futex_word=0x7f6c9940d2f0) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f6c9940d2f0, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f6c977c8d10, private=) at ./nptl/futex-internal.c:139

3 0x00007f6c98b61a30 in do_futex_wait (sem=sem@entry=0x7f6c9940d2f0, abstime=abstime@entry=0x7f6c977c8d10, clockid=0) at ./nptl/sem_waitcommon.c:111

4 0x00007f6c98b61ad3 in __new_sem_wait_slow64 (sem=0x7f6c9940d2f0, abstime=0x7f6c977c8d10, clockid=0) at ./nptl/sem_waitcommon.c:183

5 0x00007f6c99134340 in Firebird::SignalSafeSemaphore::tryEnter (this=0x7f6c9940d2f0, seconds=, milliseconds=) at /usr/home/firebird/3.0.10/src/common/classes/semaphore.cpp:202

6 0x00007f6c990437d5 in Why::(anonymous namespace)::TimerEntry::timeThread () at /usr/home/firebird/3.0.10/src/yvalve/MasterImplementation.cpp:298

7 0x00007f6c990ff9f5 in (anonymous namespace)::ThreadArgs::run (this=) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:78

8 (anonymous namespace)::threadStart (arg=0x7f6c99413e50) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:94

9 0x00007f6c98b59b43 in start_thread (arg=) at ./nptl/pthread_create.c:442

10 0x00007f6c98beba00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7f6c97fca640 (LWP 1040353) "firebird"):

0 __futex_abstimed_wait_common64 (private=, cancel=true, abstime=0x0, op=393, expected=0, futex_word=0x7f6c99411350) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=, abstime=0x0, clockid=0, expected=0, futex_word=0x7f6c99411350) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f6c99411350, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, private=) at ./nptl/futex-internal.c:139

3 0x00007f6c98b61c5f in do_futex_wait (sem=sem@entry=0x7f6c99411350, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:111

4 0x00007f6c98b61cf8 in __new_sem_wait_slow64 (sem=0x7f6c99411350, abstime=0x0, clockid=0) at ./nptl/sem_waitcommon.c:183

5 0x00007f6c99134252 in Firebird::SignalSafeSemaphore::enter (this=0x7f6c99411350) at /usr/home/firebird/3.0.10/src/common/classes/semaphore.cpp:163

6 0x00007f6c9908c715 in (anonymous namespace)::shutdownThread () at /usr/home/firebird/3.0.10/src/yvalve/why.cpp:754

7 0x00007f6c990ff9f5 in (anonymous namespace)::ThreadArgs::run (this=) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:78

8 (anonymous namespace)::threadStart (arg=0x7f6c99411390) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:94

9 0x00007f6c98b59b43 in start_thread (arg=) at ./nptl/pthread_create.c:442

10 0x00007f6c98beba00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7f6c987cb640 (LWP 1040352) "firebird"):

0 __futex_abstimed_wait_common64 (private=, cancel=true, abstime=0x7f6c987cac30, op=393, expected=0, futex_word=0x7f6c987cad40) at ./nptl/futex-internal.c:57

1 __futex_abstimed_wait_common (cancel=true, private=, abstime=0x7f6c987cac30, clockid=0, expected=0, futex_word=0x7f6c987cad40) at ./nptl/futex-internal.c:87

2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f6c987cad40, expected=expected@entry=0, clockid=clockid@entry=0, abstime=abstime@entry=0x7f6c987cac30, private=) at ./nptl/futex-internal.c:139

--Type for more, q to quit, c to continue without paging--c

3 0x00007f6c98b61a30 in do_futex_wait (sem=sem@entry=0x7f6c987cad40, abstime=abstime@entry=0x7f6c987cac30, clockid=0) at ./nptl/sem_waitcommon.c:111

4 0x00007f6c98b61ad3 in __new_sem_wait_slow64 (sem=0x7f6c987cad40, abstime=0x7f6c987cac30, clockid=0) at ./nptl/sem_waitcommon.c:183

5 0x00000000004827e0 in Firebird::SignalSafeSemaphore::tryEnter (this=this@entry=0x7f6c987cad40, seconds=seconds@entry=60, milliseconds=60000, milliseconds@entry=0) at /usr/home/firebird/3.0.10/src/common/classes/semaphore.cpp:202

6 0x000000000042f68e in Worker::wait (this=this@entry=0x7f6c987cad30, timeout=timeout@entry=60) at /usr/home/firebird/3.0.10/src/remote/server/server.cpp:6355

7 0x00000000004442ae in loopThread () at /usr/home/firebird/3.0.10/src/remote/server/server.cpp:6159

8 0x0000000000458e85 in (anonymous namespace)::ThreadArgs::run (this=) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:78

9 (anonymous namespace)::threadStart (arg=0x7f6c993edee0) at /usr/home/firebird/3.0.10/src/common/ThreadStart.cpp:94

10 0x00007f6c98b59b43 in start_thread (arg=) at ./nptl/pthread_create.c:442

11 0x00007f6c98beba00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7f6c9941ec00 (LWP 1040351) "firebird"):

0 0x00007f6c98bddd7f in __GI___poll (fds=0x7f6c993ec9b8, nfds=1, timeout=60000) at ../sysdeps/unix/sysv/linux/poll.c:29

1 0x000000000041abf1 in Select::select (this=0x7f6c993ec9a0, timeout=) at /usr/home/firebird/3.0.10/src/remote/inet.cpp:455

2 0x0000000000417134 in select_wait (selct=0x7f6c993ec9a0, main_port=0x7f6c987d06c0) at /usr/home/firebird/3.0.10/src/remote/inet.cpp:2382

3 select_multi (main_port=0x7f6c987d06c0, buffer=0x7f6c987d4d40 "", bufsize=8192, length=0x7ffdb3df2980, port=...) at /usr/home/firebird/3.0.10/src/remote/inet.cpp:2160

4 0x0000000000436823 in SRVR_multi_thread (main_port=0x7f6c987d06c0, flags=flags@entry=0) at /usr/home/firebird/3.0.10/src/remote/server/server.cpp:1560

5 0x0000000000446a2a in main (argc=, argv=) at /usr/home/firebird/3.0.10/src/remote/server/os/posix/inet_server.cpp:423

AlexPeshkoff commented 1 year ago

Here is another example, with some netstat info added. Can you see now what is the reason of those processes established without flag "keepalive" remaining active?

Taking into an account that this is FB3 Classic on Linux with systemd - I suppose problems are with systemd's firebird-classic.socket file in /usr/lib/systemd/system (or /lib/systemd/system) - is line KeepAlive=true present in it? Default file has that line.

FB Classic on linux before v.4 does not open socket itself - it's just using one opened by systemd (or xinetd on some systems).

tomaszdubiel18 commented 1 year ago

Yes, there is that line in this file.

AlexPeshkoff commented 1 year ago

In that case formally it's systemd problem - socket should have keepalive set. But telling true I doubt that :-) Sooner of all your server is misconfigured in some way.