search

FirebirdSQL / firebird

Firebird server, client and tools
https://www.firebirdsql.org/
1.24k stars 215 forks source link

Can not connect to any .fdb after stopped heavy OLTP-test (750 connections) [CORE3811] #4154

Open firebird-automations opened 12 years ago

firebird-automations commented 12 years ago

Submitted by: @pavel-zotov

Attachments: gdb_fb_smp_server.zip

hi all

ISQL Version: WI-V2.5.1.26351 Firebird 2.5 Server version: Firebird/linux AMD64 (access method), version "LI-V2.5.2.26448 Firebird 2.5" Firebird/linux AMD64 (remote server), version "LI-V2.5.2.26448 Firebird 2.5/tcp (fbozzy)/P12" Firebird/x86/Windows NT (remote interface), version "WI-V2.5.1.26351 Firebird 2.5/tcp (tlprg)/P12" on disk structure version 11.2

Yesterday I run test with 750 connections with small and medium loading of database in each of them. Today I've stopped this test (first I created some `special` stop-file which serves as signal to each connection script to be canceled - this way kills \~300 session; but after I kill all rest isql`s windows via pskill.exe).

Now the count of threads in fb_smp_server are 4: bash-3.2$ ps -FLC fb_smp_server UID PID PPID LWP C NLWP SZ RSS PSR STIME TTY TIME CMD firebird 5982 5981 5982 0 4 20882 5108 2 13:50 ? 00:00:00 /opt/firebird/bin/fb_smp_server firebird 5982 5981 5984 0 4 20882 5108 3 13:50 ? 00:00:00 /opt/firebird/bin/fb_smp_server firebird 5982 5981 5985 0 4 20882 5108 7 13:50 ? 00:00:00 /opt/firebird/bin/fb_smp_server firebird 5982 5981 5986 0 4 20882 5108 3 13:50 ? 00:00:00 /opt/firebird/bin/fb_smp_server

Unfortunatelly I can not establish new connections to any of databases that are placed on this machine. The `top` utility for fb_smp_server shows that it is in idle state:

$ top -p $(pgrep fb_smp_server)

top - 15:06:57 up 1 day, 14:37, 7 users, load average: 0.02, 0.01, 0.00 Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu1 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu2 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu3 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu4 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu5 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Cpu7 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 32933152k total, 20914936k used, 12018216k free, 224876k buffers Swap: 33792716k total, 196k used, 33792520k free, 20130404k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 5982 firebird 18 0 83528 5108 3624 S 0.0 0.0 0:00.41 fb_smp_server

The `iostat` utility also show that there is no any disk activity:

bash-3.2$ iostat -d -m -t 5 Linux 2.6.18-308.el5 (fbozzy) 04/09/2012

Time: 03:08:09 PM Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 195.40 0.66 2.44 91477 339559 sda1 0.00 0.00 0.00 1 0 sda2 187.78 0.01 1.94 1956 270086 sda3 0.00 0.00 0.00 0 0 sda4 0.00 0.00 0.00 0 0 sda5 7.61 0.64 0.50 89518 69472

Time: 03:08:14 PM Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 0.40 0.00 0.01 0 0 sda1 0.00 0.00 0.00 0 0 sda2 0.40 0.00 0.01 0 0 sda3 0.00 0.00 0.00 0 0 sda4 0.00 0.00 0.00 0 0 sda5 0.00 0.00 0.00 0 0

Time: 03:08:19 PM Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 0.00 0.00 0.00 0 0 sda1 0.00 0.00 0.00 0 0 sda2 0.00 0.00 0.00 0 0 sda3 0.00 0.00 0.00 0 0 sda4 0.00 0.00 0.00 0 0 sda5 0.00 0.00 0.00 0 0

Time: 03:08:24 PM Device: tps MB_read/s MB_wrtn/s MB_read MB_wrtn sda 1.00 0.00 0.01 0 0 sda1 0.00 0.00 0.00 0 0 sda2 1.00 0.00 0.01 0 0 sda3 0.00 0.00 0.00 0 0 sda4 0.00 0.00 0.00 0 0 sda5 0.00 0.00 0.00 0 0

But any connect to any database (via local or TCP) hangs. What should be done in such cases ?

firebird-automations commented 12 years ago

Commented by: @pavel-zotov

two files in attach:

1) gdb -q -x ... /opt/firebird/bin/fb_smp_server 1) gdb -q -x ... /opt/firebird/bin/.debug/fb_smp_server.debug

firebird-automations commented 12 years ago
Modified by: @pavel-zotov Attachment: gdb\_fb\_smp\_server\.zip \[ 12129 \]
firebird-automations commented 12 years ago

Commented by: @pavel-zotov

PS. I see the following output of netstat on this server (may be this will help):

netstat -anep | grep 3050 tcp 0 0 0.0.0.0:3050 0.0.0.0:* LISTEN 54323 382680 5982/fb_smp_server tcp 157 0 127.0.0.1:3050 127.0.0.1:38899 CLOSE_WAIT 0 0 - tcp 157 0 127.0.0.1:3050 127.0.0.1:38898 CLOSE_WAIT 0 0