search

FirebirdSQL / firebird

Firebird server, client and tools
https://www.firebirdsql.org/
1.26k stars 216 forks source link

Client Application Frozen. [CORE2763] #3156

Closed firebird-automations closed 10 months ago

firebird-automations commented 14 years ago

Submitted by: Greg (greg)

We have a client machine, running 8 same processes. It works some days and suddenly, client application gets frozen. We're connected to 3 different databases, with medium load. We also already get this pb while connected on local databases (Windows machine).

Here is a madexcept dump showing where it hungs.

date/time : 2009-11-20, 15:58:24, 679ms computer name : SRV-***************. wts client name : ************** user name : ***************. <admin> operating system : Windows 2003 Service Pack 1 build 3790 system language : French system up time : 2 days 1 hour program up time : 1 minute 30 seconds processors : 8x Intel(R) Xeon(R) CPU E5345 @ 2.33GHz physical memory : 2311/3326 MB (free/total) free disk space : (C:) 8,73 GB display mode : 1356x905, 16 bit process id : $f74 allocated memory : 117,96 MB executable : ***************.exe exec. date/time : 2009-11-19 14:31 version : 4.0.3.46 madExcept version : 3.0b callstack crc : $bb2ac731, $d5ed6834, $98c74299 count : 3 exception number : 1 exception message : The application seems to be frozen.

main thread ($12e0): 7c94ed54 +000 ntdll.dll KiFastSystemCallRet 7c942122 +00a ntdll.dll NtWaitForSingleObject 71ac17fa +06a WS2HELP.dll WahReferenceContextByHandle 71ad4709 +09e WS2_32.dll select 100205b4 +0f4 gds32.dll isc_dsql_execute2_m 10021e7c +15c gds32.dll isc_dsql_execute2 004deccb +027 ***************.exe DB 2801 +4 TDefCollection.DoUpdate 004dec9b +02b ***************.exe DB 2793 +1 TDefCollection.Update 0046dc73 +017 ***************.exe Classes 4021 +1 TCollection.Changed 0046dcf4 +010 ***************.exe Classes 4041 +2 TCollection.EndUpdate 0050d94f +02b ***************.exe IBIntf 1593 +1 TDynamicLibrary.isc_dsql_execute2 00508c8c +174 ***************.exe IBSQL 2393 +28 TIBSQL.ExecQuery 004fbfde +0b6 ***************.exe IBCustomDataSet 3606 +14 TIBCustomDataSet.InternalExecQuery 005149f6 +096 ***************.exe IBStoredProc 144 +12 TIBStoredProc.ExecProc 00585e53 +5a7 ***************.exe _SomeUnits 4912 +150 ***************.***************. 00588286 +28e ***************.exe _SomeUnits 5632 +62 ***************.***************. 00583d9b +06b ***************.exe _SomeUnits 4152 +7 ***************.***************. 005cc96a +5fa ***************.exe _SomeOtherUnits 452 +65 *************** 005ce5c2 +382 ***************.exe _SomeOtherUnits 837 +108 ***************.***************. 005d2052 +09a ***************.exe _SomeOtherUnits 1615 +4 ***************.***************. 0049763b +01b ***************.exe ExtCtrls 2250 +1 TTimer.Timer 004974a7 +02f ***************.exe ExtCtrls 2208 +4 TTimer.WndProc 0047944c +014 ***************.exe Classes 11572 +8 StdWndProc

thread $e84: 7c94ed54 +00 ntdll.dll KiFastSystemCallRet 7c942112 +0a ntdll.dll NtWaitForMultipleObjects 7c827115 +00 kernel32.dll WaitForMultipleObjectsEx 77f4cd02 +00 USER32.dll MsgWaitForMultipleObjectsEx 77f3e37c +1a USER32.dll MsgWaitForMultipleObjects 0044e5bd +0d ******.exe madExcept CallThreadProcSafe 0044e627 +37 ******.exe madExcept ThreadExceptFrame >> created by main thread ($12e0) at: 4de36b7d +00 gdiplus.dll

thread $168c (TIBEventThread): 7c94ed54 +00 ntdll.dll KiFastSystemCallRet 7c942122 +0a ntdll.dll NtWaitForSingleObject 7c82baa2 +82 kernel32.dll WaitForSingleObjectEx 7c82ba0d +0d kernel32.dll WaitForSingleObject 00489f0e +6a ***************.exe SyncObjs 241 +14 THandleObject.WaitFor 005a3dad +61 ***************.exe IBEvents 429 +9 TIBEventThread.Execute 0044e6db +2b ***************.exe madExcept HookedTThreadExecute 00477582 +36 ***************.exe Classes 9866 +7 ThreadProc 00405658 +28 ***************.exe System 12127 +33 ThreadWrapper 0044e5bd +0d ***************.exe madExcept CallThreadProcSafe 0044e627 +37 ***************.exe madExcept ThreadExceptFrame >> created by main thread ($12e0) at: 005a3e4a +36 ***************.exe IBEvents 447 +1 TIBEventThread.Create

thread $1a4: <priority:2> 7c94ed54 +00 ntdll.dll KiFastSystemCallRet 7c942122 +0a ntdll.dll NtWaitForSingleObject 71ad7fb4 +7e WS2_32.dll recv 781329a7 +38 MSVCR80.dll _endthreadex 0044e5bd +0d ***************.exe madExcept CallThreadProcSafe 0044e627 +37 ***************.exe madExcept ThreadExceptFrame >> created by main thread ($12e0) at: 78132ab6 +7f MSVCR80.dll _beginthreadex

hardware: + Cartes graphiques - ATI ES1000 (driver 8.24.3.0) - Winvnc video hook driver (driver 1.1.0.0) + Cartes multifonction - HP iLO Management Channel Interface Driver (driver 1.12.0.0) + Cartes réseau - HP NC373i Multifunction Gigabit Server Adapter (driver 3.4.10.0) - HP NC373i Multifunction Gigabit Server Adapter #⁠2 (driver 3.4.10.0) + Claviers - Clavier standard 101/102 touches ou clavier Microsoft Natural Keyboard PS/2 - Périphérique clavier PIH + Contrôleurs ATA/ATAPI IDE - Canal IDE principal - Canal IDE secondaire - Contrôleur IDE standard double canal PCI + Contrôleurs audio, vidéo et jeu - Codecs audio - Codecs vidéo - Pilotes audio hérités - Périphériques de capture vidéo hérités - Périphériques MCI + Contrôleurs de bus USB - Concentrateur USB générique - Concentrateur USB racine - Concentrateur USB racine - Concentrateur USB racine - Concentrateur USB racine - Concentrateur USB racine - Concentrateur USB racine - Contrôleur hôte PCI vers USB standard universel - Contrôleur hôte PCI vers USB standard universel - Contrôleur hôte PCI vers USB standard universel - Contrôleur hôte PCI vers USB standard universel - Contrôleur hôte PCI vers USB standard universel - Contrôleur hôte PCI vers USB standard étendu - Périphérique USB composite + Contrôleurs SCSI et RAID - Smart Array P400 Controller (driver 6.8.0.32) + Lecteurs de CD-ROM/DVD-ROM - HL-DT-ST RW/DVD GCC-C10N + Lecteurs de disque - HP LOGICAL VOLUME SCSI Disk Device + Moniteurs - Écran par défaut - Écran par défaut + Ordinateur - PC multiprocesseur ACPI + Ports (COM et LPT) - Port de communication (COM1) - Port de communication (COM2) + Processeurs - Intel(R) Xeon(R) CPU E5345 @ 2.33GHz - Intel(R) Xeon(R) CPU E5345 @ 2.33GHz - Intel(R) Xeon(R) CPU E5345 @ 2.33GHz - Intel(R) Xeon(R) CPU E5345 @ 2.33GHz - Intel(R) Xeon(R) CPU E5345 @ 2.33GHz - Intel(R) Xeon(R) CPU E5345 @ 2.33GHz - Intel(R) Xeon(R) CPU E5345 @ 2.33GHz - Intel(R) Xeon(R) CPU E5345 @ 2.33GHz + Périphériques d'interface utilisateur (HID) - Périphérique d'interface utilisateur USB - Périphérique d'interface utilisateur USB + Périphériques système - Bouton de fonctionnalité définie ACPI - Bus d'E/S étendu - Bus PCI - Contrôleur d'accès direct en mémoire - Gestionnaire de disque logique - Gestionnaire de volume - Haut-parleur système - High precision event timer (driver 1.0.0.0) - Horloge système - HP NC373i Virtual Bus Device (driver 3.4.10.0) - HP NC373i Virtual Bus Device (driver 3.4.10.0) - HP NULL IPMI Controller (driver 1.0.0.0) - HP ProLiant iLO 2 Legacy Support Function (driver 1.4.0.0) - HP ProLiant iLO 2 Management Controller Driver (driver 1.4.0.0) - N° LUN virtuel HP - Pilote BIOS de gestion de systèmes Microsoft - Pilote clavier de Terminal Server - Pilote souris de Terminal Server - Pont CPU hôte standard PCI - Pont CPU hôte standard PCI - Pont CPU hôte standard PCI - Pont CPU hôte standard PCI - Pont CPU hôte standard PCI - Pont CPU hôte standard PCI - Pont CPU hôte standard PCI - Pont CPU hôte standard PCI - Pont Intel(R) 82801 PCI - 244E - Pont ISA standard PCI - Pont PCI vers PCI standard PCI - Pont PCI vers PCI standard PCI - Pont PCI vers PCI standard PCI - Pont PCI vers PCI standard PCI - Pont PCI vers PCI standard PCI - Pont PCI vers PCI standard PCI - Pont PCI vers PCI standard PCI - Pont PCI vers PCI standard PCI - Pont PCI vers PCI standard PCI - Pont PCI vers PCI standard PCI - Pont PCI vers PCI standard PCI - Pont PCI vers PCI standard PCI - Pont PCI vers PCI standard PCI - Pont PCI vers PCI standard PCI - Pont PCI vers PCI standard PCI - Port de lecture de données ISAPNP - Périphérique de mise à jour microcode - Redirecteur de périphérique Terminal Server - Ressources de la carte mère - Système compatible ACPI Microsoft - Zone thermique ACPI - Énumérateur de périphérique logiciel Plug-and-Play + Souris et autres périphériques de pointage - Souris compatible PS/2 - Souris HID

disassembling: [...] 004decbc mov byte ptr [eax+$20], 0 004decc0 2801 mov edx, [ebp-4] 004decc3 mov eax, [ebp-4] 004decc6 mov eax, [eax+$1c] 004decc9 mov ecx, [eax] 004deccb > call dword ptr [ecx+$d0] 004decd1 2803 pop ecx 004decd2 pop ecx 004decd3 pop ebp 004decd4 ret

firebird-automations commented 14 years ago
Modified by: Greg (greg) description: We have a client machine, running 8 same processes\. It works some days and suddenly, client application gets frozen\. We're connected to 3 different databases, with medium load \(DUAL CORE CPUs\)\. We also already get this pb while connected on local databases \(Windows machine\)\. Here is a madexcept dump showing where it hungs\. date/time : 2009\-11\-20, 15:58:24, 679ms computer name : SRV\-\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\. wts client name : \*\*\*\*\*\*\*\*\*\*\*\*\*\* user name : \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\. operating system : Windows 2003 Service Pack 1 build 3790 system language : French system up time : 2 days 1 hour program up time : 1 minute 30 seconds processors : 8x Intel\(R\) Xeon\(R\) CPU E5345 @ 2\.33GHz physical memory : 2311/3326 MB \(free/total\) free disk space : \(C:\) 8,73 GB display mode : 1356x905, 16 bit process id : $f74 allocated memory : 117,96 MB executable : \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe exec\. date/time : 2009\-11\-19 14:31 version : 4\.0\.3\.46 madExcept version : 3\.0b callstack crc : $bb2ac731, $d5ed6834, $98c74299 count : 3 exception number : 1 exception message : The application seems to be frozen\. main thread \($12e0\): 7c94ed54 \+000 ntdll\.dll KiFastSystemCallRet 7c942122 \+00a ntdll\.dll NtWaitForSingleObject 71ac17fa \+06a WS2HELP\.dll WahReferenceContextByHandle 71ad4709 \+09e WS2\_32\.dll select 100205b4 \+0f4 gds32\.dll isc\_dsql\_execute2\_m 10021e7c \+15c gds32\.dll isc\_dsql\_execute2 004deccb \+027 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe DB 2801 \+4 TDefCollection\.DoUpdate 004dec9b \+02b \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe DB 2793 \+1 TDefCollection\.Update 0046dc73 \+017 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe Classes 4021 \+1 TCollection\.Changed 0046dcf4 \+010 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe Classes 4041 \+2 TCollection\.EndUpdate 0050d94f \+02b \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe IBIntf 1593 \+1 TDynamicLibrary\.isc\_dsql\_execute2 00508c8c \+174 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe IBSQL 2393 \+28 TIBSQL\.ExecQuery 004fbfde \+0b6 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe IBCustomDataSet 3606 \+14 TIBCustomDataSet\.InternalExecQuery 005149f6 \+096 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe IBStoredProc 144 \+12 TIBStoredProc\.ExecProc 00585e53 \+5a7 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe \_SomeUnits 4912 \+150 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\. 00588286 \+28e \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe \_SomeUnits 5632 \+62 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\. 00583d9b \+06b \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe \_SomeUnits 4152 \+7 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\. 005cc96a \+5fa \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe \_SomeOtherUnits 452 \+65 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\* 005ce5c2 \+382 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe \_SomeOtherUnits 837 \+108 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\. 005d2052 \+09a \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe \_SomeOtherUnits 1615 \+4 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\. 0049763b \+01b \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe ExtCtrls 2250 \+1 TTimer\.Timer 004974a7 \+02f \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe ExtCtrls 2208 \+4 TTimer\.WndProc 0047944c \+014 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe Classes 11572 \+8 StdWndProc thread $e84: 7c94ed54 \+00 ntdll\.dll KiFastSystemCallRet 7c942112 \+0a ntdll\.dll NtWaitForMultipleObjects 7c827115 \+00 kernel32\.dll WaitForMultipleObjectsEx 77f4cd02 \+00 USER32\.dll MsgWaitForMultipleObjectsEx 77f3e37c \+1a USER32\.dll MsgWaitForMultipleObjects 0044e5bd \+0d \*\*\*\*\*\*\.exe madExcept CallThreadProcSafe 0044e627 \+37 \*\*\*\*\*\*\.exe madExcept ThreadExceptFrame \>\> created by main thread \($12e0\) at: 4de36b7d \+00 gdiplus\.dll thread $168c \(TIBEventThread\): 7c94ed54 \+00 ntdll\.dll KiFastSystemCallRet 7c942122 \+0a ntdll\.dll NtWaitForSingleObject 7c82baa2 \+82 kernel32\.dll WaitForSingleObjectEx 7c82ba0d \+0d kernel32\.dll WaitForSingleObject 00489f0e \+6a \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe SyncObjs 241 \+14 THandleObject\.WaitFor 005a3dad \+61 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe IBEvents 429 \+9 TIBEventThread\.Execute 0044e6db \+2b \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe madExcept HookedTThreadExecute 00477582 \+36 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe Classes 9866 \+7 ThreadProc 00405658 \+28 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe System 12127 \+33 ThreadWrapper 0044e5bd \+0d \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe madExcept CallThreadProcSafe 0044e627 \+37 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe madExcept ThreadExceptFrame \>\> created by main thread \($12e0\) at: 005a3e4a \+36 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe IBEvents 447 \+1 TIBEventThread\.Create thread $1a4: 7c94ed54 \+00 ntdll\.dll KiFastSystemCallRet 7c942122 \+0a ntdll\.dll NtWaitForSingleObject 71ad7fb4 \+7e WS2\_32\.dll recv 781329a7 \+38 MSVCR80\.dll \_endthreadex 0044e5bd \+0d \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe madExcept CallThreadProcSafe 0044e627 \+37 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe madExcept ThreadExceptFrame \>\> created by main thread \($12e0\) at: 78132ab6 \+7f MSVCR80\.dll \_beginthreadex hardware: \+ Cartes graphiques \- ATI ES1000 \(driver 8\.24\.3\.0\) \- Winvnc video hook driver \(driver 1\.1\.0\.0\) \+ Cartes multifonction \- HP iLO Management Channel Interface Driver \(driver 1\.12\.0\.0\) \+ Cartes réseau \- HP NC373i Multifunction Gigabit Server Adapter \(driver 3\.4\.10\.0\) \- HP NC373i Multifunction Gigabit Server Adapter #⁠2 \(driver 3\.4\.10\.0\) \+ Claviers \- Clavier standard 101/102 touches ou clavier Microsoft Natural Keyboard PS/2 \- Périphérique clavier PIH \+ Contrôleurs ATA/ATAPI IDE \- Canal IDE principal \- Canal IDE secondaire \- Contrôleur IDE standard double canal PCI \+ Contrôleurs audio, vidéo et jeu \- Codecs audio \- Codecs vidéo \- Pilotes audio hérités \- Périphériques de capture vidéo hérités \- Périphériques MCI \+ Contrôleurs de bus USB \- Concentrateur USB générique \- Concentrateur USB racine \- Concentrateur USB racine \- Concentrateur USB racine \- Concentrateur USB racine \- Concentrateur USB racine \- Concentrateur USB racine \- Contrôleur hôte PCI vers USB standard universel \- Contrôleur hôte PCI vers USB standard universel \- Contrôleur hôte PCI vers USB standard universel \- Contrôleur hôte PCI vers USB standard universel \- Contrôleur hôte PCI vers USB standard universel \- Contrôleur hôte PCI vers USB standard étendu \- Périphérique USB composite \+ Contrôleurs SCSI et RAID \- Smart Array P400 Controller \(driver 6\.8\.0\.32\) \+ Lecteurs de CD\-ROM/DVD\-ROM \- HL\-DT\-ST RW/DVD GCC\-C10N \+ Lecteurs de disque \- HP LOGICAL VOLUME SCSI Disk Device \+ Moniteurs \- Écran par défaut \- Écran par défaut \+ Ordinateur \- PC multiprocesseur ACPI \+ Ports \(COM et LPT\) \- Port de communication \(COM1\) \- Port de communication \(COM2\) \+ Processeurs \- Intel\(R\) Xeon\(R\) CPU E5345 @ 2\.33GHz \- Intel\(R\) Xeon\(R\) CPU E5345 @ 2\.33GHz \- Intel\(R\) Xeon\(R\) CPU E5345 @ 2\.33GHz \- Intel\(R\) Xeon\(R\) CPU E5345 @ 2\.33GHz \- Intel\(R\) Xeon\(R\) CPU E5345 @ 2\.33GHz \- Intel\(R\) Xeon\(R\) CPU E5345 @ 2\.33GHz \- Intel\(R\) Xeon\(R\) CPU E5345 @ 2\.33GHz \- Intel\(R\) Xeon\(R\) CPU E5345 @ 2\.33GHz \+ Périphériques d'interface utilisateur \(HID\) \- Périphérique d'interface utilisateur USB \- Périphérique d'interface utilisateur USB \+ Périphériques système \- Bouton de fonctionnalité définie ACPI \- Bus d'E/S étendu \- Bus PCI \- Contrôleur d'accès direct en mémoire \- Gestionnaire de disque logique \- Gestionnaire de volume \- Haut\-parleur système \- High precision event timer \(driver 1\.0\.0\.0\) \- Horloge système \- HP NC373i Virtual Bus Device \(driver 3\.4\.10\.0\) \- HP NC373i Virtual Bus Device \(driver 3\.4\.10\.0\) \- HP NULL IPMI Controller \(driver 1\.0\.0\.0\) \- HP ProLiant iLO 2 Legacy Support Function \(driver 1\.4\.0\.0\) \- HP ProLiant iLO 2 Management Controller Driver \(driver 1\.4\.0\.0\) \- N° LUN virtuel HP \- Pilote BIOS de gestion de systèmes Microsoft \- Pilote clavier de Terminal Server \- Pilote souris de Terminal Server \- Pont CPU hôte standard PCI \- Pont CPU hôte standard PCI \- Pont CPU hôte standard PCI \- Pont CPU hôte standard PCI \- Pont CPU hôte standard PCI \- Pont CPU hôte standard PCI \- Pont CPU hôte standard PCI \- Pont CPU hôte standard PCI \- Pont Intel\(R\) 82801 PCI \- 244E \- Pont ISA standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Port de lecture de données ISAPNP \- Périphérique de mise à jour microcode \- Redirecteur de périphérique Terminal Server \- Ressources de la carte mère \- Système compatible ACPI Microsoft \- Zone thermique ACPI \- Énumérateur de périphérique logiciel Plug\-and\-Play \+ Souris et autres périphériques de pointage \- Souris compatible PS/2 \- Souris HID disassembling: \[\.\.\.\] 004decbc mov byte ptr \[eax\+$20\], 0 004decc0 2801 mov edx, \[ebp\-4\] 004decc3 mov eax, \[ebp\-4\] 004decc6 mov eax, \[eax\+$1c\] 004decc9 mov ecx, \[eax\] 004deccb \> call dword ptr \[ecx\+$d0\] 004decd1 2803 pop ecx 004decd2 pop ecx 004decd3 pop ebp 004decd4 ret =\> We have a client machine, running 8 same processes\. It works some days and suddenly, client application gets frozen\. We're connected to 3 different databases, with medium load\. We also already get this pb while connected on local databases \(Windows machine\)\. Here is a madexcept dump showing where it hungs\. date/time : 2009\-11\-20, 15:58:24, 679ms computer name : SRV\-\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\. wts client name : \*\*\*\*\*\*\*\*\*\*\*\*\*\* user name : \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\. operating system : Windows 2003 Service Pack 1 build 3790 system language : French system up time : 2 days 1 hour program up time : 1 minute 30 seconds processors : 8x Intel\(R\) Xeon\(R\) CPU E5345 @ 2\.33GHz physical memory : 2311/3326 MB \(free/total\) free disk space : \(C:\) 8,73 GB display mode : 1356x905, 16 bit process id : $f74 allocated memory : 117,96 MB executable : \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe exec\. date/time : 2009\-11\-19 14:31 version : 4\.0\.3\.46 madExcept version : 3\.0b callstack crc : $bb2ac731, $d5ed6834, $98c74299 count : 3 exception number : 1 exception message : The application seems to be frozen\. main thread \($12e0\): 7c94ed54 \+000 ntdll\.dll KiFastSystemCallRet 7c942122 \+00a ntdll\.dll NtWaitForSingleObject 71ac17fa \+06a WS2HELP\.dll WahReferenceContextByHandle 71ad4709 \+09e WS2\_32\.dll select 100205b4 \+0f4 gds32\.dll isc\_dsql\_execute2\_m 10021e7c \+15c gds32\.dll isc\_dsql\_execute2 004deccb \+027 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe DB 2801 \+4 TDefCollection\.DoUpdate 004dec9b \+02b \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe DB 2793 \+1 TDefCollection\.Update 0046dc73 \+017 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe Classes 4021 \+1 TCollection\.Changed 0046dcf4 \+010 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe Classes 4041 \+2 TCollection\.EndUpdate 0050d94f \+02b \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe IBIntf 1593 \+1 TDynamicLibrary\.isc\_dsql\_execute2 00508c8c \+174 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe IBSQL 2393 \+28 TIBSQL\.ExecQuery 004fbfde \+0b6 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe IBCustomDataSet 3606 \+14 TIBCustomDataSet\.InternalExecQuery 005149f6 \+096 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe IBStoredProc 144 \+12 TIBStoredProc\.ExecProc 00585e53 \+5a7 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe \_SomeUnits 4912 \+150 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\. 00588286 \+28e \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe \_SomeUnits 5632 \+62 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\. 00583d9b \+06b \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe \_SomeUnits 4152 \+7 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\. 005cc96a \+5fa \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe \_SomeOtherUnits 452 \+65 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\* 005ce5c2 \+382 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe \_SomeOtherUnits 837 \+108 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\. 005d2052 \+09a \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe \_SomeOtherUnits 1615 \+4 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\. 0049763b \+01b \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe ExtCtrls 2250 \+1 TTimer\.Timer 004974a7 \+02f \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe ExtCtrls 2208 \+4 TTimer\.WndProc 0047944c \+014 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe Classes 11572 \+8 StdWndProc thread $e84: 7c94ed54 \+00 ntdll\.dll KiFastSystemCallRet 7c942112 \+0a ntdll\.dll NtWaitForMultipleObjects 7c827115 \+00 kernel32\.dll WaitForMultipleObjectsEx 77f4cd02 \+00 USER32\.dll MsgWaitForMultipleObjectsEx 77f3e37c \+1a USER32\.dll MsgWaitForMultipleObjects 0044e5bd \+0d \*\*\*\*\*\*\.exe madExcept CallThreadProcSafe 0044e627 \+37 \*\*\*\*\*\*\.exe madExcept ThreadExceptFrame \>\> created by main thread \($12e0\) at: 4de36b7d \+00 gdiplus\.dll thread $168c \(TIBEventThread\): 7c94ed54 \+00 ntdll\.dll KiFastSystemCallRet 7c942122 \+0a ntdll\.dll NtWaitForSingleObject 7c82baa2 \+82 kernel32\.dll WaitForSingleObjectEx 7c82ba0d \+0d kernel32\.dll WaitForSingleObject 00489f0e \+6a \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe SyncObjs 241 \+14 THandleObject\.WaitFor 005a3dad \+61 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe IBEvents 429 \+9 TIBEventThread\.Execute 0044e6db \+2b \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe madExcept HookedTThreadExecute 00477582 \+36 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe Classes 9866 \+7 ThreadProc 00405658 \+28 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe System 12127 \+33 ThreadWrapper 0044e5bd \+0d \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe madExcept CallThreadProcSafe 0044e627 \+37 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe madExcept ThreadExceptFrame \>\> created by main thread \($12e0\) at: 005a3e4a \+36 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe IBEvents 447 \+1 TIBEventThread\.Create thread $1a4: 7c94ed54 \+00 ntdll\.dll KiFastSystemCallRet 7c942122 \+0a ntdll\.dll NtWaitForSingleObject 71ad7fb4 \+7e WS2\_32\.dll recv 781329a7 \+38 MSVCR80\.dll \_endthreadex 0044e5bd \+0d \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe madExcept CallThreadProcSafe 0044e627 \+37 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\.exe madExcept ThreadExceptFrame \>\> created by main thread \($12e0\) at: 78132ab6 \+7f MSVCR80\.dll \_beginthreadex hardware: \+ Cartes graphiques \- ATI ES1000 \(driver 8\.24\.3\.0\) \- Winvnc video hook driver \(driver 1\.1\.0\.0\) \+ Cartes multifonction \- HP iLO Management Channel Interface Driver \(driver 1\.12\.0\.0\) \+ Cartes réseau \- HP NC373i Multifunction Gigabit Server Adapter \(driver 3\.4\.10\.0\) \- HP NC373i Multifunction Gigabit Server Adapter #⁠2 \(driver 3\.4\.10\.0\) \+ Claviers \- Clavier standard 101/102 touches ou clavier Microsoft Natural Keyboard PS/2 \- Périphérique clavier PIH \+ Contrôleurs ATA/ATAPI IDE \- Canal IDE principal \- Canal IDE secondaire \- Contrôleur IDE standard double canal PCI \+ Contrôleurs audio, vidéo et jeu \- Codecs audio \- Codecs vidéo \- Pilotes audio hérités \- Périphériques de capture vidéo hérités \- Périphériques MCI \+ Contrôleurs de bus USB \- Concentrateur USB générique \- Concentrateur USB racine \- Concentrateur USB racine \- Concentrateur USB racine \- Concentrateur USB racine \- Concentrateur USB racine \- Concentrateur USB racine \- Contrôleur hôte PCI vers USB standard universel \- Contrôleur hôte PCI vers USB standard universel \- Contrôleur hôte PCI vers USB standard universel \- Contrôleur hôte PCI vers USB standard universel \- Contrôleur hôte PCI vers USB standard universel \- Contrôleur hôte PCI vers USB standard étendu \- Périphérique USB composite \+ Contrôleurs SCSI et RAID \- Smart Array P400 Controller \(driver 6\.8\.0\.32\) \+ Lecteurs de CD\-ROM/DVD\-ROM \- HL\-DT\-ST RW/DVD GCC\-C10N \+ Lecteurs de disque \- HP LOGICAL VOLUME SCSI Disk Device \+ Moniteurs \- Écran par défaut \- Écran par défaut \+ Ordinateur \- PC multiprocesseur ACPI \+ Ports \(COM et LPT\) \- Port de communication \(COM1\) \- Port de communication \(COM2\) \+ Processeurs \- Intel\(R\) Xeon\(R\) CPU E5345 @ 2\.33GHz \- Intel\(R\) Xeon\(R\) CPU E5345 @ 2\.33GHz \- Intel\(R\) Xeon\(R\) CPU E5345 @ 2\.33GHz \- Intel\(R\) Xeon\(R\) CPU E5345 @ 2\.33GHz \- Intel\(R\) Xeon\(R\) CPU E5345 @ 2\.33GHz \- Intel\(R\) Xeon\(R\) CPU E5345 @ 2\.33GHz \- Intel\(R\) Xeon\(R\) CPU E5345 @ 2\.33GHz \- Intel\(R\) Xeon\(R\) CPU E5345 @ 2\.33GHz \+ Périphériques d'interface utilisateur \(HID\) \- Périphérique d'interface utilisateur USB \- Périphérique d'interface utilisateur USB \+ Périphériques système \- Bouton de fonctionnalité définie ACPI \- Bus d'E/S étendu \- Bus PCI \- Contrôleur d'accès direct en mémoire \- Gestionnaire de disque logique \- Gestionnaire de volume \- Haut\-parleur système \- High precision event timer \(driver 1\.0\.0\.0\) \- Horloge système \- HP NC373i Virtual Bus Device \(driver 3\.4\.10\.0\) \- HP NC373i Virtual Bus Device \(driver 3\.4\.10\.0\) \- HP NULL IPMI Controller \(driver 1\.0\.0\.0\) \- HP ProLiant iLO 2 Legacy Support Function \(driver 1\.4\.0\.0\) \- HP ProLiant iLO 2 Management Controller Driver \(driver 1\.4\.0\.0\) \- N° LUN virtuel HP \- Pilote BIOS de gestion de systèmes Microsoft \- Pilote clavier de Terminal Server \- Pilote souris de Terminal Server \- Pont CPU hôte standard PCI \- Pont CPU hôte standard PCI \- Pont CPU hôte standard PCI \- Pont CPU hôte standard PCI \- Pont CPU hôte standard PCI \- Pont CPU hôte standard PCI \- Pont CPU hôte standard PCI \- Pont CPU hôte standard PCI \- Pont Intel\(R\) 82801 PCI \- 244E \- Pont ISA standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Pont PCI vers PCI standard PCI \- Port de lecture de données ISAPNP \- Périphérique de mise à jour microcode \- Redirecteur de périphérique Terminal Server \- Ressources de la carte mère \- Système compatible ACPI Microsoft \- Zone thermique ACPI \- Énumérateur de périphérique logiciel Plug\-and\-Play \+ Souris et autres périphériques de pointage \- Souris compatible PS/2 \- Souris HID disassembling: \[\.\.\.\] 004decbc mov byte ptr \[eax\+$20\], 0 004decc0 2801 mov edx, \[ebp\-4\] 004decc3 mov eax, \[ebp\-4\] 004decc6 mov eax, \[eax\+$1c\] 004decc9 mov ecx, \[eax\] 004deccb \> call dword ptr \[ecx\+$d0\] 004decd1 2803 pop ecx 004decd2 pop ecx 004decd3 pop ebp 004decd4 ret
firebird-automations commented 14 years ago

Commented by: Sean Leyne (seanleyne)

Larry Baddock posted in the devel list:

I have seen something very much like this before, in a multithreaded environment.. The solution in my case was to use fbclient2.5.0.19892.dll. My problems magically disappeared. I'm sure that there is a more up to date version of the client lib around (which is production-ready - 2.5.0.19892 was for testing), but I still use the above. Works for me.

firebird-automations commented 14 years ago

Commented by: Greg (greg)

Hi Sean,

i've read somewhere that 2.5 client library fixed this kind of problem, but, is it fully compatible with a Firebird 2.1.1 Server ? I wouldn't like to get other issues to resolve this one :).

I'll try this anyway on test server. Thanks !

firebird-automations commented 14 years ago

Commented by: @dyemanov

Yes, it's fully backward compatible (as much as one can be sure in something getting into the release candidate stage ;-).

firebird-automations commented 14 years ago

Commented by: Greg (greg)

Update :

The problem still exists after using the latest fbclient.dll.2.5.0.24643:

exception number : 1 exception message : The application seems to be frozen.

main thread ($1650): 7c82ed54 +000 ntdll.dll KiFastSystemCallRet 7c822122 +00a ntdll.dll NtWaitForSingleObject 71bf17fa +06a WS2HELP.dll WahReferenceContextByHandle 71c04709 +09e WS2_32.dll select 1001ef73 +103 gds32.dll isc_dsql_execute2_m 100208ca +0fa gds32.dll isc_dsql_execute2 004deccb +027 **.exe DB 2801 +4 TDefCollection.DoUpdate 004dec9b +02b **..exe DB 2793 +1 TDefCollection.Update 0046dc73 +017 **..exe Classes 4021 +1 TCollection.Changed 0046dcf4 +010 **..exe Classes 4041 +2 TCollection.EndUpdate 004fae53 +59b **..exe IBCustomDataSet 3134 +158 TIBCustomDataSet.InternalInitFieldDefs 0050d94f +02b **..exe IBIntf 1593 +1 TDynamicLibrary.isc_dsql_execute2 00508c8c +174 **..exe IBSQL 2393 +28 TIBSQL.ExecQuery 004fbfde +0b6 **..exe IBCustomDataSet 3606 +14 TIBCustomDataSet.InternalExecQuery 005149f6 +096 **..exe IBStoredProc 144 +12 TIBStoredProc.ExecProc

Here is some additionnal informations : - Even restarting the application do not solve the problem. At the first StoredProc called, it hangs. - The only solution i've detected so far is to make a backup/restore. Backup process is extremely slow. - We're using multi-file database (1Go per file).

The database statistics :

Database "/bdd/****.fdb" Database header page information: Flags 0 Checksum 12345 Generation 16882230 Page size 8192 ODS version 11.1 Oldest transaction 8139546 Oldest active 16536296 Oldest snapshot 16536296 Next transaction 16538363 Bumped transaction 1 Sequence number 0 Next attachment ID 343860 Implementation ID 19 Shadow count 0 Page buffers 0 Next header page 0 Database dialect 3 Creation date Jun 22, 2009 15:01:39 Attributes force write 

Variable header data: 
    Sweep interval:         0 
    Continuation file:              /bdd/\*\*\*\*\*\.fdb0 
    Last logical page:              131072

Any suggestions ?

Regards, Greg.

firebird-automations commented 14 years ago

Commented by: @dyemanov

Not a surprise that the backup process is slow. You have a gap of 8 million transactions between OIT and Next counters! I see you have the automatic sweep turned off, but had you ever performed sweep manually? So far it seems that the database is a huge collection of backversions that nobody can clean out because of the stuck OIT and disabled sweep.

firebird-automations commented 14 years ago

Commented by: Greg (greg)

Yes, i've seen it when dumping the information. I disable automatic sweep because of the unpredictable effect on the client process (it might hangs up other running queries.

Anyway, we're checking why some many transactions are left like that. That's completely abnormal.

Do you think that it might be directly in relation with the hanging processes ?

firebird-automations commented 14 years ago

Commented by: @dyemanov

This can happen if some big transaction (the one modifying a lot of data) was rolled back. It makes the OIT to freeze until the next sweep which should both cleanup the changes marked as rolled back and advance the OIT. But it seems never happening, even manually. In turn, the stuck OIT blocks the regular (cooperative and background) garbage collection in the whole database, thus making all operations slower and slower.

In other words, the stuck OIT is not necessarily a problem per se, but only if the sweep is misconfigured or completely disabled.

I cannot say for sure whether it can cause the application to freeze completely, but it could be surely related.

firebird-automations commented 14 years ago

Commented by: Greg (greg)

Even after backup/restore, i have the "frozen" problem.

Transactions are Oldest transaction 477 Oldest active 478 Oldest snapshot 478 Next transaction 531

Latest thing to do is restarting the server, i do not see other solution at this moment.

firebird-automations commented 14 years ago

Commented by: @dyemanov

Before suggesting anything else it's surely worth trying upgrading to the latest stable server version which is v2.1.3.

firebird-automations commented 14 years ago

Commented by: Greg (greg)

Sorry, we're already in 2.1.3.18185, not in 2.1.1 as i was thinking.

firebird-automations commented 14 years ago
Modified by: Greg (greg) environment: gds32\.dll : 6\.3\.1\.17910 Client : Windows server 2003 SP1\. Server : Linux Debian w/ Classic Server 2\.1\.1 =\> gds32\.dll : 6\.3\.1\.17910 Client : Windows server 2003 SP1\. Server : Linux Debian w/ Classic Server 2\.1\.3
firebird-automations commented 14 years ago

Commented by: @hvlad

Why don't you look at MON tables when you think application is frozen ? May be it just executes badly performed query ?

firebird-automations commented 14 years ago

Commented by: Greg (greg)

After further investigations, i think it's more a server performance issue than a clientside problem. A foreign key was broken on a table. Now, without this key available query take 1 sec instead of less than 1 sec for something like 500 records (2 tables with a join). Frozen thread is detected after 1 minute. Why does it take more than 1 min to get data when having 2 or more applications connected ? A concurrency problem ? (i'm trying to help, if i'm wrong, forget the last sentence ^^).

Everything is back and working again but that's quite strange.

firebird-automations commented 14 years ago

Commented by: @hvlad

> Why does it take more than 1 min to get data when having 2 or more applications connected ? A concurrency problem ?

It is not possible to answer without knowledge of query, its PLAN, database statistics at query execution time, etc

Seems this is support question...

firebird-automations commented 14 years ago

Commented by: Greg (greg)

Some news :

We can't reproduce the problem using a SuperServer instead o a Classic Server. The 8 000 000 transactions to sweep are directly linked to the crashed applications. We've monitored them and there's no leaks in a normal use.

As we're able to reproduce the problem, we'll try to create a small application & database ASAP.

firebird-automations commented 14 years ago

Commented by: @hvlad

Is it reproduced with SS and GCPolicy=cooperative ?

BTW, stuck OIT don't stop garbage collection. OST does.

8M transactions gap forced engine to allocate 8M *2 bits for TIP in memory, or 16MB for every snapshot transaction and 16MB for TIP cache in each Firebird process. With many CS processes you can eat all the memory in system and force heavy usage of OS page file... Just a guess...

firebird-automations commented 14 years ago

Commented by: @hvlad

Greg,

> As we're able to reproduce the problem, we'll try to create a small application & database ASAP.

Any news on this ?

firebird-automations commented 14 years ago

Commented by: Greg (greg)

Vlad,

After many tests with SS and GCPolicy=cooperative (which is by default AFAIK edit:oups, that's not, redoing tests), we've successfully reproduce the problem. It's harder as we need 16 processes instead of the 8 to get this problem. Transactions and GC are not the problem themselves, but a consequence of the first bug report I put in this thread (our application try to reconnect, relaunch etc... incrementing starting transactions but no rollback as we're crashed). I think it's due to the heavy use of storedproc, just a guess, I've to check this with a simpler application.

I only need time, time.... days are too short.

firebird-automations commented 14 years ago

Commented by: Sean Leyne (seanleyne)

In reviewing this case, I am convinced that the issue reported in not with the engine but within the application.

The fact that the garbage collection is not occuring explains the "frozen" application.

Further, the fact that the garbage collection is not occuring is to be directly related to the manner in which the application is connecting to the database and/or the connection settings which are being used.

dyemanov commented 10 months ago

No news since 2009, thus closed as not reproducible.