search

FirebirdSQL / firebird

Firebird server, client and tools
https://www.firebirdsql.org/
1.25k stars 215 forks source link

fbserver periodically crashes under load [CORE544] #898

Open firebird-automations opened 18 years ago

firebird-automations commented 18 years ago

Submitted by: venditere (venditere)

Is duplicated by CORE1297

Votes: 4

SFID: 1467936#⁠ Submitted By: venditere

Firebird will crash peridically and enter the following message in firebird.log \firebird\bin\fbserver.exe: terminated abnormally (4294967295) The guardian will then restart the service. It does not crash on a regular interval. Sometimes it will have only been running for 20 minutes and others it may have lasted several hours. When the server crashes it produces a .lck file for which I not yet found a use.

Environment: Firebird Superserver 1.5.2.4731 Windows Server 2003 Service Pack 1 Dual Xeon Processor. Processor Affinity set on fbserver.exe to processor 1. HyperThreading is disabled in BIOS.

Application information: Jaybird jdbc driver used. Hibernate / Spring primarily use database. Apache Tomcat container.

The load on the server is fairly constant as this is a controlled test environment. There will be about 50 clients from 2 servers at all times. There will be many more writes than reads in this test scenario. The size of the database is not large about 200 MB. In order to provide more information I installed the .pdb files for firebird and attached WinDbg in order to obtain the following information:

******************************************************* ************************

CThread::setThreadIndex FAILED hr = 0x80004002 CThread::setThreadIndex FAILED hr = 0x80004002 CThread::setThreadIndex FAILED hr = 0x80004002

FAULTING_IP: fbserver!cache_transactions+8a [C:\fb15build\latest\firebird2\src\jrd\tpc.cpp @ 420] 0048634a 8903 mov [ebx],eax

EXCEPTION_RECORD: ffffffff -- (.exr ffffffffffffffff) ExceptionAddress: 0048634a (fbserver! cache_transactions+0x0000008a) ExceptionCode: c0000005 (Access violation) ExceptionFlags: 00000000 NumberParameters: 2 Parameter[0]: 00000001 Parameter[1]: 0f3d586c Attempt to write to address 0f3d586c

FAULTING_THREAD: 00000988

DEFAULT_BUCKET_ID: APPLICATION_FAULT

PROCESS_NAME: fbserver.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced memory at "0x%08lx". The memory could not be "%s".

WRITE_ADDRESS: 0f3d586c

BUGCHECK_STR: ACCESS_VIOLATION

LAST_CONTROL_TRANSFER: from 00485f2b to 0048634a

STACK_TEXT:
04ddf980 00485f2b 00007fb0 0f3d586c 0014f2e1 fbserver! cache_transactions+0x8a [C:\fb15build\latest\firebird2 \src\jrd\tpc.cpp @ 420] 04ddf994 00440849 04ddfaac 0014f2e2 04ddfaac fbserver! TPC_initialize_tpc+0x5b [C:\fb15build\latest\firebird2 \src\jrd\tpc.cpp @ 140] 04ddfa3c 004224ff 04ddfaac 00000005 0a2f9f70 fbserver! TRA_start+0x319 [C:\fb15build\latest\firebird2 \src\jrd\tra.cpp @ 1599] 04ddfa74 0044546b 0f674984 000001f4 0f67486c fbserver! jrd8_start_multiple+0x14f [C:\fb15build\latest\firebird2\src\jrd\jrd.cpp @ 3758] 04ddfbf4 00404c3d 0f3cd5a4 0f3cd59c 000000e8 fbserver! VIO_verb_cleanup+0x74b [C:\fb15build\latest\firebird2 \src\jrd\vio.cpp @ 2926] 00000000 00000000 00000000 00000000 00000000 fbserver! Firebird::BePlusTree<Firebird::BlockInfo,Firebird::Bloc kInfo,Firebird::MemoryPool::InternalAllocator,Firebird: :DefaultKeyValue<Firebird::BlockInfo>,Firebird::BlockIn fo,100,100>::add+0x15d [C:\fb15build\latest\firebird2 \src\common\classes\tree.h @ 488]

FOLLOWUP_IP: fbserver!cache_transactions+8a [C:\fb15build\latest\firebird2\src\jrd\tpc.cpp @ 420] 0048634a 8903 mov [ebx],eax

SYMBOL_STACK_INDEX: 0

FOLLOWUP_NAME: MachineOwner

SYMBOL_NAME: fbserver!cache_transactions+8a

MODULE_NAME: fbserver

IMAGE_NAME: fbserver.exe

DEBUG_FLR_IMAGE_TIMESTAMP: 41bd4f03

STACK_COMMAND: \~21s ; kb

FAILURE_BUCKET_ID: ACCESS_VIOLATION_fbserver! cache_transactions+8a

BUCKET_ID: ACCESS_VIOLATION_fbserver! cache_transactions+8a

Followup: MachineOwner ---------

firebird-automations commented 18 years ago
Modified by: Alice F. Bird (firebirds) description: SFID: 1467936#⁠ Submitted By: venditere Firebird will crash peridically and enter the following message in firebird\.log \\firebird\\bin\\fbserver\.exe: terminated abnormally \(4294967295\) The guardian will then restart the service\. It does not crash on a regular interval\. Sometimes it will have only been running for 20 minutes and others it may have lasted several hours\. When the server crashes it produces a \.lck file for which I not yet found a use\. Environment: Firebird Superserver 1\.5\.2\.4731 Windows Server 2003 Service Pack 1 Dual Xeon Processor\. Processor Affinity set on fbserver\.exe to processor 1\. HyperThreading is disabled in BIOS\. Application information: Jaybird jdbc driver used\. Hibernate / Spring primarily use database\. Apache Tomcat container\. The load on the server is fairly constant as this is a controlled test environment\. There will be about 50 clients from 2 servers at all times\. There will be many more writes than reads in this test scenario\. The size of the database is not large about 200 MB\. In order to provide more information I installed the \.pdb files for firebird and attached WinDbg in order to obtain the following information: \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* CThread::setThreadIndex FAILED hr = 0x80004002 CThread::setThreadIndex FAILED hr = 0x80004002 CThread::setThreadIndex FAILED hr = 0x80004002 FAULTING\_IP: fbserver\!cache\_transactions\+8a \[C:\\fb15build\\latest\\firebird2\\src\\jrd\\tpc\.cpp @ 420\] 0048634a 8903 mov \[ebx\],eax EXCEPTION\_RECORD: ffffffff \-\- \(\.exr ffffffffffffffff\) ExceptionAddress: 0048634a \(fbserver\! cache\_transactions\+0x0000008a\) ExceptionCode: c0000005 \(Access violation\) ExceptionFlags: 00000000 NumberParameters: 2 Parameter\[0\]: 00000001 Parameter\[1\]: 0f3d586c Attempt to write to address 0f3d586c FAULTING\_THREAD: 00000988 DEFAULT\_BUCKET\_ID: APPLICATION\_FAULT PROCESS\_NAME: fbserver\.exe ERROR\_CODE: \(NTSTATUS\) 0xc0000005 \- The instruction at "0x%08lx" referenced memory at "0x%08lx"\. The memory could not be "%s"\. WRITE\_ADDRESS: 0f3d586c BUGCHECK\_STR: ACCESS\_VIOLATION LAST\_CONTROL\_TRANSFER: from 00485f2b to 0048634a STACK\_TEXT: 04ddf980 00485f2b 00007fb0 0f3d586c 0014f2e1 fbserver\! cache\_transactions\+0x8a \[C:\\fb15build\\latest\\firebird2 \\src\\jrd\\tpc\.cpp @ 420\] 04ddf994 00440849 04ddfaac 0014f2e2 04ddfaac fbserver\! TPC\_initialize\_tpc\+0x5b \[C:\\fb15build\\latest\\firebird2 \\src\\jrd\\tpc\.cpp @ 140\] 04ddfa3c 004224ff 04ddfaac 00000005 0a2f9f70 fbserver\! TRA\_start\+0x319 \[C:\\fb15build\\latest\\firebird2 \\src\\jrd\\tra\.cpp @ 1599\] 04ddfa74 0044546b 0f674984 000001f4 0f67486c fbserver\! jrd8\_start\_multiple\+0x14f \[C:\\fb15build\\latest\\firebird2\\src\\jrd\\jrd\.cpp @ 3758\] 04ddfbf4 00404c3d 0f3cd5a4 0f3cd59c 000000e8 fbserver\! VIO\_verb\_cleanup\+0x74b \[C:\\fb15build\\latest\\firebird2 \\src\\jrd\\vio\.cpp @ 2926\] 00000000 00000000 00000000 00000000 00000000 fbserver\! Firebird::BePlusTree<Firebird::BlockInfo,Firebird::Bloc kInfo,Firebird::MemoryPool::InternalAllocator,Firebird: :DefaultKeyValue<Firebird::BlockInfo>,Firebird::BlockIn fo,100,100>::add\+0x15d \[C:\\fb15build\\latest\\firebird2 \\src\\common\\classes\\tree\.h @ 488\] FOLLOWUP\_IP: fbserver\!cache\_transactions\+8a \[C:\\fb15build\\latest\\firebird2\\src\\jrd\\tpc\.cpp @ 420\] 0048634a 8903 mov \[ebx\],eax SYMBOL\_STACK\_INDEX: 0 FOLLOWUP\_NAME: MachineOwner SYMBOL\_NAME: fbserver\!cache\_transactions\+8a MODULE\_NAME: fbserver IMAGE\_NAME: fbserver\.exe DEBUG\_FLR\_IMAGE\_TIMESTAMP: 41bd4f03 STACK\_COMMAND: \~21s ; kb FAILURE\_BUCKET\_ID: ACCESS\_VIOLATION\_fbserver\! cache\_transactions\+8a BUCKET\_ID: ACCESS\_VIOLATION\_fbserver\! cache\_transactions\+8a Followup: MachineOwner \-\-\-\-\-\-\-\-\- =\> SFID: 1467936#⁠ Submitted By: venditere Firebird will crash peridically and enter the following message in firebird\.log \\firebird\\bin\\fbserver\.exe: terminated abnormally \(4294967295\) The guardian will then restart the service\. It does not crash on a regular interval\. Sometimes it will have only been running for 20 minutes and others it may have lasted several hours\. When the server crashes it produces a \.lck file for which I not yet found a use\. Environment: Firebird Superserver 1\.5\.2\.4731 Windows Server 2003 Service Pack 1 Dual Xeon Processor\. Processor Affinity set on fbserver\.exe to processor 1\. HyperThreading is disabled in BIOS\. Application information: Jaybird jdbc driver used\. Hibernate / Spring primarily use database\. Apache Tomcat container\. The load on the server is fairly constant as this is a controlled test environment\. There will be about 50 clients from 2 servers at all times\. There will be many more writes than reads in this test scenario\. The size of the database is not large about 200 MB\. In order to provide more information I installed the \.pdb files for firebird and attached WinDbg in order to obtain the following information: \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* CThread::setThreadIndex FAILED hr = 0x80004002 CThread::setThreadIndex FAILED hr = 0x80004002 CThread::setThreadIndex FAILED hr = 0x80004002 FAULTING\_IP: fbserver\!cache\_transactions\+8a \[C:\\fb15build\\latest\\firebird2\\src\\jrd\\tpc\.cpp @ 420\] 0048634a 8903 mov \[ebx\],eax EXCEPTION\_RECORD: ffffffff \-\- \(\.exr ffffffffffffffff\) ExceptionAddress: 0048634a \(fbserver\! cache\_transactions\+0x0000008a\) ExceptionCode: c0000005 \(Access violation\) ExceptionFlags: 00000000 NumberParameters: 2 Parameter\[0\]: 00000001 Parameter\[1\]: 0f3d586c Attempt to write to address 0f3d586c FAULTING\_THREAD: 00000988 DEFAULT\_BUCKET\_ID: APPLICATION\_FAULT PROCESS\_NAME: fbserver\.exe ERROR\_CODE: \(NTSTATUS\) 0xc0000005 \- The instruction at "0x%08lx" referenced memory at "0x%08lx"\. The memory could not be "%s"\. WRITE\_ADDRESS: 0f3d586c BUGCHECK\_STR: ACCESS\_VIOLATION LAST\_CONTROL\_TRANSFER: from 00485f2b to 0048634a STACK\_TEXT: 04ddf980 00485f2b 00007fb0 0f3d586c 0014f2e1 fbserver\! cache\_transactions\+0x8a \[C:\\fb15build\\latest\\firebird2 \\src\\jrd\\tpc\.cpp @ 420\] 04ddf994 00440849 04ddfaac 0014f2e2 04ddfaac fbserver\! TPC\_initialize\_tpc\+0x5b \[C:\\fb15build\\latest\\firebird2 \\src\\jrd\\tpc\.cpp @ 140\] 04ddfa3c 004224ff 04ddfaac 00000005 0a2f9f70 fbserver\! TRA\_start\+0x319 \[C:\\fb15build\\latest\\firebird2 \\src\\jrd\\tra\.cpp @ 1599\] 04ddfa74 0044546b 0f674984 000001f4 0f67486c fbserver\! jrd8\_start\_multiple\+0x14f \[C:\\fb15build\\latest\\firebird2\\src\\jrd\\jrd\.cpp @ 3758\] 04ddfbf4 00404c3d 0f3cd5a4 0f3cd59c 000000e8 fbserver\! VIO\_verb\_cleanup\+0x74b \[C:\\fb15build\\latest\\firebird2 \\src\\jrd\\vio\.cpp @ 2926\] 00000000 00000000 00000000 00000000 00000000 fbserver\! Firebird::BePlusTree,Firebird::BlockIn fo,100,100\>::add\+0x15d \[C:\\fb15build\\latest\\firebird2 \\src\\common\\classes\\tree\.h @ 488\] FOLLOWUP\_IP: fbserver\!cache\_transactions\+8a \[C:\\fb15build\\latest\\firebird2\\src\\jrd\\tpc\.cpp @ 420\] 0048634a 8903 mov \[ebx\],eax SYMBOL\_STACK\_INDEX: 0 FOLLOWUP\_NAME: MachineOwner SYMBOL\_NAME: fbserver\!cache\_transactions\+8a MODULE\_NAME: fbserver IMAGE\_NAME: fbserver\.exe DEBUG\_FLR\_IMAGE\_TIMESTAMP: 41bd4f03 STACK\_COMMAND: \~21s ; kb FAILURE\_BUCKET\_ID: ACCESS\_VIOLATION\_fbserver\! cache\_transactions\+8a BUCKET\_ID: ACCESS\_VIOLATION\_fbserver\! cache\_transactions\+8a Followup: MachineOwner \-\-\-\-\-\-\-\-\-
firebird-automations commented 17 years ago

Commented by: Thomas Stamm (thomas)

I'm encountering the same problem, an for me it is a SHOW STOPPER, it happens in an production environment, after using nearly the same configuration for years.

Nearly same Environment except I already tried newest version 1.5.4.4910

Firebird Superserver 1.5.4.4910 Windows Server 2003 R2 Service Pack 1 Dual Xeon Processor. Processor Affinity set on fbserver.exe to processor 2. HyperThreading is disabled in BIOS.

I already installed the debug Version and now have and dump file of this exception, if it is of interest I can post it (\~150MB).

00 062ff980 0048647b 00007fb0 0bc8252c 014fadb0 fbserver!cache_transactions+0x8e [C:\fb15build\R1_5_4\firebird2\src\jrd\tpc.cpp @ 421] 01 062ff994 00440d99 062ffaac 014fadb3 062ffaac fbserver!TPC_initialize_tpc+0x5b [C:\fb15build\R1_5_4\firebird2\src\jrd\tpc.cpp @ 140] 02 062ffa3c 0042282f 062ffaac 00000005 05c40340 fbserver!TRA_start+0x319 [C:\fb15build\R1_5_4\firebird2\src\jrd\tra.cpp @ 1608] 03 062ffa58 77e67159 062ffbb0 00000001 0001d4c0 fbserver!jrd8_start_multiple+0x14f [C:\fb15build\R1_5_4\firebird2\src\jrd\jrd.cpp @ 3758] WARNING: Stack unwind information not available. Following frames may be wrong. 04 062ffd70 00404c5d 003d3b34 003d3b2c 00000148 kernel32!ResetEvent+0x83 05 062ffdb8 00404440 003d3b2c 0813bed8 0813beb0 fbserver!Firebird::BePlusTree<Firebird::BlockInfo,Firebird::BlockInfo,Firebird::MemoryPool::InternalAllocator,Firebird::DefaultKeyValue<Firebird::BlockInfo>,Firebird::BlockInfo,100,100>::add+0x15d [C:\fb15build\R1_5_4\firebird2\src\common\classes\tree.h @ 488] 06 062ffdf0 0040486a 003d3998 0854e580 0261dfcc fbserver!Firebird::MemoryPool::addFreeBlock+0x40 [C:\fb15build\R1_5_4\firebird2\src\common\classes\alloc.cpp @ 520] 07 062ffe7c 0041b9cc 00000007 0041b9e5 0040acea fbserver!Firebird::MemoryPool::deallocate+0x15a [C:\fb15build\R1_5_4\firebird2\src\common\classes\alloc.cpp @ 625] 08 062ffe84 0041b9e5 0040acea 062fff30 0051716d fbserver!ThreadPriorityScheduler::InternalGet+0xc [C:\fb15build\R1_5_4\firebird2\src\jrd\os\win32\thd_priority.cpp @ 86] 09 062ffe88 0040acea 062fff30 0051716d 0000001d fbserver!ThreadPriorityScheduler::Set+0x5 [C:\fb15build\R1_5_4\firebird2\src\jrd\os\win32\thd_priority.cpp @ 95] 0a 062ffe90 0051716d 0000001d 0854e93c 0854e580 fbserver!put_specific+0xa [C:\fb15build\R1_5_4\firebird2\src\jrd\thd.cpp @ 1816] 0b 062fff50 0051914e 0592c0b4 0854e580 0854e800 fbserver!process_packet+0x22d [C:\fb15build\R1_5_4\firebird2\src\remote\server.cpp @ 3146] 0c 062fff84 77bcb530 00000000 00000000 00000000 fbserver!thread+0xce [C:\fb15build\R1_5_4\firebird2\src\remote\server.cpp @ 4863] 0d 062fffb8 77e66063 00ff2008 00000000 00000000 msvcrt!endthreadex+0xa3 0e 062fffec 00000000 77bcb4bc 00ff2008 00000000 kernel32!GetModuleFileNameA+0xeb

------------

0048687e 8b742434 mov esi,dword ptr [esp+34h] 00486882 33d2 xor edx,edx 00486884 8bc6 mov eax,esi 00486886 f7f1 div eax,ecx 00486888 2bf2 sub esi,edx 0048688a 3bf5 cmp esi,ebp 0048688c 7718 ja fbserver!cache_transactions+0x96 (004868a6) 0048688e 56 push esi 0048688f 57 push edi 00486890 e83b000000 call fbserver!allocate_tpc (004868d0) 00486895 8b4c2434 mov ecx,dword ptr [esp+34h] 00486899 83c408 add esp,8 0048689c 03f1 add esi,ecx 0048689e 8903 mov dword ptr [ebx],eax ds:0023:0bc8252c=???????? 004868a0 3bf5 cmp esi,ebp 004868a2 8bd8 mov ebx,eax 004868a4 76e8 jbe fbserver!cache_transactions+0x7e (0048688e) 004868a6 8b442434 mov eax,dword ptr [esp+34h] 004868aa 55 push ebp 004868ab 50 push eax 004868ac 6a00 push 0 004868ae 57 push edi 004868af e81c93fbff call fbserver!TRA_get_inventory (0043fbd0) 004868b4 8b442420 mov eax,dword ptr [esp+20h] 004868b8 83c410 add esp,10h 004868bb 5f pop edi

firebird-automations commented 17 years ago
Modified by: @dyemanov Link: This issue is duplicated by [CORE1297](https://github.com/FirebirdSQL/firebird/issues?q=CORE1297+in%3Atitle) \[ [CORE1297](https://github.com/FirebirdSQL/firebird/issues?q=CORE1297+in%3Atitle) \]
firebird-automations commented 16 years ago
Modified by: @pcisar Workflow: jira \[ 10568 \] =\> Firebird \[ 14889 \]