Firebird 4.04 - Superserver (64 Bit) - Crash during backup & Crash normal use

[NilsBoedeker]

Hi,

We use FB 4.04 for two month and we observe a lot of FB crashs during normal use and especial during backup prozess.

a) The crashes happen often when the FB server is under load, 24/7, between 125 and 200 connections and 300 and 400 active statement if we start an backup . We use the parameter -SE but the crash also happend without this parmeter

b) The crash also happend under normal use (without backup. Every 2 days (+- 1 day) we have also an crash of fbserver. the IBGuard restarts the fbserver.exe process.

We use FB with very much RAM (48 GB) and 16 CPUs, FB Settings via. https://cc.ib-aid.com/democalc.html

Additional: We use an own written UDR and der FreeAdhocUDF.dll, we know that UDFs and are possible problem for stability but the most queries are without using this UDFs & UDRs. But the crash dump don't point in this direction

Have you any idea how we can solve the issue wirt the crash during the backup?

Du you need any additional information & data

With best regards

Nils Bödeker

Crash Dump Analyse attached:

---

Verlag Eugen Ulmer

Web- und Softwareentwicklung

Nils Bödeker

Bürgerwohlsweg 7

28215 Bremen

Tel: [+49 172 7468066]

nboedeker@ulmer.de

---

Details

************* Preparing the environment for Debugger Extensions Gallery repositories ************** ExtensionRepository : Implicit UseExperimentalFeatureForNugetShare : true AllowNugetExeUpdate : true NonInteractiveNuget : true AllowNugetMSCredentialProviderInstall : true AllowParallelInitializationOfLocalRepositories : true EnableRedirectToV8JsProvider : false -- Configuring repositories ----> Repository : LocalInstalled, Enabled: true ----> Repository : UserExtensions, Enabled: true >>>>>>>>>>>>> Preparing the environment for Debugger Extensions Gallery repositories completed, duration 0.000 seconds ************* Waiting for Debugger Extensions Gallery to Initialize ************** >>>>>>>>>>>>> Waiting for Debugger Extensions Gallery to Initialize completed, duration 0.015 seconds ----> Repository : UserExtensions, Enabled: true, Packages count: 0 ----> Repository : LocalInstalled, Enabled: true, Packages count: 41 Microsoft (R) Windows Debugger Version 10.0.27553.1004 AMD64 Copyright (c) Microsoft Corporation. All rights reserved. Loading Dump File [D:\tmp\firebird.exe.5968.dmp] User Mini Dump File with Full Memory: Only application data is available ************* Path validation summary ************** Response Time (ms) Location Deferred srv* Symbol search path is: srv* Executable search path is: Windows 10 Version 17763 MP (16 procs) Free x64 Product: Server, suite: TerminalServer SingleUserTS Edition build lab: 17763.1.amd64fre.rs5_release.180914-1434 Debug session time: Tue Apr 2 16:34:52.000 2024 (UTC + 2:00) System Uptime: 41 days 17:39:48.307 Process Uptime: 0 days 5:32:06.000 ................................................................ ........ Loading unloaded module list ..... This dump file has an exception of interest stored in it. The stored exception information can be accessed via .ecxr. (1750.1c08): Security check failure or stack buffer overrun - code c0000409 (first/second chance not available) Subcode: 0x7 FAST_FAIL_FATAL_APP_EXIT For analysis of this file, run !analyze -v ucrtbase!abort+0x4e: 00007ffe`cd6ce77e cd29 int 29h 0:014> !analyze -v ******************************************************************************* * * * Exception Analysis * * * ******************************************************************************* *** WARNING: Check Image - Checksum mismatch - Dump: 0x10252d, File: 0x10606e - C:\ProgramData\Dbg\sym\ucrtbase.dll\48AC8393fa000\ucrtbase.dll *** WARNING: Unable to verify checksum for fbclient.dll *** WARNING: Unable to verify checksum for engine13.dll KEY_VALUES_STRING: 1 Key : Analysis.CPU.mSec Value: 358 Key : Analysis.Elapsed.mSec Value: 3372 Key : Analysis.IO.Other.Mb Value: 1 Key : Analysis.IO.Read.Mb Value: 0 Key : Analysis.IO.Write.Mb Value: 2 Key : Analysis.Init.CPU.mSec Value: 61 Key : Analysis.Init.Elapsed.mSec Value: 11831 Key : Analysis.Memory.CommitPeak.Mb Value: 98 Key : FailFast.Name Value: FATAL_APP_EXIT Key : FailFast.Type Value: 7 Key : Failure.Bucket Value: FAIL_FAST_FATAL_APP_EXIT_c0000409_ucrtbase.dll!abort Key : Failure.Hash Value: {e31753ac-c98a-8055-3663-47e707543d20} Key : Timeline.OS.Boot.DeltaSec Value: 3605988 Key : Timeline.Process.Start.DeltaSec Value: 19926 Key : WER.OS.Branch Value: rs5_release Key : WER.OS.Version Value: 10.0.17763.1 Key : WER.Process.Version Value: 4.0.4.3010 FILE_IN_CAB: firebird.exe.5968.dmp NTGLOBALFLAG: 0 APPLICATION_VERIFIER_FLAGS: 0 CONTEXT: 00007ffed0b388d1 -- (.cxr 0x7ffed0b388d1) rax=480000021e850f0f rbx=8341d23300000086 rcx=894800000098838b rdx=850f00007d830845 rsi=0327840ff8b54148 rdi=483cfe24068a0000 rip=0ffda817418d0000 rsp=000003e6830f02fd rbp=3e80d68b48ff3345 r8=0a8a00000341840f r9=0f583cc52241c18a r10=4cc18a0000052884 r11=0f403cf02401428d r12=f2f9800000022484 r13=418d000a2c5a840f r14=000282860f013c3e r15=0565840ff3f98000 iopl=2 nv dn di ng nz na pe cy cs=2e79 ss=1065 ds=000a es=8b44 fs=0870 gs=8b4c efl=00006583 0ffda817`418d0000 ?? ??? Resetting default scope EXCEPTION_RECORD: 00000000703eccc0 -- (.exr 0x703eccc0) ExceptionAddress: 00001fa00010004f ExceptionCode: 69727720 ExceptionFlags: 6e657474 NumberParameters: 2818099 Parameter[0]: 00000202002b002b Parameter[1]: 36313a6b6162670a Parameter[2]: 00001fa00010004f Parameter[3]: 0053002b002b0033 Parameter[4]: 00000206002b002b Parameter[5]: 3536313a6b616267 Parameter[6]: 6572203030303031 Parameter[7]: 7277207364726f63 Parameter[8]: 670a0d6e65747469 Parameter[9]: 323536313a6b6162 Parameter[10]: 00007ffec1a74528 Parameter[11]: 00000000703edaa0 Parameter[12]: 00000000703edbe0 Parameter[13]: 0000000000000001 Parameter[14]: 00007ffebc12af48 PROCESS_NAME: firebird.exe ERROR_CODE: (NTSTATUS) 0xc0000409 - Das System hat in dieser Anwendung den berlauf eines stapelbasierten Puffers ermittelt. Dieser berlauf k nnte einem b sartigen Benutzer erm glichen, die Steuerung der Anwendung zu bernehmen. EXCEPTION_CODE_STR: c0000409 EXCEPTION_PARAMETER1: 0000000000000007 IP_ON_HEAP: 0ffda817418d0000 The fault address in not in any loaded module, please check your build's rebase log at \bin\build_logs\timebuild\ntrebase.log for module which may contain the address if it were loaded. UNALIGNED_STACK_POINTER: 000003e6830f02fd FAULTING_THREAD: 00001c08 STACK_TEXT: 00000000`703eb150 00007ffe`c188512e : 00000000`00000003 00009a81`00000003 ffffffff`fffffffe 00007ffe`c18f3cd1 : ucrtbase!abort+0x4e 00000000`703eb180 00007ffe`c18f0b8c : 00000000`703eb230 00000000`00000000 00000000`001de280 00000000`00000000 : VCRUNTIME140!_purecall+0x2e 00000000`703eb1b0 00007ffe`c1909259 : 00000000`463f2b30 00000000`463f2b38 00000000`00000000 00000000`703eb2a8 : fbclient!isc_version+0x118c 00000000`703eb200 00007ffe`c1905d14 : 00000000`463f2b38 00000000`703eb328 00000000`703edfb8 00000000`00000000 : fbclient!isc_version+0x19859 00000000`703eb300 00007ffe`bbeefcf2 : 00006d5d`94d032b0 00000000`ffffffff 00007ffe`bbb30000 00007ffe`c1884be6 : fbclient!isc_version+0x16314 00000000`703eb360 00007ffe`bbde10c9 : 00000000`00000000 00000000`703edef8 00007ffe`bc03bfa8 00007ffe`bc03bfa8 : engine13!firebird_plugin+0x180b92 00000000`703eb3a0 00007ffe`c1881030 : 00007ffe`bbfb2a40 00007ffe`00000103 00000000`703edeb0 00007ffe`d0b7e9c0 : engine13!firebird_plugin+0x71f69 00000000`703eb3e0 00007ffe`c188358f : 00007ffe`bbfb2a40 00000000`703eb4c8 00007ffe`00000103 00007ffe`c188bcd2 : VCRUNTIME140!_CallSettingFrame+0x20 00000000`703eb410 00007ffe`c188bc94 : 00000000`703eb4c8 00000000`703eb4c8 00007ffe`bc03bfa8 00000000`ffffffff : VCRUNTIME140!__FrameHandler3::FrameUnwindToState+0x11b 00000000`703eb480 00007ffe`c1882fd9 : 00007ffe`bc03bfa8 00000000`703ebb80 00006d5d`94d04040 00000000`703edeb0 : VCRUNTIME140!__FrameHandler3::FrameUnwindToEmptyState+0x54 00000000`703eb4b0 00007ffe`c188c15d : 00000000`703ebb80 00000000`703ebd00 00000000`703ebd00 00000000`703eb600 : VCRUNTIME140!__InternalCxxFrameHandler<__FrameHandler3>+0x111 00000000`703eb510 00007ffe`bbf6c6fc : 00000000`703edeb0 00007ffe`bc10b604 00000000`703edeb0 00007ffe`00000000 : VCRUNTIME140!__CxxFrameHandler3+0x6d 00000000`703eb560 00007ffe`d0bd4caf : 00007ffe`bc2cec00 00000000`703ebb00 00000000`703eeb20 00000000`00000000 : engine13!firebird_plugin+0x1fd59c 00000000`703eb590 00007ffe`d0b3610c : 00000000`703ebb00 00007ffe`bc2cec40 00007ffe`bc2ce5d4 00000000`703f0000 : ntdll!RtlpExecuteHandlerForUnwind+0xf 00000000`703eb5c0 00007ffe`c188bfd5 : 00007ffe`bc10a060 00007ffe`c188bfd5 00007ffe`c1a505f8 ffffffff`fffffffe : ntdll!RtlUnwindEx+0x51c 00000000`703ebcd0 00007ffe`c18826d4 : 00007ffe`bc10a060 00000000`703ec328 00000000`703edae0 00000000`0000000d : VCRUNTIME140!__FrameHandler3::UnwindNestedFrames+0x109 00000000`703ec290 00007ffe`c18829cb : 00000000`703edae0 00000000`703eeb20 00000000`703ec3f9 00007ffe`bc039150 : VCRUNTIME140!CatchIt<__FrameHandler3>+0xb4 00000000`703ec320 00007ffe`c18830df : 00007ffe`bc039150 00000000`703ec4f0 00000000`703eccc0 00000000`703ec500 : VCRUNTIME140!FindHandler<__FrameHandler3>+0x2db 00000000`703ec440 00007ffe`c188c15d : 00000000`703ecb00 00000000`703edae0 00000000`703edae0 00000000`703eccc0 : VCRUNTIME140!__InternalCxxFrameHandler<__FrameHandler3>+0x217 00000000`703ec4a0 00007ffe`bbf6c6fc : 00000000`703eeb20 00007ffe`bc109fac 00000000`703eeb20 00000000`00000000 : VCRUNTIME140!__CxxFrameHandler3+0x6d 00000000`703ec4f0 00007ffe`d0bd4c2f : 00000000`00000000 00000000`703eca90 00000000`00000001 00007ffe`bbb30000 : engine13!firebird_plugin+0x1fd59c 00000000`703ec520 00007ffe`d0b34cef : 00000000`703eca90 00000000`00000001 00000000`00000000 00007ffe`bbb30000 : ntdll!RtlpExecuteHandlerForException+0xf 00000000`703ec550 00007ffe`d0b38ae6 : 00000000`703eccc0 00007ffe`d0b388d1 00000000`703eccc0 00000000`00000000 : ntdll!RtlDispatchException+0x40f 00000000`703ecc80 00007ffe`cd8149b9 : 00000000`22163cd0 00007ffe`bc12af48 00000000`00000000 00000000`703edc48 : ntdll!RtlRaiseException+0x316 00000000`703edac0 00007ffe`c188485d : 00000000`0000000f 00000000`00000001 00000000`00000001 00007ffe`bbea57c9 : KERNELBASE!RaiseException+0x69 00000000`703edba0 00007ffe`bbe64853 : 00007ffe`bbb30000 00000000`703eef90 00000000`00000053 00000000`00000001 : VCRUNTIME140!_CxxThrowException+0xad 00000000`703edc10 00007ffe`bbe64674 : 00000000`703eef90 00007ffe`bc039258 00000000`00000000 00007ffe`bbea0b61 : engine13!firebird_plugin+0xf56f3 00000000`703edc40 00007ffe`bbe6473f : 00000000`703eef90 00000000`00000016 00000000`00000016 00007ffe`bc039964 : engine13!firebird_plugin+0xf5514 00000000`703edd70 00007ffe`bbe72439 : 00000000`703eef90 00000000`703eef90 00000000`703eef90 00000000`703edfb0 : engine13!firebird_plugin+0xf55df 00000000`703edeb0 00007ffe`bbe7309d : 00000000`703ee750 00000000`2b720e50 00000000`2b72ec80 00000000`2b89d670 : engine13!firebird_plugin+0x1032d9 00000000`703ee650 00007ffe`bbe70065 : 00000000`703eef90 00000000`703eef90 00000000`3488c3b8 00000000`2b720e50 : engine13!firebird_plugin+0x103f3d 00000000`703ee8d0 00007ffe`bbe68065 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : engine13!firebird_plugin+0x100f05 00000000`703eeb20 00007ffe`bbe6488c : 00000000`00000000 00000000`35fb1830 00000000`02098fa0 00000000`00000000 : engine13!firebird_plugin+0xf8f05 00000000`703efd50 00007ffe`bbdf8dd1 : 00000000`00000001 00007ffe`bbe9e99d 00000000`00000001 00000000`000695f0 : engine13!firebird_plugin+0xf572c 00000000`703efe60 00007ffe`bbeb3da5 : 00000000`22163b40 00000000`35fb1830 00000000`22163b40 00000000`00000000 : engine13!firebird_plugin+0x89c71 00000000`703efec0 00007ffe`cd68268a : 00000000`4ef86ef0 00000000`2d564f20 00000000`00000000 00000000`00000000 : engine13!firebird_plugin+0x144c45 00000000`703eff00 00007ffe`d04d7ac4 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ucrtbase!thread_start+0x3a 00000000`703eff30 00007ffe`d0b8a4e1 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0x14 00000000`703eff60 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x21 SYMBOL_NAME: ucrtbase!abort+4e MODULE_NAME: ucrtbase IMAGE_NAME: ucrtbase.dll STACK_COMMAND: ~14s ; .cxr ; kb FAILURE_BUCKET_ID: FAIL_FAST_FATAL_APP_EXIT_c0000409_ucrtbase.dll!abort OS_VERSION: 10.0.17763.1 BUILDLAB_STR: rs5_release OSPLATFORM_TYPE: x64 OSNAME: Windows 10 IMAGE_VERSION: 10.0.17763.1490 FAILURE_ID_HASH: {e31753ac-c98a-8055-3663-47e707543d20} Followup: MachineOwner ---------

[hvlad]

Moved crash analysis under the spoiler.

[hvlad]

Coud you provide a crash dump (D:\tmp\firebird.exe.5968.dmp), please ?

[NilsBoedeker]

Hi

as RAR File it have a size of 1.6 GB (uncompressed 20 GB). I can't upload it hier

I send it bei WE Transfer to hvlad@users.sourceforge.net. Please confirm if you can load it

Nils

[hvlad]

Sure, downloading. Thanks

[NilsBoedeker]

If you need the database or the installation copy of firebird, please ask

[NilsBoedeker]

I an second crash dump from today (3.5 GB), same Error-Message. If you need this crash dump also please give me a message

[hvlad]

Yes, let me look at it too.

[NilsBoedeker]

I send:

a) the second dump b) the installation dir of firebird

in 2 splitted rar files via WE transfer.

Do you also need the database?

Nils

[hvlad]

Both dumps shows almost identical problem with blobs in fbclient.dll. It is not clear what exactly happens, unfortunately.

Do you also need the database?

Yes, please, if it helps to reproduse the issue.

[NilsBoedeker]

On the way... 2 splitted files via wetransfer.

Sometimes I got on client site the error:

invalid BLOB handle

before crash.

I check the database for errors, but there a no errors:

[hvlad]

Downloading, thanks. Is it enough to backup database to get the crash ? As for "invalid BLOB handle" - it is definitely related but should not crash the Firebird.

[NilsBoedeker]

Is it enough to backup database to get the crash ?

No, the error don‘t happed everytime on the live system and never on my test system.

It only happens in multi session use and more often when heavy losd is on the server.

It also happend without running backup, but more often during backup because this means more load.

[hvlad]

Did backup many times in a different ways, no luck - it doesn't crashes.

[NilsBoedeker]

This error only occurs under load on our live system.

The database is 24/7 in use and has permanent 125 and 200 connections and 300 and 400 active statements.

On average, fbserver.exe crashes once every 24 hours. often during backup process (i assume more load).

This is very difficult to simulate. Have you any idea how we can log this crash better?

[aafemt]

Perhaps this link can help: https://ib-aid.com/articles/how-to-configure-firebird-dump-collection-at-windows/

[hvlad]

@aafemt: @NilsBoedeker already provides me with dumps, why do you post this ?

[aafemt]

I'm sorry, I missed that you already have the dump when read that you are unable to reproduce the issue using backup.