Fbembed.dll crash in garbage_collector [CORE3520]

[firebird-automations]

Submitted by: Antti Nivala (nivant)

Attachments: Dump12572.mdmp Dump10482.mdmp Dump11640.mdmp Dump11841.mdmp Call stack.txt Dump12236.mdmp

We have received several crash reports from our customers. Our investigation of the crash dump files shows that the crash is occurring in Firebird code in fbembed.dll (which in our case has been renamed to fbclient.dll). The crash location is in garbage_collector:

fbclient.dll!garbage_collector(void * arg=0x0003d978) Line 4117 C++ fbclient.dll!`anonymous namespace'::threadStart(void * arg=0x00ec09e0) Line 283 C++ msvcr80.dll!_callthreadstartex() Line 348 + 0x6 bytes C msvcr80.dll!_threadstartex(void * ptd=0x0229a430) Line 326 + 0x5 bytes C kernel32.dll!@BaseThreadInitThunk@12() + 0x12 bytes
ntdll.dll!___RtlUserThreadStart@8() + 0x27 bytes
ntdll.dll!__RtlUserThreadStart@8() + 0x1b bytes

I am attaching the crash dump files to this issue (Windows platform).

[firebird-automations]

Modified by: Antti Nivala (nivant)

Attachment: Dump12572\.mdmp \[ 11962 \] Attachment: Dump10482\.mdmp \[ 11963 \] Attachment: Dump11640\.mdmp \[ 11964 \]

[firebird-automations]

Modified by: Antti Nivala (nivant)

Attachment: Dump11841\.mdmp \[ 11965 \] Attachment: Call stack\.txt \[ 11966 \]

[firebird-automations]

Commented by: @hvlad

Are you load\unload fbclient.dll dynamically (using LoadLibrary) ? Could you confirm that crash happens when fbclient.dll is unloaded or when last attachment is closed ? Could you try more recent version 2.1.4 ?

[firebird-automations]

Commented by: Antti Nivala (nivant)

> Are you load\unload fbclient.dll dynamically (using LoadLibrary) ?

We load fbclient.dll (which is really fbembed.dll) dynamically using LoadLibrary, but we then hold that reference for the lifetime of the process. At the time of these crashes, fbclient.dll is not being unloaded / has not been unloaded.

> Could you confirm that crash happens when fbclient.dll is unloaded or when last attachment is closed ?

No, fbclient.dll has not being unloaded. These crashes have occurred at a time when there is still other activity. It is possible that there are no active attachments to the database at the time of the crash, but fbclient.dll is not being unloaded because we still hold the DLL handle that we got from the LoadLibrary call.

> Could you try more recent version 2.1.4 ?

Unfortunately not. These crash dump files are from our clients. We haven't been able to reproduce the crash on our machines. The production version of our software current ships with Firebird Embedded 2.1.2. We will start shipping a version with 2.1.4 in September 2011, and then we will find out in 6 to 12 months if this problems still occurs in 2.1.4...

[firebird-automations]

Commented by: Antti Nivala (nivant)

I'm attaching another crash dump file to this issue because this crash dump seems to have a similar call stack. It is not identical, though, so this might be a crash for some other reason as well. This dump file is Dump12236.mdmp and the call stack from the crash is as follows:

Unhandled exception at 0x10063f28 (fbclient.dll) in minidump.mdmp: 0xC0000005: Access violation reading location 0x0000000000000010.

fbclient.dll!Firebird::GenericMap<Firebird::Pair<Firebird::Full<Firebird::StringBase<Firebird::StringComparator>...::clear() Line 103 + 0x4 bytes fbclient.dll!Firebird::GenericMap<Firebird::Pair<Firebird::Full<Firebird::StringBase<Firebird::StringComparator... Line 70 + 0x5 bytes fbclient.dll!Jrd::Attachment::\~Attachment() Line 6214 + 0x7b bytes C++ fbclient.dll!garbage_collector(void * arg=0x0000000000000000) Line 4149 + 0x11 bytes C++ fbclient.dll!`anonymous namespace'::threadStart(void * arg=0x00000000017235b0) Line 282 + 0x5 bytes C++ msvcr80.dll!_callthreadstartex() Line 348 + 0xd bytes C msvcr80.dll!_threadstartex(void * ptd=0x0000000000000000) Line 326 + 0x5 bytes C kernel32.dll!BaseThreadInitThunk() + 0xd bytes
ntdll.dll!RtlUserThreadStart() + 0x21 bytes

[firebird-automations]

Modified by: Antti Nivala (nivant)

Attachment: Dump12236\.mdmp \[ 11967 \]

[firebird-automations]

Commented by: @hvlad

Unfortunately, crash dumps provided are very small and have almost no additional info except of stack traces. Could you produce more detailed dumps ? Full process memory with heap is preferable.

PS If you can't upgrade even to 2.1.4 - how do you expect to fix the issue with 2.1.2 ? :)

[firebird-automations]

Commented by: Antti Nivala (nivant)

We have received the crash dumps from Microsoft's Winqual service. By the way, has the Firebird project registered for Winqual? The project would automatically receive crash dumps from Microsoft, and you could configure the dumps to collect full heap in cases you need it. You're probably familiar with Winqual already, but in case not, I'd be happy to assist you.

Winqual collects minidumps by default. I've now configured the related bucket ID to collect full heap if the crash occurs again. If we receive full dumps in the future, I will add them here. This issue is not reproducable on our machines so we are relying only on the crash reports that we have received via Microsoft's Winqual service. According to Winqual, the crash has been reported 10 to 20 times.

> If you can't upgrade even to 2.1.4 - how do you expect to fix the issue with 2.1.2 ? :)

We can certainly upgrade to 2.1.4, and actually have already upgraded. What I meant is that we don't have access to the customers/machines from which these crash reports have come from so we can't go to them and say "let's try with 2.1.4". But when we start delivering our new version in September 2011, including 2.1.4, version 2.1.4 will spread to our customer base.

My purpose in reporting this crash is simply that it would eventually be fixed (e.g., in 2.1.5, or 2.5.x, or 3.x). When we release a new version, we always try to be up to date with Firebird. Currently we are stuck with the 2.1.x series because 2.5 does not have shared page cache for Firebird Embedded, making Firebird Embedded 2.5 unsuitable for our use.