the server crash very often: The Firebird Server - DefaultInstance service terminated unexpectedly. [CORE3213]

[firebird-automations]

Submitted by: vander clock stephane (arkadia)

Is related to CORE3206

Attachments: fbserver.exe.15660.dmp

Votes: 1

hello,

now that i check the windows log i discover that our fb server crash very often (1 times / 2 days in average): The Firebird Server - DefaultInstance service terminated unexpectedly. It has done this 1 time(s). The following corrective action will be taken in 0 milliseconds: Restart the service.

hard to know why... no firebird.log file are even created !

The server is very intensive, i remembered that the last time it's crash, it's was when i do myself an heavy update query (using the ISQL). but redoing the query just after the crash not make the server crash again ...

Actually i set up the cache memory to use 85% of all available memory (so 28GB of 32GB are used in the task manager).

any idea how to debug it more deeply ?

[firebird-automations]

Commented by: Sean Leyne (seanleyne)

This case appears to be no different that the previous case you opened.

It appears that you weren't happy with the responses you got to that case, and have opened this case to try and restate you point.

[firebird-automations]

Modified by: Sean Leyne (seanleyne)

Link: This issue is related to [CORE3206](https://github.com/FirebirdSQL/firebird/issues?q=CORE3206+in%3Atitle) \[ [CORE3206](https://github.com/FirebirdSQL/firebird/issues?q=CORE3206+in%3Atitle) \]

[firebird-automations]

Commented by: vander clock stephane (arkadia)

no, the previous was that the crash must be logged in the firebird log ! (it's more clear to have only one log file to check that one log + event windows) to do that it's very simple, when the server start init a flag to 1 and when he stop normally init the flag to 0. if the firebird start and see the flag to 1 then add an entry in the firebird.log As i don't know if this was not already done under firebird, i set up the issue as a "bug"

now this new issus is that i discover that the server crash very often and this is surrely a bug that we must investigate, mostly we must understand why .... and i don't know and to debug this...

[firebird-automations]

Commented by: @dyemanov

http://www.ibphoenix.com/main.nfs?a=ibphoenix&page=ibp_pdb_win32

[firebird-automations]

Commented by: vander clock stephane (arkadia)

thanks dmtry !

also

"You may also download the "pdb" (Debug) Firebird installation and extract its contents to the \bin folder - This will allow Dr.Watson to create a much more useful and detailed log file. This is optional but highly recommended to do. The more detail the better."

where to find the last pdb for fb2.5 ?

[firebird-automations]

Commented by: @dyemanov

At the same place where you have downloaded the FB binaries from. Look for the package with "pdb" suffix.

[firebird-automations]

Commented by: vander clock stephane (arkadia)

hello,

is their any doc that explain better how to do http://www.ibphoenix.com/main.nfs?a=ibphoenix&page=ibp_pdb_win32 on windows 2008 server R2 ? I see that the firebird process still crash often but no bump file are created :(

thanks by advance stéphane

[firebird-automations]

Commented by: vander clock stephane (arkadia)

hello,

Is someone interested by my dump that show the server crash every 2/3 days ? this can be interresting to study ... the server crash when the activity on him become to be too huge. it's not connected to a database corruption because it's happen also just after (few hours for exemple) a fresh backup/restore

regards stéphane

[firebird-automations]

Commented by: @dyemanov

If you have a crash dump, please upload it here or make available for download from somewhere.

[firebird-automations]

Commented by: vander clock stephane (arkadia)

hello dimitry,

but is the dumps can contain some private data (like password or database content) that can make dangerous to put them in public places ?

Also now the server crash not really randomly. it's seam to crash when : 1/ the activity on the server is relativelly important (but it's can be also a consequence of this important activity and not the activity himself, like memory consumption for exemple) 2/ when we do massive update.

yesterday the server crash 3 times when i try to inserts some rows (but nothing special in the sql, simple insert). nothing was written in the firebird.log

Now about the crash dump, i follow these instructions from :

http://www.ibphoenix.com/main.nfs?a=ibphoenix&page=ibp_pdb_win32 that finally point me to : http://msdn.microsoft.com/en-us/library/bb787181%28VS.85%29.aspx

my reg :

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps] "DumpCount"=dword:0000000a "DumpType"=dword:00000002 "DumpFolder"="c:\\CrashDumps"

but no dump are created :( reading carrefully the doc they say the the dumps are created after "a user-mode application crashes"... i don't thing that fb_inet_server is a user mode application :( so probably the probleme is here ?

their is no doc about how to generate dump file for firebird server on windows 2008 system ?

thanks you by advance stephane

[firebird-automations]

Commented by: vander clock stephane (arkadia)

Now i try to use adplus (the debug tool in the windows SDK v7.1) but when i use it i see that the memory used by the firebird process grow all the time still the process began to use 100% of all the cpu activity and close to 4 GO of memory :(

Is their any windows 2008 guy here that can explain how to retrieve the dump file ?

[firebird-automations]

Commented by: @hvlad

Stephane,

seems you already found all the docs how to generate crash dump for Firebird. Yes, fb_inet_server is user-mode application, it is not a device driver, i think ;) So, you already enabled system to collect dump's. Make sure you are not used Guardian. Make sure you read carefully description of DumpFolder value - about ACL's and services default folder.

Unfortunately i have nor Win2008, nor Vista to check this by myself :(

[firebird-automations]

Commented by: Stefano Monterisi (stemon)

Hi, there is a workaround or update on this error? I have the same situation. There is a solution? I have installed 2.5.1 without lucky.... This error affect all release? (Superserver, classic, etc..) Thanks in advance! Stefano

[firebird-automations]

Commented by: @hvlad

Stefano,

do you have crash dump ? Any records in firebird.log ? Why do you think you have the same issue ?

You provided zero bits of info about your "same situation", we can't help.

[firebird-automations]

Commented by: Stefano Monterisi (stemon)

Hi Vlad, thank you; I will try to report you firebird.log or crash dump (I note that is difficult to obtalin) I have windows Server 2008 R2 64 bits, Firebird 2.5.1 64 bits , 32 go of memory Thanks in advance, Stefano

[firebird-automations]

Commented by: Stefano Monterisi (stemon)

Hi Vlad, I have always the same error in log. INET/inet_error: read error errno=10054

Any hint on this? Thanks in advance! Stefano Monterisi

[firebird-automations]

Commented by: @hvlad

Stefano,

no, these errors can't help, unfortunately.

To investigate and fix the issue we need a reproducible test case or, at least, memory dump of crashing process.

There is a lot of articles in MSDN of how to catch process crash and take memory dump of it.

[firebird-automations]

Commented by: Stefano Monterisi (stemon)

Hi Vlad, I have used dump operation from http://msdn.microsoft.com/en-us/library/bb787181%28VS.85%29.aspx But dump is not created....the folder is always empty.... There is another system for obtain dump memory on crash? (I am in a danger situation; firebird superclassic is on internet server with a lot of users; you think that using classic server can help me in this situation, while search the bug ?) Thanks in advance!!!! Stefano

[firebird-automations]

Commented by: @hvlad

Are you sure your firebird.log have only INET error's ?

It should contain also records about crashes, if Firebird catched it. In this case process exited correctly and OS doesn't see it as a crash.

If you have recorded fatal exceptions in firebird.log, set in firebird.conf

BugcheckAbort = 1

and restart Firebird service

[firebird-automations]

Commented by: Stefano Monterisi (stemon)

Hi Vlad, on server there are only INET error's in firebird.log ON client instead, there are: INET/inet_error: send errorno= 10054 REMOTE INTERFACE / gds__detach: unsuccessful detach from database. Uncomitted work may be lost

The problem is that the crash is present when several user are connected, and the service restart. If I set BugcheckAbort = 1 then the service remain down, true? So I have to stop users... Thanks in advance...

[firebird-automations]

Commented by: Stefano Monterisi (stemon)

I have superclassic release, there is no guardian in memory? Thanks

[firebird-automations]

Commented by: Stefano Monterisi (stemon)

Hi Vlad, I have a little UDF written in Free Pascale 64bits that create guid strings; Maybe this library? Thanks in advance! Stefano

[firebird-automations]

Commented by: @hvlad

FPC for Win64 is *broken* !!! This is known issue and it was already discussed in fb-devel and FPC tracker. See http://mantis.freepascal.org/bug_view_page.php?bug_id=17360

[firebird-automations]

Commented by: Stefano Monterisi (stemon)

Hi, Vlad I have used FPC 2.4.3 for a little udf... I can try to recompile it with Delphi XE2 64 Bits; Have you Known reported problem with Delphi xe2 ? I hope to solve with UDF recompile... Thanks in advance! Stefano

[firebird-automations]

Commented by: @hvlad

Stefano,

no, i don't know about issues with Delphi x64 UDF's.

Btw, you don't need custom UDF to work with guid's, there is built-in functions to do it.

[firebird-automations]

Commented by: Stefano Monterisi (stemon)

Vlad, I am a Delphi Programmer, and a Firebird novice.... I want to obtain a "{21EC2020-3AEA-1069-A2DD-08002B30309D}" guid string to assign on insert Trigger, and not binary GUID. There is a built-in function to obtain this? What? Thank you!

[firebird-automations]

Commented by: @hvlad

Stefano,

*bug* tracker is not a place for support questions. Please, use appropriate support list:

http://www.firebirdsql.org/en/support/

and use documentation:

http://www.firebirdsql.org/en/documentation/ http://www.firebirdsql.org/file/documentation/reference_manuals/reference_material/html/langrefupd25-intfunc.html http://www.firebirdsql.org/file/documentation/reference_manuals/reference_material/html/langrefupd25-intfunc-gen_uuid.html http://www.firebirdsql.org/file/documentation/reference_manuals/reference_material/html/langrefupd25-intfunc-uuid_to_char.html

[firebird-automations]

Commented by: Stefano Monterisi (stemon)

Ok, Vlad, thank for support; I contact you if the bug in not into UDF. Thanks a lot. Stefano

[firebird-automations]

Commented by: Stefano Monterisi (stemon)

Hi Vlad, the problem is in UDF. Using native gen_uuid the service is online from 24 hours without crashing... Thank you for help! Stefano Monterisi

[firebird-automations]

Commented by: Andrew Ayre (communicare)

Hi, We seem to have the same problem. We have several sites where the Firebird server is terminating approximately 4 times a week, with no particular pattern.

The Windows logs show: 'The Firebird Server - DefaultInstance service terminated unexpectedly.' but there is nothing in the firebird log.

We are running Firebird 2.5.1 SuperServer architecture on Server 2008 R2, Serivice Pack 1, 16GB RAM, 64bit. Approximate DB size is 25GB.

In 32 bit environments we do not see have the problem.

Does anyone have any idea why this is happening? Is it related to UDFs? Does the Firebird architecture make any difference?

Is it possible for me to obtain an more detailed logs?

Thanks, Tom

[firebird-automations]

Commented by: @hvlad

Tom,

Try to use recently released 2.5.2 Read messages above and make your system to produce memory dump on process crash

[firebird-automations]

Commented by: Andrew Ayre (communicare)

Thanks Vlad, we've now switched to 2.5.2 on one site, however there's no change in the behaviour. Several terminations in the past few days. Unfortunately nothing was logged to the crash dump. I've followed the instructions at http://msdn.microsoft.com/en-us/library/bb787181%28VS.85%29.aspx.

The earlier link, http://www.ibphoenix.com/main.nfs?a=ibphoenix&page=ibp_pdb_win32 no longer works.

Our next step will be to change to Classic to see if this makes any difference.

Thanks, Tom

[firebird-automations]

Commented by: Andrew Ayre (communicare)

Hi Vlad,

We're still having issues Firebird (SuperServer configuration) crashing, but have not been able to generate any memory dumps. Do you have any more instructions on how to do this? The previously mentioned links are no longer available.

Thanks, Tom

[firebird-automations]

Commented by: @dyemanov

http://www.ibphoenix.com/resources/documents/development/doc_35

[firebird-automations]

Commented by: Andrew Ayre (communicare)

Thanks Dmitry,

However after following all the instructions, there's still no crash dump.

Has anyone successfully generated crash dumps from Firebird with Server 2008 or Windows 7?

Thanks, Tom

[firebird-automations]

Commented by: Andrew Ayre (communicare)

Hi Vlad,

I've finally managed to get some crash dumps, but they are over 10MB so too big to attach to the tracker. Can I send them to you directly?

Thanks, Tom

[firebird-automations]

Commented by: @hvlad

Tom,

could you upload it somewhere ? For example at http://ge.tt If you need to secure it - create archive (rar, zip or 7z) with password protection and send pwd to me (and url, of course)

Thanks, Vlad

[firebird-automations]

Commented by: Andrew Ayre (communicare)

Hi Vlad,

I've changed the settings to produce a minidump instead of a full dump.

There was a crash yesterday afternoon and I've attached the minidump for this. If it's not sufficient I'll be happy to upload the full dump somewhere as you suggested.

Many thanks, Tom

[firebird-automations]

Modified by: Andrew Ayre (communicare)

Attachment: fbserver\.exe\.15660\.dmp \[ 12286 \]

[firebird-automations]

Commented by: @hvlad

Tom,

i can see very fe bits of information at minidump. All i can say right now is that issue seems to be with your custom UDF library CcUDF, as WinDbgBG shows following:

⁠ Call Site

00 ntdll!RtlpWaitOnCriticalSection+0xb4 01 ntdll!RtlEnterCriticalSection+0xd1 02 CcUDF+0xd139 03 0x4b4f460 04 0x1ce830 05 0x20

[firebird-automations]

Commented by: Andrew Ayre (communicare)

Thanks Vlad,

A full dump is at http://ge.tt/9ry4NPY/v/0?c.

Does this give you any more information about the cause?

Cheers, Tom

[firebird-automations]

Commented by: @hvlad

Tom,

I can add only ordinal number of failed entrypoint at CcUDF:

00 ntdll!RtlpWaitOnCriticalSection+0xb4 01 ntdll!RtlEnterCriticalSection+0xd1 02 CcUDF!Ordinal43+0xd139 03 0x948f020 04 0x1033d0 05 0x20

i.e. it is entrypoint with ordinal number 43.

PS next time compress dump, please !

[firebird-automations]

Commented by: Andrew Ayre (communicare)

Great - I'll look into that entry point.

Thanks!

[dyemanov]

Closed as outdated (and possibly resolved).