search

FirebirdSQL / firebird

Firebird server, client and tools
https://www.firebirdsql.org/
1.24k stars 212 forks source link

FB 2.5.8 hangs [CORE5958] #6212

Open firebird-automations opened 5 years ago

firebird-automations commented 5 years ago

Submitted by: Slavomir Skopalik (skopaliks)

Firebird hangs (stop responding to any client), utilize one CPU core. Dump file created, ask me privately .

Call stack of thread that consume one core: ntoskrnl.exe!IoAcquireRemoveLockEx+0xe7 ntoskrnl.exe!memset+0x22a ntoskrnl.exe!KeWaitForMutexObject+0x2cb ntoskrnl.exe!KeDetachProcess+0x1225 ntoskrnl.exe!PsReturnProcessNonPagedPoolQuota+0x3b3 ntoskrnl.exe!CcSetDirtyPinnedData+0x433 fb_inet_server.exe+0x239a10 fb_inet_server.exe+0x1c0e1e fb_inet_server.exe+0x1c2105 fb_inet_server.exe+0x1c2493 fb_inet_server.exe+0x1ba506 fb_inet_server.exe+0xbcb88 fb_inet_server.exe+0xbeeb6 fb_inet_server.exe+0xc1d87 fb_inet_server.exe+0xc274e fb_inet_server.exe+0x1d4cd2 fb_inet_server.exe+0x1d5088 fb_inet_server.exe+0x1d5c2f fb_inet_server.exe+0xd14dc fb_inet_server.exe+0xd3119 fb_inet_server.exe+0xd31f3 fb_inet_server.exe+0xd0555 fb_inet_server.exe+0x5a889 fb_inet_server.exe+0x250896 fb_inet_server.exe+0x4faa5 fb_inet_server.exe+0x20f5e fb_inet_server.exe+0x356f33 fb_inet_server.exe+0x35d6ed fb_inet_server.exe+0x35df11 fb_inet_server.exe+0x7498d MSVCR80.dll!endthreadex+0x47 MSVCR80.dll!endthreadex+0x104 kernel32.dll!BaseThreadInitThunk+0xd ntdll.dll!RtlUserThreadStart+0x21

Commits: FirebirdSQL/firebird@cd8ef85bf9d8b2ef611fbb8374fb8c1240ef092a

firebird-automations commented 5 years ago

Commented by: @hvlad

Send a link to me, please

firebird-automations commented 5 years ago
Modified by: @hvlad assignee: Vlad Khorsun \[ hvlad \]
firebird-automations commented 5 years ago

Commented by: @hvlad

The problem is that one attachment works hard and not responds to the signals. The hard work is garbage collection of index entries (IDX_garbage_collect). What makes it so hard ? The length of backversions chain. It contains 419610 items !

I will improve engine responsiveness in this place but i doubt such database could work with approproate performance.

firebird-automations commented 5 years ago

Commented by: Slavomir Skopalik (skopaliks)

Thanks for analysis! I will find who doing this.

Some related information for server improvement. In past, I see similar problem on other customer and during clean up FB allocates really huge amount of RAM (>30GB).

In this case, server has only 10GB RAM and FB process takes \~12GB. Can you check RAM allocation during this procedure?

firebird-automations commented 5 years ago

Commented by: Slavomir Skopalik (skopaliks)

Vlad, one question, the 419610 items is for one record (max versions) or all records together (total versions)?

gstat for worst table at this moment: Primary pointer page: 224, Index root page: 225 Average record length: 195.68, total records: 321882 Average version length: 18.97, total versions: 5764, max versions: 639 Data pages: 5171, data page slots: 5187, average fill: 84%

firebird-automations commented 5 years ago

Commented by: @hvlad

Backversions chain i see in dump is for one record. Record versions from this chain is cleaned up from disk at the moment. This record versions exists in memory only to allow clean up unused index entries.

firebird-automations commented 5 years ago

Commented by: @hvlad

As for the memory usage: the record length is 16 724 bytes, 16 724 * 419 610 = 7 017 557 640 or 6.5GB. Attachment memory usage is 7 066 709 360 bytes, which is close to the value above.

firebird-automations commented 5 years ago

Commented by: @hvlad

Fix is committed into v2.5.9

firebird-automations commented 5 years ago

Commented by: @hvlad

Could you comment if it helps ?

firebird-automations commented 5 years ago

Commented by: Slavomir Skopalik (skopaliks)

I will, however it is not easy to reproduce same situation.