search

FirebirdSQL / firebird

Firebird server, client and tools
https://www.firebirdsql.org/
1.24k stars 215 forks source link

New connection to server causes CPU core to run at 100% #6905

Open ghost opened 3 years ago

ghost commented 3 years ago

This has occurred intermittently on several client sites and today was the first time I managed to get online to one and see what was happening.

Server was running Firebird 3.0.7. I do not know reproduction steps yet. The server was upgraded from 3.0.3 to 3.0.7 on 2021-11-11 and I do not know how many times they've experienced this high cpu usage scenario since then, today was first time in a week. I suspect the problem didn't happen under 3.0.3 as I don't think we had been told about it prior to that.

Using procexp64.exe I can see 2 threads running high cpu and both show the same stack dump which is as follows

ntoskrnl.exe!KeSynchronizeExecution+0x5a36
ntoskrnl.exe!KeWaitForMutexObject+0x1c27
ntoskrnl.exe!KeWaitForMutexObject+0x1799
ntoskrnl.exe!KeWaitForMutexObject+0x520
ntoskrnl.exe!KeQuerySystemTimePrecise+0xe76
ntoskrnl.exe!KeWaitForMutexObject+0x3ede
ntoskrnl.exe!KeSynchronizeExecution+0x306b
OdbcFb.dll!SQLColumnPrivilegesW+0x45f0
OdbcFb.dll!SQLColumnPrivilegesW+0x1b614
OdbcFb.dll!SQLColumnPrivilegesW+0x69a09
OdbcFb.dll!SQLColumnPrivilegesW+0x648d1
OdbcFb.dll!SQLColumnPrivilegesW+0x5c5e0
ODBC32.dll!SQLAllocHandle+0x66a
ODBC32.dll!SQLDriverConnectW+0xbea
msdasql.dll!DllCanUnloadNow+0x8947
msdasql.dll!DllUnregisterServer+0x5fd1
oledb32.dll!DllGetClassObject+0x190a
oledb32.dll+0x55e6
oledb32.dll+0x4f82
comsvcs.dll!DispManGetContext+0xb4a
comsvcs.dll!DispManGetContext+0x7b3
oledb32.dll!DllGetClassObject+0xa451
oledb32.dll!DllMain+0xa44c
oledb32.dll!DllGetClassObject+0x6b4e
msado15.dll+0x238c
msado15.dll+0x21e8
msado15.dll+0x1ed2
PC4LicSvc.exe!TADOConnection.DoConnect+0x92
PC4LicSvc.exe!TCustomConnection.SetConnected+0x6a
PC4LicSvc.exe!TCustomConnection.Open+0x16
PC4LicSvc.exe!CDatabase.Open+0x13e
PC4LicSvc.exe!CDataStoreLink.GetDatabase+0x89
PC4LicSvc.exe!CDataStoreLink.Query+0x2f
PC4LicSvc.exe!CLicSrvRequestProcessor.processListUsers+0x28d
PC4LicSvc.exe!CLicSrvRequestProcessor.processClientRequest+0x107d
PC4LicSvc.exe!TLicServerSocketClientThread.processSocketConnection+0x6fd
PC4LicSvc.exe!TLicServerSocketClientThread.processItem+0x40
PC4LicSvc.exe!CMultiQueueBufferedProcessor<IValueHolder<NativeUInt>>.threadExecute+0x189
PC4LicSvc.exe!CQueueProcessorThread<IValueHolder<NativeUInt>>.Execute+0x84
PC4LicSvc.exe!ThreadProc+0x43
PC4LicSvc.exe!ThreadWrapper+0x3d
PC4LicSvc.exe!ThreadExceptFrame+0x4f
KERNEL32.DLL!BaseThreadInitThunk+0x14
ntdll.dll!RtlUserThreadStart+0x21
aafemt commented 3 years ago

I see no firebird.exe anywhere in stack and by offsets I'm jugging that symbols in OdbcFb.dll were resolved wrongly. Besides the stack ends up in "Wait" functions that hardly can consume CPU.

ghost commented 3 years ago

This is the first time I've used procexp64.exe to resolve symbols on a stack dump and pdb was converted from a delphi map file so it's possible they aren't matching up but they look like they resolved correctly for our code (PC4LicSvc.exe) I also would have thought the wait would have been low cpu usage but saw that issue 5485 had a similar issue with cpu usage

mrotteveel commented 3 years ago

What is taking much CPU, your application or Firebird server? If Firebird is taking a lot of CPU, then you have dumped the stack of the wrong process (i.e. your application instead of Firebird).

ghost commented 3 years ago

It's showing in our app so maybe I need to log this with fbodbc project (assuming stack dump is correct). Hopefully I'll also get a chance again soon to get more stack dumps to see if they all show exactly the same thing.

ghost commented 2 years ago

If anyone is interested. After downloading the source for fbodbc I'm hoping that the problem is the locking in the FB ODBC connect and that it is safe to simply add SafeConnect=N to the connection string to disable the critical section locking it's doing. We never share connections per thread so hopefully that'll sort the problem out.