Slow MGen start

[rualark]

MGen can take up to 10 seconds to start. During these 10 seconds CPU load is 100% (single core load). Windows restart fixes that.

Seems that this problem arises after many MGen starts.

After Windows restart render takes 7-8 seconds

[rualark]

After several hours of tasks (about 1500 tasks), MGen start takes about 3-6 seconds instead of 1.

[rualark]

After reboot:

[rualark]

Idea: https://superuser.com/questions/952141/windows-10-system-process-taking-massive-amounts-of-ram

[rualark]

Example command line:

[rualark]

During run:

[rualark]

[rualark]

After reboot:

[rualark]

After reboot and Reaper load:

[rualark]

After 8 hours of working:

[rualark]

-job=300 configs\GenMP1\sv_test-soundcheck.pl

[rualark]

After several days of working, now starting pause is about 9-11 seconds:

[rualark]

Stack trace during MGen pause: win32u.dll!NtGdiGetPixel+0x14 gdi32full.dll!GetPixel+0xb2 GDI32.dll!GetPixel+0x36 MGen.exe+0x1be07e MGen.exe+0x26d125 MGen.exe+0x21653d MGen.exe+0x1ae3c4 MGen.exe+0x2d53bb MGen.exe+0x1f0d22 MGen.exe+0x202094 MGen.exe+0x1feea0 MGen.exe+0x1981c6 MGen.exe+0x19993a MGen.exe+0x192a59 MGen.exe+0x19350c USER32.dll!CallWindowProcW+0x4dd USER32.dll!CallWindowProcW+0x1cc USER32.dll!GetTopWindow+0x153 ntdll.dll!KiUserCallbackDispatcher+0x24 win32u.dll!NtUserMessageCall+0x14 USER32.dll!SendMessageW+0x279 USER32.dll!SendMessageA+0x55 MGen.exe+0x1c0e96 MGen.exe+0x1cc869 MGen.exe+0x1733b9 MGen.exe+0x17212b MGen.exe+0x197932 MGen.exe+0x19993a MGen.exe+0x192a59 MGen.exe+0x19350c USER32.dll!CallWindowProcW+0x4dd USER32.dll!CallWindowProcW+0x1cc USER32.dll!SetRect+0xbc ntdll.dll!KiUserCallbackDispatcher+0x24 win32u.dll!NtUserCreateWindowEx+0x14 USER32.dll!CreateWindowExW+0xba2 USER32.dll!CreateWindowExW+0x241 USER32.dll!CreateWindowExA+0x82 MGen.exe+0x19bcaf MGen.exe+0x1948dd MGen.exe+0x1a8334 MGen.exe+0x1a8db5 MGen.exe+0x1fed5f MGen.exe+0x325920 MGen.exe+0x22f29a MGen.exe+0x323cac MGen.exe+0x22e176 MGen.exe+0x1a4bb0 MGen.exe+0x1a4867 MGen.exe+0x22ed3d MGen.exe+0x17beaf MGen.exe+0x3e2137 MGen.exe+0x394287 KERNEL32.DLL!BaseThreadInitThunk+0x14 ntdll.dll!RtlUserThreadStart+0x21

Another run: ntoskrnl.exe!KeSynchronizeExecution+0x5f76 ntoskrnl.exe!RtlRbInsertNodeEx+0x55a win32u.dll!NtGdiGetPixel+0x14 gdi32full.dll!GetPixel+0xb2 GDI32.dll!GetPixel+0x36 MGen.exe+0x1be07e MGen.exe+0x26cec8 MGen.exe+0x26d244 MGen.exe+0x240130 MGen.exe+0x241930 MGen.exe+0x1979ac MGen.exe+0x19993a MGen.exe+0x251ac8 MGen.exe+0x192a59 MGen.exe+0x19350c USER32.dll!CallWindowProcW+0x4dd USER32.dll!CallWindowProcW+0x1cc USER32.dll!GetDC+0x8b ntdll.dll!KiUserCallbackDispatcher+0x24 win32u.dll!NtUserCreateWindowEx+0x14 USER32.dll!CreateWindowExW+0xba2 USER32.dll!CreateWindowExW+0x241 USER32.dll!CreateWindowExA+0x82 MGen.exe+0x19bcaf MGen.exe+0x1948dd MGen.exe+0x194797 MGen.exe+0x24fd24 MGen.exe+0x2450de MGen.exe+0x23e910 MGen.exe+0x23e68a MGen.exe+0x17317a MGen.exe+0x1720ac MGen.exe+0x197932 MGen.exe+0x19993a MGen.exe+0x192a59 MGen.exe+0x19350c USER32.dll!CallWindowProcW+0x4dd USER32.dll!CallWindowProcW+0x1cc USER32.dll!SetRect+0xbc ntdll.dll!KiUserCallbackDispatcher+0x24 win32u.dll!NtUserCreateWindowEx+0x14 USER32.dll!CreateWindowExW+0xba2 USER32.dll!CreateWindowExW+0x241 USER32.dll!CreateWindowExA+0x82 MGen.exe+0x19bcaf MGen.exe+0x1948dd MGen.exe+0x1a8334 MGen.exe+0x1a8db5 MGen.exe+0x1fed5f MGen.exe+0x325920 MGen.exe+0x22f29a MGen.exe+0x323cac MGen.exe+0x22e176 MGen.exe+0x1a4bb0 MGen.exe+0x1a4867 MGen.exe+0x22ed3d MGen.exe+0x17beaf MGen.exe+0x3e2137 MGen.exe+0x394287 KERNEL32.DLL!BaseThreadInitThunk+0x14 ntdll.dll!RtlUserThreadStart+0x21

Another run: ntoskrnl.exe!KeSynchronizeExecution+0x5f76 ntoskrnl.exe!KeWaitForSingleObject+0x1070 ntoskrnl.exe!KeWaitForSingleObject+0x9ee ntoskrnl.exe!KeWaitForSingleObject+0x1c9 ntoskrnl.exe!PsGetHostSilo+0x353 ntoskrnl.exe!KeWaitForSingleObject+0x2c10 ntoskrnl.exe!KiCheckForKernelApcDelivery+0x25 ntoskrnl.exe!KeLeaveCriticalRegion+0x37 win32kbase.sys!SURFMEM::bCreateDIB+0xbc7 win32kbase.sys!SURFMEM::bCreateDIB+0x184 win32kfull.sys!NtGdiGetPixel+0x2f3 ntoskrnl.exe!setjmpex+0x8fc3 win32u.dll!NtGdiGetPixel+0x14 gdi32full.dll!GetPixel+0xb2 GDI32.dll!GetPixel+0x36 MGen.exe+0x1be07e MGen.exe+0x26cef8 MGen.exe+0x26d244 MGen.exe+0x240130 MGen.exe+0x241930 MGen.exe+0x1979ac MGen.exe+0x19993a MGen.exe+0x251ac8 MGen.exe+0x192a59 MGen.exe+0x19350c USER32.dll!CallWindowProcW+0x4dd USER32.dll!CallWindowProcW+0x1cc USER32.dll!GetDC+0x8b ntdll.dll!KiUserCallbackDispatcher+0x24 win32u.dll!NtUserCreateWindowEx+0x14 USER32.dll!CreateWindowExW+0xba2 USER32.dll!CreateWindowExW+0x241 USER32.dll!CreateWindowExA+0x82 MGen.exe+0x19bcaf MGen.exe+0x1948dd MGen.exe+0x194797 MGen.exe+0x24fd24 MGen.exe+0x2450de MGen.exe+0x23e910 MGen.exe+0x23e68a MGen.exe+0x17317a MGen.exe+0x1720ac MGen.exe+0x197932 MGen.exe+0x19993a MGen.exe+0x192a59 MGen.exe+0x19350c USER32.dll!CallWindowProcW+0x4dd USER32.dll!CallWindowProcW+0x1cc USER32.dll!SetRect+0xbc ntdll.dll!KiUserCallbackDispatcher+0x24 win32u.dll!NtUserCreateWindowEx+0x14 USER32.dll!CreateWindowExW+0xba2 USER32.dll!CreateWindowExW+0x241 USER32.dll!CreateWindowExA+0x82 MGen.exe+0x19bcaf MGen.exe+0x1948dd MGen.exe+0x1a8334 MGen.exe+0x1a8db5 MGen.exe+0x1fed5f MGen.exe+0x325920 MGen.exe+0x22f29a MGen.exe+0x323cac MGen.exe+0x22e176 MGen.exe+0x1a4bb0 MGen.exe+0x1a4867 MGen.exe+0x22ed3d MGen.exe+0x17beaf MGen.exe+0x3e2137 MGen.exe+0x394287 KERNEL32.DLL!BaseThreadInitThunk+0x14 ntdll.dll!RtlUserThreadStart+0x21

[rualark]

At the same time Apache seems to drain CPU without real load: https://www.youtube.com/watch?v=B7PgTdfA7es&feature=youtu.be

[rualark]

Detected hang position:

[rualark]

[21:10, 14.3.2018] : Ты знаешь, интересное развитие вся эта проблема с CPU имеет. Сейчас у меня задержка перед MGen уже 25 секунд. При этом все виртуалки стали жрать много CPU. Я поизучал что там внутри виртуалок происходит. Apache особо и запросов то не получает, но грузится по полной. Интересно. [21:12, 14.3.2018] : Вот при такой нагрузке он жрет CPU :))) [21:12, 14.3.2018] : Total accesses: 78 - Total Traffic: 1.5 MB .313 requests/sec - 6.0 kB/second - 19.2 kB/request 4 requests currently being processed, 146 idle workers [21:23, 14.3.2018] : Вот так забавно выглядит: https://www.youtube.com/watch?v=B7PgTdfA7es&feature=youtu.be [21:24, 14.3.2018] : Все хорошо, а со второй минуты начинает жарить [21:26, 14.3.2018] : Не пойму на самом деле причину, в этот момент только запрос stems.php отрабатывает, но вроде быстро [21:26, 14.3.2018] : Такое ощущение что эта проблема и MGen чем-то похожи [22:18, 14.3.2018] : Нашел точку в коде, где зависает программа в начале. Это стандартный код Windows: [22:18, 14.3.2018] : // Dispatch commands specified on the command line. Will return FALSE if // app was launched with /RegServer, /Register, /Unregserver or /Unregister. if (!ProcessShellCommand(cmdInfo)) return FALSE; [22:27, 14.3.2018] : Попробовал билды 32-битный и 64-битный - одинаковый результат. Пауза уже 45 секунд

[rualark]

Then suddenly it crashes:

And also BEX crashes

[rualark]

Probably Apache was fixed by fixing infinite loop php script. Cannot fix MGen pause so far.

[rualark]

Function Name	Total CPU (ms)	Total CPU (%)	Module
- MGen.exe (PID: 20608)	19539	100,00%	MGen.exe
! - ntdll.dll!0x007ff87b61ef91	18952	97,00%	ntdll.dll
!! - kernel32.dll!0x007ff878f01fe4	18952	97,00%	kernel32.dll
!!! - __scrt_common_main_seh	18936	96,91%	MGen.exe
!!!! - AfxWinMain	18931	96,89%	MGen.exe
!!!!! - CMGenApp::InitInstance	16022	82,00%	MGen.exe
!!!!!! - CWinApp::ProcessShellCommand	15995	81,86%	MGen.exe
!!!!!!! - CCmdTarget::OnCmdMsg	15995	81,86%	MGen.exe
!!!!!!!! - _AfxDispatchCmdMsg	15995	81,86%	MGen.exe
!!!!!!!!! - CWinApp::OnFileNew	15995	81,86%	MGen.exe
!!!!!!!!!! - CDocManager::OnFileNew	15995	81,86%	MGen.exe
!!!!!!!!!!! - CSingleDocTemplate::OpenDocumentFile	15995	81,86%	MGen.exe
!!!!!!!!!!!! - CDocTemplate::CreateNewFrame	15973	81,75%	MGen.exe
!!!!!!!!!!!!! - CFrameWndEx::LoadFrame	15972	81,74%	MGen.exe
!!!!!!!!!!!!!! - CFrameWnd::LoadFrame	15351	78,57%	MGen.exe
!!!!!!!!!!!!!!! - CFrameWnd::Create	15350	78,56%	MGen.exe
!!!!!!!!!!!!!!!! - CWnd::CreateEx	15350	78,56%	MGen.exe
!!!!!!!!!!!!!!!!! - IsolationAwareCreateWindowExA	15350	78,56%	MGen.exe
!!!!!!!!!!!!!!!!!! - user32.dll!0x007ff87968a2b2	15350	78,56%	user32.dll
!!!!!!!!!!!!!!!!!!! - user32.dll!0x007ff87968d461	15350	78,56%	user32.dll
!!!!!!!!!!!!!!!!!!!! - user32.dll!0x007ff87968ddc2	15350	78,56%	user32.dll
!!!!!!!!!!!!!!!!!!!!! - win32u.dll!0x007ff878a61f04	15350	78,56%	win32u.dll
!!!!!!!!!!!!!!!!!!!!!! - ntdll.dll!0x007ff87b653b14	11546	59,09%	ntdll.dll
!!!!!!!!!!!!!!!!!!!!!!! - user32.dll!0x007ff879693edc	11541	59,07%	user32.dll
!!!!!!!!!!!!!!!!!!!!!!!! - user32.dll!0x007ff87967b54c	11541	59,07%	user32.dll
!!!!!!!!!!!!!!!!!!!!!!!!! - user32.dll!0x007ff87967b85d	11541	59,07%	user32.dll
!!!!!!!!!!!!!!!!!!!!!!!!!! - AfxWndProc	11541	59,07%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!! - AfxCallWndProc	11541	59,07%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!! - CWnd::WindowProc	11541	59,07%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - CWnd::OnWndMsg	11541	59,07%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - CMainFrame::OnCreate	11541	59,07%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - CMainFrame::OnApplicationLook	11529	59,01%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - CMFCVisualManager::SetDefaultManager	11528	59,00%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - CMFCVisualManager::AdjustFrames	11515	58,93%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - user32.dll!0x007ff879675445	11515	58,93%	user32.dll
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - user32.dll!0x007ff87967ad09	11515	58,93%	user32.dll
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - win32u.dll!0x007ff878a61164	11515	58,93%	win32u.dll
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - ntoskrnl.exe!0xfffff8000f1fa553	11515	58,93%	ntoskrnl.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - win32kfull.sys!0xfffff16ed8c6a92b	11515	58,93%	win32kfull.sys
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - win32kfull.sys!0xfffff16ed8c6b4b4	11515	58,93%	win32kfull.sys
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - win32kfull.sys!0xfffff16ed8c6df8b	11515	58,93%	win32kfull.sys
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - win32kfull.sys!0xfffff16ed8c6e386	11515	58,93%	win32kfull.sys
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - win32kfull.sys!0xfffff16ed8c6f877	11515	58,93%	win32kfull.sys
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - ntoskrnl.exe!0xfffff8000f59546e	11515	58,93%	ntoskrnl.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - ntdll.dll!0x007ff87b653b14	11515	58,93%	ntdll.dll
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - user32.dll!0x007ff8796919c3	11515	58,93%	user32.dll
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - user32.dll!0x007ff87967b54c	11515	58,93%	user32.dll
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - user32.dll!0x007ff87967b85d	11515	58,93%	user32.dll
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - AfxWndProc	11515	58,93%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - AfxCallWndProc	11515	58,93%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - CWnd::WindowProc	11515	58,93%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - CWnd::OnWndMsg	11515	58,93%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - CFrameWndEx::OnChangeVisualManager	11515	58,93%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - CFrameImpl::OnChangeVisualManager	11515	58,93%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - CMFCRibbonBar::RecalcLayout	11515	58,93%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - CMFCRibbonQuickAccessToolBar::GetRegularSize	11515	58,93%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - CMFCRibbonButtonsGroup::GetRegularSize	11515	58,93%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - CMFCRibbonBaseElement::GetSize	11515	58,93%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - CMFCRibbonButton::GetCompactSize	11515	58,93%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - CMFCRibbonBaseElement::GetDropDownImageWidth	11515	58,93%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - CMenuImages::Size	11515	58,93%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! - CMenuImages::Initialize	11515	58,93%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + CMFCToolBarImages::UpdateInternalImage	5518	28,24%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + CMFCToolBarImages::SmoothResize	3821	19,56%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + CMFCToolBarImages::MapBmpTo3dColors	2149	11,00%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + CMFCToolBarImages::LoadStr	27	0,14%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + CTooltipManager::UpdateTooltips	13	0,07%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + user32.dll!0x007ff879675445	1	0,01%	user32.dll
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + CDockingManager::DockPane	4	0,02%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + CDockingManager::EnableAutoHidePanes	4	0,02%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + CMFCRibbonBar::Create	2	0,01%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + CMainFrame::CreateDockingWindows	1	0,01%	MGen.exe
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! + CFrameWndEx::OnCreate	1	0,01%	MGen.exe

[rualark]

Classic or browser toolbar (0.1 s):

Advanced menu (2 s):

Ribbon (11 s):