qctools Version 7.3 for Windows is crashing or not responding

[glwhicks]

Hi all! AMIA was fantastic! Great to see the qctools presentations and meet everybody. I downloaded Version 7.3 for Windows before I left for Pittsburgh on our machines and was eager to get get started Qcing with my new found knowledge, but alas, most of the press files(.mkv) have caused the program to either crash or simply not respond. I did get one file to open. We use: Dell computers running Windows 7 Enterprise 64-bit Operating System, 16GB RAM, Intel(R) Xeon(R) CPU E5-1620 v3 @3.50GHz

[iwsundstrom]

In addition, we just tested moving the xml.gz file out of the folder and opening our large test file. It reached completion and generated all the graphs without any problems. We then decided to test "Export->To .qctools.xml.gz" to create a new xml.gz file. Our plan was to close QCTools and open the test file with the freshly generated xml. However, QCTools crashed when we tried to do the export. Not sure if this is helpful, but thought we should pass it along.

[JeromeMartinez]

This version is giving me an error when I try to open it: Any thoughts?

Due to a problem when we applied our patches for release to this branch. New file: https://mediaarea.net/temp/QCTools_0.7.3.20161207_long_xml_load_error_223_fixed_Windows_x64_WithDebugInfo.zip

[iwsundstrom]

Great, thanks. Here's the crash dump produced with that version: Link: https://slashtmp.iu.edu/files/download?FILE=isundstr%2F23516BNOIgr pw: MDPICrashDump5

[ElderOrb]

Just checked version @JeromeMartinez provided - and managed to open MDPI_40000001519547_01_pres.mkv.qctools.xml.gz with no crash on Windows 10.

@iwsundstrom Did you use the same xml.gz file @glwhicks posted ? If different - could you please provide that file?

Also, just to clarify, was it crash on export or on import?

[JeromeMartinez]

With the build at #218 (comment), I have sometimes a "Out of memory! Please try 64bit build." message despite the fact I run the 64-bit version, or a crash, depending of how loaded (RAM usage) is my machine. I propose an alternative patch for fixing this issue.

[iwsundstrom]

@ElderOrb Same file with xml.gz, and the crash is from an import with the x64 debug version provided by Jerome. The "export" story was just an additional note that I thought might be helpful.

[ElderOrb]

@iwsundstrom How much memory do you have on this machine? While switching to 64bit could help (just because it allows to use more than 4Gb), it will not help if you don't have enough memory or memory is heavily fragmented. QCTools definitely requires some optimizations and @JeromeMartinez already started optimizing xml parsing. So maybe makes sense to check his version if he can provide a link

[iwsundstrom]

@ElderOrb It has 16GB of RAM. I'm excited to test the optimized version as it does sound like XML parsing is problem.

[ElderOrb]

How much free memory do you have before opening xml.gz and after getting crash? Besides the lack of memory it might also be related to memory fragmentation, which is different story. Also, you have video file, correct? Which mean as the result of opening xml.gz thumbnails being generated and they also can consume a lot. Could you please make simple experiment and move/rename video file and then open xml.gz last 64bit debug version? This should remove influence of thumbnails and possibly resolve the crash

[iwsundstrom]

In regards to how much memory is used during a crash: Before I open a file I am using about 3.4 GB. During opening the memory builds up to about 9.8 GB used max, wavering up and down between there and lower. This is while opening the video file with the XML.gz beside it.

As for the test of just opening the XML.gz (without the video or thumbnails), that worked. The file successfully opened and populated the graphs, although QCTools was "Not Responding" for a short while.

I'm wondering if including some sort of loading or progress bar may be a good feature in the future, beyond resolving the crashes. It is not very user friendly to open large files and not be given feedback on loading progress.

[JeromeMartinez]

I'm wondering if including some sort of loading or progress bar may be a good feature in the future, beyond resolving the crashes. It is not very user friendly to open large files and not be given feedback on loading progress.

This is already the case when there is no sidecar .gz file (graph is populated in another thread), it could be implemented also for loading the .gz, the proposed patch is in this direction (there is some "hooks" for being able to display some info to the user), but having the progress bar has IMHO some impact in the workflow, it is more development time :(.

[ElderOrb]

We can leave importing in the UI thread and add progress bar (QApplication::processEvents trick). And I absolutely agree, during importing huge files progress bar is 'must have' thing

[JeromeMartinez]

New snapshot: https://mediaarea.net/temp/QCTools_0.7.3.20161208_UncompressByChunks_Windows_i386_WithDebugInfo.zip

[JeromeMartinez]

We can leave importing in the UI thread and add progress bar (QApplication::processEvents trick).

I guess it is a good first step. I was more thinking to have the .gz parsing in a separate thread, in order to let the user drag n drop more files, navigate in the software... As he does when a file is processed. 2nd step.

[iwsundstrom]

Sad to report that the new snapshot still crashes. Here's a crash dump: Link: https://slashtmp.iu.edu/files/download?FILE=isundstr%2F51958E89DwA Pw: MDPICrashDump6

[ElderOrb]

Based on what I see in the dump, this time 'access violation' happened inside 'ModifyOutput' function, which is used during video parsing, so this is not lack of memory but something different. Unfortunately I don't have that video (and don't even have such a big video right now), @dericed any help on generating such a long video with the same characteristics?

Also I noticed you still have application verifier enabled, could you please disable it (in very rare cases it might produce false positives) and see if application still crashes? Also just to exclude lack of memory (although as I said, it is highly unlikely this issue is related to allocations), would be interesting to see how x64 version will behave. But for this we need ask @JeromeMartinez to produce 64bit build from his branch

AVRF Tid [0xdc8] Frame [0x00]: QCTools!FFmpeg_Glue::ModifyOutput Failure Bucketing

INVALID_POINTER_READ Tid [0xdc8] Frame [0x00]: QCTools!FFmpeg_Glue::ModifyOutput

BUGCHECK_STR: INVALID_POINTER_READ_AVRF

DEFAULT_BUCKET_ID: INVALID_POINTER_READ_AVRF

STACK_TEXT:
0027cbd8 01275696 QCTools!FFmpeg_Glue::ModifyOutput+0x66 0027cbf4 012755c7 QCTools!FFmpeg_Glue::AddOutput+0x67 0027cc2c 01286fb9 QCTools!FileInformation::FileInformation+0xed9 0027cd58 01292c00 QCTools!MainWindow::addFile+0xa0 0027cdd8 01290176 QCTools!MainWindow::dropEvent+0xd6 0027ce0c 0158a844 QCTools!QWidget::event+0x584 0027ce5c 016104da QCTools!QMainWindow::event+0x31a 0027ce7c 016053af QCTools!QApplicationPrivate::notify_helper+0xef 0027ce94 01604470 QCTools!QApplication::notify+0x11c0 0027cfe0 0130c181 QCTools!QCoreApplication::notifyInternal2+0xb1 0027d028 01655ff8 QCTools!QWidgetWindow::handleDropEvent+0x158 0027d098 016554de QCTools!QWidgetWindow::event+0x21e 0027d0cc 0160539a QCTools!QApplicationPrivate::notify_helper+0xda 0027d0e8 01604808 QCTools!QApplication::notify+0x1558 0027d234 0130c181 QCTools!QCoreApplication::notifyInternal2+0xb1 0027d27c 01434560 QCTools!QGuiApplicationPrivate::processDrop+0x60 0027d2d0 014e8d7d QCTools!QWindowSystemInterface::handleDrop+0x2d 0027d2f4 01792b87 QCTools!QWindowsOleDropTarget::Drop+0x1f7 0027d344 76a02d89 ole32!CPrivDragDrop::PrivDragDrop+0xcc 0027d378 762e5a4a rpcrt4!Invoke+0x2a 0027d3b8 763605f1 rpcrt4!NdrStubCall2+0x2ea 0027d7bc 76a0e7e6 ole32!CStdStubBuffer_Invoke+0xb6 0027d804 76a0e876 ole32!SyncStubInvoke+0x3c 0027d84c 76a0edd0 ole32!StubInvoke+0xb9 0027d898 769289eb ole32!CCtxComChnl::ContextInvoke+0xfa 0027d974 769288e0 ole32!MTAInvoke+0x1a 0027d990 769294b2 ole32!STAInvoke+0x46 0027d9bc 76a0eccd ole32!AppInvoke+0xab 0027d9f0 76a0eb41 ole32!ComInvokeWithLockAndIPID+0x372 0027dad4 76a0f1fd ole32!ComInvoke+0xc5 0027dafc 7692930f ole32!ThreadDispatch+0x23 0027db10 769292ce ole32!ThreadWndProc+0x161 0027db54 75f362fa user32!InternalCallWinProc+0x23 0027db80 75f36d3a user32!UserCallWinProcCheckWow+0x109 0027dbf8 75f377c4 user32!DispatchMessageWorker+0x3b5 0027dc58 75f3788a user32!DispatchMessageW+0xf 0027dc68 01362a9f QCTools!QEventDispatcherWin32::processEvents+0x40f 0027fa1c 02011e54 QCTools!QWindowsGuiEventDispatcher::processEvents+0x14 0027fa2c 01364349 QCTools!QEventLoop::exec+0x119 0027fa7c 0130b35d QCTools!QCoreApplication::exec+0x14d 0027fad4 0128e9d5 QCTools!main+0x165 0027fb78 02018174 QCTools!WinMain+0xe4 0027fbac 01f7c486 QCTools!__scrt_common_main_seh+0xf6 7efde008 ffffffff unknown!fillpattern+0x0 7efde00c 01180000 QCTools!_wpgmptr+0x0 7efde010 770c0200 ntdll!PebLdr+0x0 7efde014 0060f738 unknown!unknown+0x0

STACK_COMMAND: .ecxr ; kb ; dt ntdll!LdrpLastDllInitializer BaseDllName ; dt ntdll!LdrpFailureData ; dps 27cbd8 ; kb

THREAD_SHA1_HASH_MOD_FUNC: 149eb2a0da52c8b38c064648615db142b8fd336c

THREAD_SHA1_HASH_MOD_FUNC_OFFSET: b58a2edb0e3a82915bf0f8a0aee65193bc7c7531

THREAD_SHA1_HASH_MOD: 52ccbee68d27742fc5b25097be25e66065a90a48

FAULT_INSTR_CODE: 13c83

FAULTING_SOURCE_LINE: c:\programmation\build_7220\qctools_allinclusive\qctools\source\core\ffmpeg_glue.cpp

FAULTING_SOURCE_FILE: c:\programmation\build_7220\qctools_allinclusive\qctools\source\core\ffmpeg_glue.cpp

[iwsundstrom]

I disabled Application Verifier and was able to open the test file very quickly. I retried it multiple times (closing QCTools and opening it again) and it worked every time. I also tried using the snapshot to open other long videos that haven't worked with previous versions and was successful there to. It appears the issue has been fixed and that Application Verifier caused the previous crash! Thanks for working with us on this issue. We really appreciate it!

[dericed]

Closing for now as this seems resolved. Re-open if needed.