qctools Version 7.3 for Windows is crashing or not responding

[glwhicks]

Hi all! AMIA was fantastic! Great to see the qctools presentations and meet everybody. I downloaded Version 7.3 for Windows before I left for Pittsburgh on our machines and was eager to get get started Qcing with my new found knowledge, but alas, most of the press files(.mkv) have caused the program to either crash or simply not respond. I did get one file to open. We use: Dell computers running Windows 7 Enterprise 64-bit Operating System, 16GB RAM, Intel(R) Xeon(R) CPU E5-1620 v3 @3.50GHz

[dericed]

Thanks Glenn. Can you run ffmpeg -i ON_YOUR_CRASHY_FILE.mkv and post the results here, so we can recreate this. Is there a particular point where it crashes (before graphing, when opening player, etc)?

[glwhicks]

I'll try running the ffmpeg tomorrow, but it crashes before anything comes up on the screen, so before graphing.

[dericed]

Also you could try opening QCTools, go to the preferences and deselect some of the filters, then open the file. Possible that the combination of filter and file is the issue.

[iwsundstrom]

Hi, this is Ian Sundstrom, working with Glenn at MDPI. I ran a .mkv that cannot open in QCTools through FFMPEG, and it completed without any errors. Here is a report log from running the file through FFMPEG: ffmpeg-20161116-115854.zip I also tested the file using the different filters in preferences, but the file still caused crashes. For the first time, I also began receiving error reports from QCTools when it crashed or froze (although this seems unrelated to changing the filters, as restoring them to the defaults still resulted in error reports). Here are a couple examples of them (from two separate crashes): Let me know if you want me to try anything else out!

[dericed]

Extract of log:

ffmpeg started on 2016-11-16 at 11:58:54
Report written to "ffmpeg-20161116-115854.log"
Command line:
ffmpeg -report -i "E:\\20161110-VHS-Memnon_Archiving_Services_Inc\\MDPI_40000002189225.20161110-175707\\data\\MDPI_40000002189225_01_pres.mkv" "E:\\test\\2189225_out.mkv"
ffmpeg version N-82324-g872b358 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.4.0 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-dxva2 --enable-libmfx --enable-nvenc --enable-avisynth --enable-bzlib --enable-libebur128 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenh264 --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-libzimg --enable-lzma --enable-decklink --enable-zlib
  libavutil      55. 36.100 / 55. 36.100
  libavcodec     57. 66.101 / 57. 66.101
  libavformat    57. 57.100 / 57. 57.100
  libavdevice    57.  2.100 / 57.  2.100
  libavfilter     6. 66.100 /  6. 66.100
  libswscale      4.  3.100 /  4.  3.100
  libswresample   2.  4.100 /  2.  4.100
  libpostproc    54.  2.100 / 54.  2.100

[...]

Input #0, matroska,webm, from 'E:\20161110-VHS-Memnon_Archiving_Services_Inc\MDPI_40000002189225.20161110-175707\data\MDPI_40000002189225_01_pres.mkv':
  Metadata:
    title           : VIETNAM A TELEVISION HISTORY
    DATE_DIGITIZED  : 2016-11-03
    BARCODE         : 40000002189225
    DESCRIPTION     : Indiana University-Bloomington. Herman B Wells Library. DS558 .V48 1987 v.2. File use: Preservation Master. Original filename: MDPI_40000002189225_01_pres
    COMMENT         : File Created by Memnon Archiving Services
    ENCODER         : Lavf57.0.100
  Duration: 02:03:47.55, start: 0.000000, bitrate: 77859 kb/s
    Stream #0:0, 1, 1/1000: Video: ffv1 (FFV1 / 0x31564646), yuv422p10le, 720x486, SAR 9:10 DAR 4:3, 29.97 fps, 29.97 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      ENCODER         : Lavc57.1.100 ffv1
      DURATION        : 02:03:47.553000000
    Stream #0:1, 1, 1/1000: Audio: pcm_s24le, 48000 Hz, mono, s32 (24 bit), 1152 kb/s (default)
    Metadata:
      ENCODER         : Lavc57.1.100 pcm_s24le
      DURATION        : 02:03:47.553000000
    Stream #0:2, 1, 1/1000: Audio: pcm_s24le, 48000 Hz, mono, s32 (24 bit), 1152 kb/s (default)
    Metadata:
      ENCODER         : Lavc57.1.100 pcm_s24le
      DURATION        : 02:03:47.553000000
    Stream #0:3, 1, 1/1000: Audio: pcm_s24le, 48000 Hz, mono, s32 (24 bit), 1152 kb/s (default)
    Metadata:
      ENCODER         : Lavc57.1.100 pcm_s24le
      DURATION        : 02:03:47.553000000
    Stream #0:4, 1, 1/1000: Audio: pcm_s24le, 48000 Hz, mono, s32 (24 bit), 1152 kb/s (default)
    Metadata:
      ENCODER         : Lavc57.1.100 pcm_s24le
      DURATION        : 02:03:47.553000000
Successfully opened the file.

[ElderOrb]

Crash is not reproducible on my Windows 10 x64. Is it possible to produce crash dump or provide me with sample video causing crash ?

[ElderOrb]

Here are the steps for producing crash dump:

1. Download procdump from here: https://technet.microsoft.com/en-us/sysinternals/dd996900.aspx
2. Extract into any folder you like
3. Launch cmd.exe, navigate to the procdump folder
4. From command line execute the following: procdump.exe -e -ma -w qctools.exe
5. Make qctools.exe to crash, dump should be created inside the procdump folder

[glwhicks]

Thanks for looking into this. Here's some additional information: The files we are QCing are 45Gb to 95Gb (1.5 hours to 4 hours in content length)in size, possibly making it difficult to provide you with a sample file, in addition to us not being permitted to share the files outside of our lab. More often than crashing, the program just won't respond. We've never been able to successfully load anything with a file size that is much larger then 40Gb. This has been consistent with our experience concerning large files we've encountered in the past with other formats, as well. We are now into a large run of VHS files, most of which are 2hrs or longer(60Gb-95Gb). We are unable to use QCTools to QC these files. It is our hope that in the near future QCTools will be able to accommodate larger file sizes. I'm not sure if theses two issues are related.

My associate, Ian, will run a crash dump on one of our offending files. Hopefully by the end of today.

Glenn Hicks Quality Control Specialist Indiana University-MDPI glwhicks@indiana.edu (812)320-7410

---

From: elderorb notifications@github.com Sent: Thursday, November 17, 2016 3:26 PM To: bavc/qctools Cc: Hicks, Glenn William; Author Subject: Re: [bavc/qctools] qctools Version 7.3 for Windows is crashing or not responding (#218)

Crash is not reproducible on my Windows 10 x64. Is it possible to produce crash dump or provide me with sample video causing crash ?

You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/bavc/qctools/issues/218#issuecomment-261359715, or mute the threadhttps://github.com/notifications/unsubscribe-auth/APlXikHfnOcMDj8HlF-T1XHcwWcFjPV7ks5q_Lh0gaJpZM4Kx18p.

[iwsundstrom]

Here's a procdump for a different file. The file I used for the FFMPEG log wouldn't cause QCTools to crash today (it would simply not respond to being dragged into the program), and thus couldn't produce a procdump, so I opted for one that was causing a crash.

Here is the file: https://www.slashtmp.iu.edu/files/download?FILE=isundstr%2F96274TJlbiI The password is: qctools2016

[dericed]

This command provides a close sample to @iwsundstrom except I'm not exactly sure how to replicate the 1/1000 timebase that file uses.

ffmpeg -f lavfi -i mandelbrot=s=720x486:r=29.97 -f lavfi -i sine -f lavfi -i sine -f lavfi -i sine -f lavfi -i sine -pix_fmt yuv422p10le -c:v ffv1 -aspect 4/3 -c:a pcm_s24le -ar 48000 -map 0 -map 1 -map 2 -map 3 -map 4 -vf settb=expr=1/1000 -af asettb=expr=1/1000 -t 3 test.mkv

Increase -t 3 if you want a longer sample.

Also @glwhicks @iwsundstrom can you confirm what settings you have in your preferences (what filter and first/all tracks).

[dericed]

@glwhicks You could probably just cut the first 100 MB off an example to replicate the issue.

[iwsundstrom]

We usually only use signalstats and PSNR filters, however, when you suggested testing a variety of different ones, we did so and QCTools still always crashed.

[dericed]

ok, i was suspicious of the audio, but apparently you're not even using that.

[glwhicks]

I was wondering about that as well, Dave. However, we've never been able to see the audio graph populate with any video format. My theory is the .mkv format might be the issue.

The VHS files have four channels of audio, 2 linear and 2 hi-fi. Might that be causing some problems, perhaps? However, I have a 47Gb(1hr 2mins) UMatic file with only two channels of audio that doesn't respond either.

Glenn Hicks Quality Control Specialist Indiana University-MDPI glwhicks@indiana.edu (812)320-7410

---

From: Dave Rice notifications@github.com Sent: Thursday, November 17, 2016 4:23 PM To: bavc/qctools Cc: Hicks, Glenn William; Mention Subject: Re: [bavc/qctools] qctools Version 7.3 for Windows is crashing or not responding (#218)

ok, i was suspicious of the audio, but apparently you're not even using that.

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/bavc/qctools/issues/218#issuecomment-261373651, or mute the threadhttps://github.com/notifications/unsubscribe-auth/APlXihx_A91wcLJmFGGEByFkFUayldA5ks5q_MXFgaJpZM4Kx18p.

[ElderOrb]

Thank you for the dump provided, looking at this. Could you please also check if the latest qctools snapshot behaves in the same way?

[dericed]

@JeromeMartinez created a debug version of the Windows build at https://mediaarea.net/temp/QCTools_0.7.3.20161118_Windows_i386_WithDebugInfo.zip. This contains the exe and the corresponding pdb file. @glwhicks @iwsundstrom could you re-test with this version.

[glwhicks]

I just tried the debugged version and the first file I dropped into it crashed it. It was a 55Gb .mkv file

[metacynicv2]

I'll give it a test on our Windows machine today and see if I get the same results.

On Mon, Nov 21, 2016 at 8:00 AM, glwhicks notifications@github.com wrote:

I just tried the debugged version and the first file I dropped into it crashed it. It was a 55Gb .mkv file

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/bavc/qctools/issues/218#issuecomment-261979646, or mute the thread https://github.com/notifications/unsubscribe-auth/AI_8UXpvWAznx6IlPOE94wMKHCDp9ZXEks5rAcAXgaJpZM4Kx18p .

Kelly Haydon | Preservationist 415.558.2158 http://415.558.2158 | kelly@bavc.org kelly@bavc.org | @bavcpreserve https://twitter.com/BAVCPreserve

Archivists can use the power of archives to promote accountability, open government, diversity, and social justice. In doing so, it is essential to distinguish objectivity from neutrality. - Howard Zinn

[ElderOrb]

@glwhicks Could you please collect crash dump for debug build? Release dump didn't reveal any obvious reasons for crash..

[ElderOrb]

@JeromeMartinez it is possible to produce 64bit build for Windows? Just thinking this random crashes could be 'out of memory' exception.

[metacynicv2]

I am loading a 40gig ffv1/mkv video file into the debugged Windows build and so far no crashing (though it is slow but that is probably my machine). OS is Windows 7 Professional, 64 bit.

Audio seems to be loading normally.

Not sure if this is relevant, but I do have an MDPI qctools report that doesn't work at all in the recent Windows build, the application doesn't crash but the graphs do not load.

On Mon, Nov 21, 2016 at 1:47 PM, elderorb notifications@github.com wrote:

@JeromeMartinez https://github.com/JeromeMartinez it is possible to produce 64bit build for Windows? Just thinking this random crashes could be 'out of memory' exception.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/bavc/qctools/issues/218#issuecomment-262077637, or mute the thread https://github.com/notifications/unsubscribe-auth/AI_8UYrKDfQOuaZEKzo5JwTrSgLYNHZAks5rAhGDgaJpZM4Kx18p .

Kelly Haydon | Preservationist 415.558.2158 http://415.558.2158 | kelly@bavc.org kelly@bavc.org | @bavcpreserve https://twitter.com/BAVCPreserve

Archivists can use the power of archives to promote accountability, open government, diversity, and social justice. In doing so, it is essential to distinguish objectivity from neutrality. - Howard Zinn

[JeromeMartinez]

@JeromeMartinez it is possible to produce 64bit build for Windows?

We already provide 64-bit builds, look at "Windows_x64" in the snapshots. For example Windows 64-bit 20161116.

For official release, the Installer has actually only 32-bit version despite the "Universal" name, but the "64 bit only without installer" is the pure 64-bit version.

Just thinking this random crashes could be 'out of memory' exception.

As it is a the start of the parsing, and also a 0x00000004 memory access, I bet more on a NULL pointer not tested somewhere.

[ElderOrb]

As it is a the start of the parsing, and also a 0x00000004 memory access, I bet more on a NULL pointer not tested somewhere.

Could be... But so far I managed to reproduce crash only in two cases:

1. Out of memory with ~20 ~5Gb mkv files (32 bit version only). Crashed after several seconds after opening files.
2. Access violation on attempt to open incomplete mkv. But in this case qctools crashed nearly immediately on opening file.

[dericed]

@kellyhaydon can you share the mdpi xml?

[metacynicv2]

I'm not sure I can here - @glwhicks https://github.com/glwhicks?

But the file is in our shared "qctools 2.0" google doc.

On Mon, Nov 21, 2016 at 4:49 PM, Dave Rice notifications@github.com wrote:

@kellyhaydon https://github.com/kellyhaydon can you share the mdpi xml?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/bavc/qctools/issues/218#issuecomment-262114660, or mute the thread https://github.com/notifications/unsubscribe-auth/AI_8UeJFSLwZSm88PkyY6CJAKTZHa-XCks5rAjwHgaJpZM4Kx18p .

Kelly Haydon | Preservationist 415.558.2158 http://415.558.2158 | kelly@bavc.org kelly@bavc.org | @bavcpreserve https://twitter.com/BAVCPreserve

Support BAVC today. www.bavc.org/donate https://bavc.org/donate

Archivists can use the power of archives to promote accountability, open government, diversity, and social justice. In doing so, it is essential to distinguish objectivity from neutrality. - Howard Zinn

[glwhicks]

My associate Ian, will be in tomorrow to provide the requested crashdump. In the meantime, here is the requested XML. In addition I'm providing the QCTools report generated by Memnon. Perhaps this might provide some insight: https://slashtmp.iu.edu/files/download?FILE=glwhicks%2F46627HHrzl0 pw= MDPIXMLfile

https://slashtmp.iu.edu/files/download?FILE=glwhicks%2F15154X03p67 pw= MDPIQCToolsreport

[glwhicks]

To be clear, the provided XML and QCTools report are from a file that crashed the program

[iwsundstrom]

Here's a crash dump using the debug version of QCTools. The file used to crash it is the same one as the file Glenn has posted the XML for directly above this post.

https://slashtmp.iu.edu/files/download?FILE=isundstr%2F32112Lg5qql pw: MDPICrashDump

[ElderOrb]

Call stack from the last dump file:

00328648 00fc3294 0000000d 00000009 00328f14 QCTools!QClipData::initialize+0x8d 0032868c 00fcb828 05384ae8 00000056 00000007 QCTools!storePixels<5>+0x1354 00328f14 00fcd14c 0518d09c 00000400 00000020 QCTools!QRasterPaintEngine::alphaPenBlt+0x4e8 0032903c 00fd696d 05397360 00329080 00fd0eba QCTools!QRasterPaintEngine::drawCachedGlyphs+0x41c 00329070 00fd0eba 00fd0edc 00000017 00000000 QCTools!QRasterPaintEngine::shouldDrawCachedGlyphs+0x4d 00329eb4 0100204f 01002181 000003a0 000003a0 QCTools!QRasterPaintEngine::drawTextItem+0x13a 00329f14 00ffa66c 00329fb4 0032a008 0032a56c QCTools!QTextEngine::fontEngine+0x2f 0032a550 00f61d2a 0032d24c 0032ce54 00000000 QCTools!QTextLine::draw+0xdac 0032cf00 00f5da38 07993bf8 00000000 00000000 QCTools!qt_format_text+0x108a 0032cf7c 01197d99 0032d264 00008b84 0032d2a0 QCTools!QPainter::drawText+0xd8 0032cfac 012aa0c2 0032d24c 012b723b 0032d24c QCTools!QStyle::drawItemText+0x1a9 0032d038 01b0512f 00000021 00000000 000000a3 QCTools!QWindowsXPStylePrivate::drawBackground+0x1d2 0032d050 00000000 00000000 000000a1 00000013 QCTools!_malloc_base+0x38

From the first look seems like memory corruption. Would be interesting to get another dump with application verifier ( https://www.microsoft.com/en-us/download/details.aspx?id=20028 ) enabled.

[ElderOrb]

Here are the steps for configuring application verifier:

1. Download & launch application verifier
2. Add qctools.exe to applications list, then select 'basics' in test area and press save to apply changes

3. Launch qctools & procdump, produce another dump - this time qctools should work much slower, but from the other side it should crash much faster

[iwsundstrom]

Here's another dump produced using the same file as before, with Application Verifier installed and set up, and using QCTools Debug: https://slashtmp.iu.edu/files/download?FILE=isundstr%2F58699uuKnih pw: MDPICrashDump2

With this crash, the file was dragged into QCTools twice, without it crashing or responding in any way (but each did produce an exception in cmd). The third time I opened the file using File->Open and this resulted in the crash.

[ElderOrb]

Unfortunately debug symbols doesn't match.

Debug QCTools is: Timestamp: Fri Nov 18 12:21:26 2016 (582EC816) CheckSum: 01846681 ImageSize: 01FA8000 File version: 0.7.3.41566 Product version: 0.7.3.41566

... and QCTools used during producing dump is: Timestamp: Tue Oct 11 06:36:00 2016 (57FC5E20) CheckSum: 018393CC ImageSize: 01F97000 File version: 0.7.3.41459 Product version: 0.7.3.41459

Looks like release QCTools was used for producing dump file, please re-create dump using debug version from here: https://mediaarea.net/temp/QCTools_0.7.3.20161118_Windows_i386_WithDebugInfo.zip

Also, could you please try 64bit build from the latest snapshot (just want to ensure this is not 32bit memory limitation issue). For example this one: http://old.mediaarea.net/download/snapshots/binary/qctools/20161124/QCTools_0.7.3.20161124_Windows_x64_WithoutInstaller.zip

[iwsundstrom]

My apologies, I must have accidentally opened the regular version of QCTools from the taskbar when attempting to crash it repeatedly. This dump is definitely from the debug version: https://slashtmp.iu.edu/files/download?FILE=isundstr%2F7119942gXQr PW: MDPICrashDump3

As for the snapshot, I downloaded it and was able to crash it. Did you want an additional dump from that build or just to verify that it also crashes?

[ElderOrb]

Yes, please! It could crash due to different reason

[iwsundstrom]

No problem. Here's the snapshot crash dump: https://slashtmp.iu.edu/files/download?FILE=isundstr%2F32764bnGisb PW: MDPICrashDump4

[ElderOrb]

Thank you! This time dump revealed potential reason for the crash - seems like you have xml.gz file for the video you are opening and this might be corrupted. If you really have xml.gz - could you please provide it for reviewing? Also, if this is the case, please try to rename this file or move to another folder and try to reopen video.

[glwhicks]

Here is a link to download the xml.gz file: https://slashtmp.iu.edu/files/download?FILE=glwhicks%2F15154X03p67 pw= MDPIQCToolsreport

[glwhicks]

Ah ha! I moved the xml.gz file to a different folder. I loaded the file into QCTools, and presto it's starting to populate the graphs, even the audio graph! Using the xml.gz file I provided in the previous link, might you be able to help pinpoint the area of corruption within the xml.gz report? This will help us to communicate to the script writers on our end where to look for the problem. Thanks!

[ElderOrb]

Hmm... Interesting, from the first look file seems to be ok, moreover I managed to open it successfully using x64 build. So it really looks like out of memory. Could you please check 64 bit snapshot?

[glwhicks]

Ian will be in the lab tomorrow, so I'll have him do that. I've never been able to get the audio graph to populate with a ffv1 file(ever!), so I do think we're on to something here. In addition, the size of the file is significantly larger than files that have caused QCTools to not respond or crash. Kelly Haydon mentioned a while back, the possibility of the xml.gz file being responsible for the non responsive audio graph.

[dericed]

Does

I've never been able to get the audio graph to populate with a ffv1 file(ever!)

mean

I've never been able to get the audio graph to populate with a ffv1 file that has a neighboring qctools.xml.gz file(ever!)

? If so then your xml.gz just wasn't created with any audio data.

[ElderOrb]

Just tried 32bit build under debugger - now I'm 100% sure this is out of memory. First qctools allocated 564029759 bytes for a buffer to extract xml.gz file, and then it tried to create std::string from buffer (which means allocation of at least 564029759 one more time) - and this time allocation failed.

@dericed & @JeromeMartinez what is the reason for not producing 64bit installers for windows?

[glwhicks]

Yes, Dave, correct. Never with a neighboring xml.gz file. @ElderOrb, I'm not a code type person, might you be able to translate what you just said. I think I understand, but I'm not sure. Thanks!

[ElderOrb]

@glwhicks just try x64 version from here - https://mediaarea.net/download/snapshots/binary/qctools/20161204/ - should fix the issue

[glwhicks]

Okay! Thanks so much for all the great input!

[ElderOrb]

Welcome! but please keep in mind the link provided is not released version, but development snapshot.

[JeromeMartinez]

what is the reason for not producing 64bit installers for windows?

There is still some 32-bit Windows there (Microsoft did not "kill" 32-bit as Apple did for macOS, and there is unfortunately a 32-bit version of Windows 10), so I provide the default installer in 32-bit in order to be compatible with all versions of Windows. Providing several .exe is sometimes a source of confusion for end user who does not know what is 32 and 64-bit, so this is a policy to provide the 64-bit version only in an archive. And this was not problematic up to now. I could add a 64-bit installer beside the 32-bit one (remind that we have users who does not know what is the kind of OS version they have, so offering 2 version on the download page may be confusing), and/or a 32/64-bit installer but it would double the size of the .exe. What would be your preference?

Edit: discussion moved to https://github.com/MediaArea/MediaArea-Utils/issues/96

[iwsundstrom]

Unfortunately, I just ran the x64 version of QCTools and it crashed while trying to open the test file with it's xml.gz. Is there a debug version of x64 QCTools that I can use to send another crash report?

[ElderOrb]

https://www.google.com/url?q=https%3A%2F%2Fmediaarea.net%2Ftemp%2FQCTools_0.7.3.20161207_long_xml_load_error_223_Windows_x64_WithDebugInfo.zip&sa=D&sntz=1&usg=AFQjCNELQJ5CrD8JGu5mGvh99Cs7eD4msA

[iwsundstrom]

This version is giving me an error when I try to open it: Any thoughts?