search

bavc / qctools

QCTools (Quality Control Tools for Video Preservation) is a free and open source software tool that helps users analyze and understand their digitized video files through use of audiovisual analytics and filtering. QCTools is funded by the National Endowment for the Humanities and the Knight Foundation, and is developed by the Bay Area Video Coalition.
http://www.bavc.org/qctools
Other
306 stars 41 forks source link

Memory limitations in windows build? #89

Closed kieranjol closed 8 years ago

kieranjol commented 8 years ago

Hello,

Large files are not opening in the windows build but everything is ok on Mac. I couldn't find a pattern in terms of file size or codec, but I did notice that qctools would not let me add any more files once the memory usage for the program was approaching 2 gigs. This occurs on two different xeon 3.50ghz/ win 7 pro 64 bit/ 16gigs of ram (HP Z420 workstation). The files are saved locally to an internal raid, some are on a network. It doesn't seem to matter where they are stored.

I can add a lot of small files up to the point that memory approaches 1.8 gigs, then it won't allow me to add any more. Or I can add two different 1080i v210.mov files that are a combined 8 gigs, but I can't add any more files as this comes close to 1.8 gigs of ram usage for qctools.It will allow me to add little 10MB files, just not large ones. We don't have any issues with any of these files on our macs.

i can add a 700 MB prores.mov and it sometimes opens in qctools. if I close qctools and reopen, and drag and drop the same file in, it only loads the 700mb video some of the time, and uses different amounts of memory each time. if it fails, it can still use around 1.3 gigs, if it loads it can use either 1.2 gigs or 1.6 gigs.

Very large files, around 90 gigs, just use 256megs of ram and never open.

Let me know if you think the issue may be something else, or if you require any more details. I can't figure out exactly what's going on.

here's an ffmpeg output from that prores file: C:\Users\Ingest_1>C:\Users\Ingest_1\Downloads\ffmpeg-20150113-git-b23a866-win32- static\bin\ffmpeg.exe -i "\MV262aa.mov" ffmpeg version N-69040-gb23a866 Copyright (c) 2000-2015 the FFmpeg developers built on Jan 12 2015 22:02:37 with gcc 4.9.2 (GCC) configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab le-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca -- enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-lib modplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrw b --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinge r --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --en able-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable- libx265 --enable-libxavs --enable-libxvid --enable-lzma --enable-decklink --enab le-zlib libavutil 54. 16.100 / 54. 16.100 libavcodec 56. 20.100 / 56. 20.100 libavformat 56. 18.101 / 56. 18.101 libavdevice 56. 4.100 / 56. 4.100 libavfilter 5. 7.100 / 5. 7.100 libswscale 3. 1.101 / 3. 1.101 libswresample 1. 1.100 / 1. 1.100 libpostproc 53. 3.100 / 53. 3.100 Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '\RACKSTATION\MV262aa.mov': Metadata: creation_time : 2014-09-22 13:32:11 timecode : 06:04:39:22 Duration: 00:02:15.48, start: 0.241200, bitrate: 43369 kb/s Stream #0:0(eng): Video: prores (apcn / 0x6E637061), yuv422p10le(bt470bg/smp te170m/unknown), 720x576, 40647 kb/s, 24.98 fps, 25 tbr, 2500 tbn, 2500 tbc (def ault) Metadata: creation_time : 2014-09-22 13:34:43 handler_name : Apple Alias Data Handler encoder : Apple ProRes 422 timecode : 06:04:39:22 Stream #0:1(eng): Audio: pcm_s24le (in24 / 0x34326E69), 48000 Hz, mono, s32 (24 bit), 1152 kb/s (default) Metadata: creation_time : 2014-09-22 13:34:44 handler_name : Apple Alias Data Handler Stream #0:2(eng): Audio: pcm_s24le (in24 / 0x34326E69), 48000 Hz, mono, s32 (24 bit), 1152 kb/s (default) Metadata: creation_time : 2014-09-22 13:34:44 handler_name : Apple Alias Data Handler Stream #0:3(eng): Data: none (tmcd / 0x64636D74) (default) Metadata: creation_time : 2014-09-22 13:34:44 handler_name : Apple Alias Data Handler reel_name : 001 timecode : 06:04:39:22

here's a v210.mov which is 17 gigs but uses up very little memory (anywhere form 100MB to 250mb) in qctools for some reason. Anything to do with this: Found duplicated MOOV Atom. Skipped it ?

C:\Users\Ingest_1>C:\Users\Ingest_1\Downloads\ffmpeg-20150113-git-b23a866-win32- static\bin\ffmpeg.exe -i "C:\Users\Ingest_1\Downloads\monsignor reid collection roll 11 usable extract bestlight.mov" ffmpeg version N-69040-gb23a866 Copyright (c) 2000-2015 the FFmpeg developers built on Jan 12 2015 22:02:37 with gcc 4.9.2 (GCC) configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab le-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca -- enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-lib modplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrw b --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinge r --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --en able-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable- libx265 --enable-libxavs --enable-libxvid --enable-lzma --enable-decklink --enab le-zlib libavutil 54. 16.100 / 54. 16.100 libavcodec 56. 20.100 / 56. 20.100 libavformat 56. 18.101 / 56. 18.101 libavdevice 56. 4.100 / 56. 4.100 libavfilter 5. 7.100 / 5. 7.100 libswscale 3. 1.101 / 3. 1.101 libswresample 1. 1.100 / 1. 1.100 libpostproc 53. 3.100 / 53. 3.100 [mov,mp4,m4a,3gp,3g2,mj2 @ 0417bfe0] Found duplicated MOOV Atom. Skipped it Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'C:\Users\Ingest_1\Downloads\monsignor r eid collection roll 11 usable extract bestlight.mov': Metadata: creation_time : 2015-04-08 09:20:38 timecode : 01:00:00:00 Duration: 00:10:55.16, start: 0.000000, bitrate: 221187 kb/s Stream #0:0(eng): Video: v210 (v210 / 0x30313276), yuv422p10le, 720x576, 221 184 kb/s, 25 fps, 25 tbr, 25k tbn, 25k tbc (default) Metadata: creation_time : 2015-04-08 09:20:38 handler_name : Apple Alias Data Handler encoder : Uncompressed 10-bit 4:2:2 Stream #0:1(eng): Data: none (tmcd / 0x64636D74) (default) Metadata: creation_time : 2015-04-08 09:29:38 handler_name : Apple Alias Data Handler timecode : 01:00:00:00 At least one output file must be specified

kieranjol commented 8 years ago

P.S, will run a test later on windows 10 with a core2quad and see if I can replicate the issue.

JeromeMartinez commented 8 years ago

I don't worry: I bet you'll have the same problem, the tool has the issue, not the OS :).

kieranjol commented 8 years ago

Thanks for the quick reply. So is it a memory issue? Is it connected to this issue in some way? https://github.com/bavc/qctools/issues/80

JeromeMartinez commented 8 years ago

I don't know yet, I did not take the time to investiguate, I am just pretty sure that the issue is in my code and not due to your OS/CPU.

kieranjol commented 8 years ago

Cool. And just to be safe, the same issue occurs with my cpu/OS :]

On Fri, Aug 7, 2015 at 8:52 PM, Jérôme Martinez notifications@github.com wrote:

I don't know yet, I did not take the time to investiguate, I am just pretty sure that the issue is in my code and not due to your OS/CPU.

— Reply to this email directly or view it on GitHub https://github.com/bavc/qctools/issues/89#issuecomment-128811875.

dericed commented 8 years ago

Can you re-test in https://github.com/bavc/qctools/releases/tag/v0.7.2.

kieranjol commented 8 years ago

I apologise for not being able to test the windows build sooner. This issue has been fixed and windows support is 7x better now! Thanks so much!