Out of memory error when refacing

[ooofest]

This was happening in a similar manner yesterday, but as of the main branch at commit 63d501c34bf16180649bea2f38dc882309316442 , I am getting this error when attempting to run Refacer on a 380MB MP4 (h264):

Trying FFMPEG h264_nvenc encoder FFMPEG h264_nvenc encoder works Video codec for FFMPEG: h264_nvenc CUDA mode with providers ['CUDAExecutionProvider', 'CPUExecutionProvider'] inswapper-shape: [1, 3, 128, 128] Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch(). Total frames: 52567 Extracting frames: 21%|████████████ | 11276/52567 [00:20<01:14, 557.87it/s] Traceback (most recent call last): File "D:\refacer-main\venv\lib\site-packages\gradio\routes.py", line 427, in run_predict output = await app.get_blocks().process_api( File "D:\refacer-main\venv\lib\site-packages\gradio\blocks.py", line 1323, in process_api result = await self.call_function( File "D:\refacer-main\venv\lib\site-packages\gradio\blocks.py", line 1051, in call_function prediction = await anyio.to_thread.run_sync( File "D:\refacer-main\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "D:\refacer-main\venv\lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread return await future File "D:\refacer-main\venv\lib\site-packages\anyio_backends_asyncio.py", line 807, in run result = context.run(func, *args) File "D:\refacer-main\app.py", line 31, in run return refacer.reface(video_path,faces) File "D:\refacer-main\refacer.py", line 196, in reface flag, frame = cap.read() cv2.error: OpenCV(4.7.0) D:\a\opencv-python\opencv-python\opencv\modules\core\src\alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 2764800 bytes in function 'cv::OutOfMemoryError'

Although I have configured to use GPU (3090 with 24GB), the system RAM shot up from 17GB used to over 53GB (out of 64GB total) and I did not see GPU memory being allocated in the monitor.

As mentioned, I have experienced an out of memory error since the new (faster) codebase was pushed in the past day or so. While some files can process fine, I notice that smaller files are still hit-or-miss, so I can't supply a definite criteria of when this might occur.

[xaviviro]

That makes sense, as one of the features that enables the program to run so quickly is by first loading all the frames into memory and then processing them in parallel threads. This works well as it is incredibly fast, as long as you have enough memory. A potential solution could be to work in batches of frames and then merge them. Sharing this kind of error helps me a lot to improve the software. Thank you.

[ooofest]

Thanks for looking into it.

It does seem odd, because in this case I'm wondering why the extracted images would require so much memory when the original video was only 330MB. Even if we generously offer that each of the extracted 52K frames was 2MB in size, that doesn't add up to 32GB that Refacer apparently allocated from system RAM before it hit logic which said that it ran out of headroom.

And at the point where it stopped processing, it stopped due to a reported lack of 3MB - but that was still easily available from both system RAM and GPU VRAM.

So, I feel that looking into the memory management here - preloading or not - will only be a good thing.

It might be a consideration to write the extracted frames to a temporary folder and then work on them in parallel from there. Not as fast as keeping everything in memory, but as you implied loading every video's full set of frames into memory will have issues on systems without lots of memory (and in my case, 64GB RAM and 24GB of VRAM would seem to be plenty for processing a 33OMB h264 file,)

[xaviviro]

Could you please post the result of running:

ffprobe -show_streams -select_streams v YOUR_VIDEO.mp4

[rmhckrkz]

Nvidia RTX 2070

C:\Python\REFACE\refacer>python app.py Trying FFMPEG h264_nvenc encoder FFMPEG h264_nvenc encoder works Video codec for FFMPEG: h264_nvenc CUDA mode with providers ['CUDAExecutionProvider', 'CPUExecutionProvider'] inswapper-shape: [1, 3, 128, 128] Running on local URL: http://127.0.0.1:7860

Total frames: 2757 link, set share=True in launch(). Extracting frames: 41%|████████████████▎ | 1126/2757 [00:02<00:03, 472.68it/s] Traceback (most recent call last): File "C:\Users\RMuratuly\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\routes.py", line 427, in run_predict output = await app.get_blocks().process_api( File "C:\Users\RMuratuly\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\blocks.py", line 1323, in process_api result = await self.call_function( File "C:\Users\RMuratuly\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\gradio\blocks.py", line 1051, in call_function prediction = await anyio.to_thread.run_sync( File "C:\Users\RMuratuly\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\anyio\to_thread.py", line 33, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "C:\Users\RMuratuly\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread return await future File "C:\Users\RMuratuly\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\anyio_backends_asyncio.py", line 807, in run result = context.run(func, *args) File "C:\Python\REFACE\refacer\app.py", line 31, in run return refacer.reface(video_path,faces) File "C:\Python\REFACE\refacer\refacer.py", line 196, in reface flag, frame = cap.read() cv2.error: OpenCV(4.7.0) D:\a\opencv-python\opencv-python\opencv\modules\core\src\alloc.cpp:73: error: (-4:Insufficient memory) Failed to allocate 2764800 bytes in function 'cv::OutOfMemoryError' ffmpeg.txt

[ooofest]

ffprobe -show_streams -select_streams v "D:\test_refacer.mp4"

ffprobe version 6.0-full_build-www.gyan.dev Copyright (c) 2007-2023 the FFmpeg developers built with gcc 12.2.0 (Rev10, Built by MSYS2 project) configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-sdl2 --enable-libaribb24 --enable-libdav1d --enable-libdavs2 --enable-libuavs3d --enable-libzvbi --enable-librav1e --enable-libsvtav1 --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxvid --enable-libaom --enable-libjxl --enable-libopenjpeg --enable-libvpx --enable-mediafoundation --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-liblensfun --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc --enable-d3d11va --enable-dxva2 --enable-libvpl --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libilbc --enable-libgsm --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint libavutil 58. 2.100 / 58. 2.100 libavcodec 60. 3.100 / 60. 3.100 libavformat 60. 3.100 / 60. 3.100 libavdevice 60. 1.100 / 60. 1.100 libavfilter 9. 3.100 / 9. 3.100 libswscale 7. 1.100 / 7. 1.100 libswresample 4. 10.100 / 4. 10.100 libpostproc 57. 1.100 / 57. 1.100 Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'D:\test_refacer.mp4': Metadata: major_brand : isom minor_version : 512 compatible_brands: isomiso2avc1mp41 encoder : Lavf55.33.100 Duration: 00:29:12.29, start: 0.000000, bitrate: 1805 kb/s Stream #0:00x1: Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 1280x720 [SAR 1:1 DAR 16:9], 1672 kb/s, 30 fps, 30 tbr, 15360 tbn (default) Metadata: handler_name : VideoHandler vendor_id : [0][0][0][0] Stream #0:10x2: Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 124 kb/s (default) Metadata: handler_name : SoundHandler vendor_id : [0][0][0][0] [STREAM] index=0 codec_name=h264 codec_long_name=H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 profile=High codec_type=video codec_tag_string=avc1 codec_tag=0x31637661 width=1280 height=720 coded_width=1280 coded_height=720 closed_captions=0 film_grain=0 has_b_frames=2 sample_aspect_ratio=1:1 display_aspect_ratio=16:9 pix_fmt=yuv420p level=31 color_range=unknown color_space=unknown color_transfer=unknown color_primaries=unknown chroma_location=left field_order=progressive refs=1 is_avc=true nal_length_size=4 id=0x1 r_frame_rate=30/1 avg_frame_rate=30/1 time_base=1/15360 start_pts=0 start_time=0.000000 duration_ts=26914304 duration=1752.233333 bit_rate=1672563 max_bit_rate=N/A bits_per_raw_sample=8 nb_frames=52567 nb_read_frames=N/A nb_read_packets=N/A extradata_size=42 DISPOSITION:default=1 DISPOSITION:dub=0 DISPOSITION:original=0 DISPOSITION:comment=0 DISPOSITION:lyrics=0 DISPOSITION:karaoke=0 DISPOSITION:forced=0 DISPOSITION:hearing_impaired=0 DISPOSITION:visual_impaired=0 DISPOSITION:clean_effects=0 DISPOSITION:attached_pic=0 DISPOSITION:timed_thumbnails=0 DISPOSITION:captions=0 DISPOSITION:descriptions=0 DISPOSITION:metadata=0 DISPOSITION:dependent=0 DISPOSITION:still_image=0 TAG:language=und TAG:handler_name=VideoHandler TAG:vendor_id=[0][0][0][0] [/STREAM]

[rmhckrkz]

@xaviviro @ooofest @alunit3 @fAIseh00d any progress, gentleman?

[fAIseh00d]

@rmhckrkz @ooofest Might I suggest a quick workaround, so you don't have to wait for a "proper" fix? You can split your videos without re-encoding them using this. 720p video will result in ~(1280x720x3)bytes=2.63Mb per frame depending on the format, so 16Gb can hold roughly 6200 frames. The video in question is 52567 frames long, so it requires ~135Gb RAM. Splitting the video into 3 parts probably will suffice for 64Gb RAM.

[ooofest]

Thanks, I was using a different splitter program (paid) for the same effect, but felt this bug was useful to offer from both usability and long-term memory management perspectives, as well.

And you helped clarify that I was accidentally off in my quickie calculations for space needed to process this file, carried the decimal point in the wrong position - whoops.

Which still doesn't explain why even a 1280x720 mp4 with 10.6K frames would run into memory allocation errors about 1/3 of the way into face processing, considering there remains plenty of RAM and VRAM in this system. It doesn't seem that raw size of the file's frames is the only factor in this issue.

[shyt47]

Has this been resolved yet? I also get this error.