Memory leak on long video (200 clips)

[srameshr]

Generating a video from clips with the kenBurns model of width 720 fails in between when running on an AWS machine with sufficient memory.

Using xvfb-run -s '-screen 0 1024x768x24 -ac +extension GLX +render -noreset' node index.js fails with an out of memory error if the video is 720P in width.

m5ad.large | 2 CPU | 8 GiB RAM | 1 x 75 GB NVMe SSD Storage | Up to 2.120 Gbps | Up to 10 Gbps

Error:

LLVM ERROR: Unable to allocate section memory

[mifi]

Does it work on other AWS EC2 machines?

[srameshr]

Its does not work on any AWS or DO machines. I tried with a 32 GB RAM as well and increased the swap size too. My local mac has a 16 GB memory and it works perfectly fine for any length of the video.

[mifi]

Which OS are you running? Could you try newest Ubuntu?

[srameshr]

I tried Fedora, Ubuntu & CentOS. Fails on all the distros.

[mifi]

how do you install it? do you use npm i -g editly? which version of node.js? headless machine or x11?

[srameshr]

I install it by running npm install as editly is a dependency in our project. Node 14.X and NPM 6.X. Here are the steps required to install all the dependencies required to run editly on AWS linux / centos distros. Ignore the indexes, they are off.

    1 sudo yum update -y
    2  export LC_CTYPE="en_US.UTF-8"
    3  sudo yum install gcc-c++ make
        4  curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash
    6  . ~/.nvm/nvm.sh
    7 nvm install node

       14  npm -v
       15  node- v
       cd ~
        sudo yum install mesa-libGL -y
    sudo yum install -y Xvfb
    sudo yum install -y mesa-dri-drivers
     sudo yum install fontconfig freetype freetype-devel fontconfig-devel libstdc++ -y
   19  cd /usr/local/bin
   21  sudo mkdir ffmpeg
   22  cd /usr/local/bin/ffmpeg
   23  sudo wget https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz
   27  sudo tar -xf ffmpeg-release-amd64-static.tar.xz
   30  sudo ln -sfn /usr/local/bin/ffmpeg/ffmpeg-*-amd64-static/ffmpeg /usr/bin/ffmpeg
   31  sudo ln -sfn /usr/local/bin/ffmpeg/ffmpeg-*-amd64-static/ffprobe /usr/bin/ffprobe
   32  cd ~

   # read here https://aws.amazon.com/premiumsupport/knowledge-center/ec2-memory-swap-file/
   swapon -s

   sudo dd if=/dev/zero of=/swapfileone bs=128M count=256
   sudo chmod 600 /swapfileone
   sudo mkswap /swapfileone
   sudo swapon /swapfileone

[srameshr]

@mifi Tried it and it broke again! Fails around 73% of completion when the video width is 1280 PX and fails around 96% of completion when the video width trying to construct is 720 PX

 72% Done with transition, switching to next transitionFromClip (53)
createFrameSource image clip 54 layer 0
createFrameSource image-overlay clip 54 layer 1
createFrameSource news-title clip 54 layer 2
createFrameSource subtitle clip 54 layer 3
 72%  72%  72%  72%  72%  72%  72%  72%  72%  73%  73%

LLVM ERROR: Unable to allocate section memory
    at afterWriteDispatched (internal/stream_base_commons.js:156:25)
    at writeGeneric (internal/stream_base_commons.js:147:3)
    at Socket._writeGeneric (net.js:784:11)
    at Socket._write (net.js:796:8)
    at writeOrBuffer (_stream_writable.js:352:12)
    at Socket.Writable.write (_stream_writable.js:303:10)
    at /root/video-generator/node_modules/editly/index.js:509:49
    at new Promise (<anonymous>)

[mifi]

Could you share your json spec?

[srameshr]

const editlySpec = {
                outPath: path.resolve(__dirname, `../output/${fileName}`),
                width: parseInt(process.env.VIDEO_WIDTH, 10),
                height: parseInt(process.env.VIDEO_HEIGHT, 10),
                defaults: {
                    duration: SLIDE_DURATION,
                    transition: {
                    duration: 0.5,
                    name: 'fade',
                    },
                    layer: {
                        fontPath: path.resolve(__dirname, `../assets/fonts/CircularStd-Bold.ttf`),
                    },

                },
                audioFilePath: path.resolve(__dirname, `../assets/music/song.mp3`),

                clips: clips.map((clip, index) => {
                    return {
                        duration: SLIDE_DURATION,
                        layers: [{
                            type: 'image',
                            path: path.resolve(__dirname, '../tmp/' + clip.path),
                            zoomDirection: index % 2 === 0 ? 'in' : 'out',
                        }, 
                        {
                            type: 'image-overlay',
                            path: path.resolve(__dirname, '../assets/images/logo_transparent.png'),
                            position: {  x: 0.99, y: 0.05, originX: 'right', originY: 'top' },
                            width: 0.10,
                            height: 0.10,
                        }, 
                        {
                            type: 'news-title',
                            text: clip.name
                        }, {
                            type: 'subtitle',
                            text: clip.caption,
                            backgroundColor: 'rgba(0,0,0,0.5)'
                        }]
                    }
                })
            }

[mifi]

How many clips is it? Could you post the output if you run with verbose: true ? Also are you running newest version of editly?

cat node_modules/editly/package.json

[srameshr]

"version": "0.6.3" There are around 200 clips, with each clip duration being 10 seconds.

[mifi]

Ok, it does sound like a memory leak then. I've never tried such a long video with 200 clips. Gonna have to find where in the code it's leaking. I'm guessing it's not related to AWS, and it will probably fail on another computer too.

[srameshr]

It works on my mac with 500 clips.

[mifi]

Does it have the same amount of memory? And is it the same input/output resolution on the test on your mac?

[srameshr]

It has half the memory of my AWS machine. Input is same on both the machines.

[mifi]

Maybe the AWS machine doesn't have hardware acceleration, while the Mac has. Then maybe in node-canvas there is some memory leak only with software acceleration.

[srameshr]

@mifi Anything on this? Tried the latest version. Fails with the same error.

[mifi]

Could you run editly with the verbose flag and share the full "calculated" spec that gets printed in the beginning?

[ariel-frischer]

I might be having a similar issue, I was using c5d.2xlarge aws ec2 with elastic beanstalk (Docker running on 64bit Amazon Linux 2/3.0.3). I only had 15 clips but each clip has like a video + an overlayed alpha channel .mov with custom text animations. My issue is that it just stops on the 3rd to last clip, with no error message or anything, I'm assuming ec2/node could be running out of memory or there is a memory leak. Editly actually finished creating the file, but doesn't render the last 3 clips I input. I will test using a larger ec2 instance and relay my results.

ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)
  configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 78.100 / 55. 78.100
  libavcodec     57.107.100 / 57.107.100
  libavformat    57. 83.100 / 57. 83.100
  libavdevice    57. 10.100 / 57. 10.100
  libavfilter     6.107.100 /  6.107.100
  libavresample   3.  7.  0 /  3.  7.  0
  libswscale      4.  8.100 /  4.  8.100
  libswresample   2.  9.100 /  2.  9.100
  libpostproc    54.  7.100 / 54.  7.100

-i myVideo.mp4 -t 8.828571428571427 -vf fps=30,scale=(iw*sar)*max(1920/(iw*sar)\,1080/ih):ih*max(1920/(iw*sar)\,1080/ih),crop=1920:1080 -map v:0 -vcodec rawvideo -pix_fmt rgba -f image2pipe -

ffmpeg cli:

ffmpeg -f rawvideo -vcodec rawvideo -pix_fmt rgba -s 1920x1080 -r 30 -i - -stream_loop -1 -i soundTrack.mp3 -shortest -map 0:v:0 -map 1:a:0 -acodec aac -b:a 128k -vf format=yuv420p -vcodec libx264 -profile:v high -preset:v medium -crf 18 -movflags faststart -y /backend/tmp/vayn_in_home_article_INITIATE_AI_d8becfe3-485b-4cb2-ac7f-59e080af1db8.mp4

[ariel-frischer]

Update: Improving ec2 instance to c5d.9xlarge seemed to have no effect, same issue occurs. If I run this same editly render on my local windows machine it works :/ no error message to figure out how to fix is very stressful.

[ariel-frischer]

I noticed in my verbose logs ffmpeg exits (0 code), but then tries to write another frame, then cleanup, then closes the videos it was supposed to write... You can see it here:

[srameshr]

@ariel-frischer Are those stock footages from pexel.com?

[ariel-frischer]

@ariel-frischer Are those stock footages from pexel.com?

Yea mostly... why?

[srameshr]

@ariel-frischer I am building a tool for YouTubers that picks up videos from pexels.com and was just curious about how others are using this tool. Are you sharing what you are working on in public?

[ariel-frischer]

@ariel-frischer I am building a tool for YouTubers that picks up videos from pexels.com and was just curious about how others are using this tool. Are you sharing what you are working on in public?

I'm making a text to video platform, I guess your right I could hide the urls, didn't see much of the harm though.

[srameshr]

Sent you a mail.

[ariel-frischer]

Sent you a mail.

? I didn't seem to get anything...

[srameshr]

The one with gmail.com or .edu? I sent it to gmail.

[ariel-frischer]

The one with gmail.com or .edu? I sent it to gmail.

Gmail is better I just updated it.

[ariel-frischer]

Update: Somehow my editly version was not on the latest version 💯 things should be working now as expected.

[ariel-frischer]

Ok, I've figured out the issue, I noticed editly just stopped at around the time my audio file finished, even though loopAudio was set to true. The actual issue was that my FFMPEG version 3.4.8-0ubuntu0.2 was not high enough so there could have been some bugs with stream loop -1. My new ffmpeg version 4.3.1 seemed to solve this issue, all the clips are now rendering correctly with looped audio.