Zombie processes after reader.close()

[svleeuwen]

Hi,

When MoviePy finishes rendering I'm still left with a lot of zombie ffmpeg processes. (http://cl.ly/image/1X2h0o0J103K)

These don't use any resources, but there is a maximum nr of processes ofcourse. Is this bad? Can this be overcome?

[Zulko]

What do you mean by "after rendering" ? Even after the python session is closed ?

If you still have VideoFileClips around it is normal that their processes are still there. Do you destroy the VideoFileClips ?

One thing I generally do is encapsulate the whole rendering process inside a function. This way all the clips are destroyed when we leave the function and there are no preocesses left.

Tell me if this didn't answer your question.

[svleeuwen]

My MoviePy code is executed by a Celery task send from a Django view. So as long as the parent process lives (in this case Celery) the ffmpeg processes will stay alive. (without using resources though)

I explicitly call clip.reader.close() because somehow a reference to the ffmpeg_reader still exists when I just do del clip. (where clip is a VideoFileClip instance)

[svleeuwen]

I see that they get reused when compiling another video. So it's actually not so bad.

[Zulko]

That's strange. Are you reusing some clips in common in your different requests ?

[alesgenova]

svleeuwen, did you ever get to fix it? I'm also using moviepy in combination with django and celery, and ffmpeg processes get left behing I've tried many combinations of clip.close(), del clip, del clip.audio and so on, but I don't seem to be able to solve it once and for all. Specifically the ffmpeg processes that seem to get left behind the most are ones for reading the audio from the input VideoFileClip. Any help is appreciated

[svleeuwen]

@alesgenova: I was under pressure to finish the project so I just watched the processes and killed them if they stayed around for too long.

But did you try @mbeacom's fix above?

[alesgenova]

Yes, I gave that a shot, but it didn't solve it completely, it was still unreliable. If anyone is interested, I ended up having all my moviepy/celery tasks in a separate queue whose worker is set to have --max-tasks-per-child=1, so that the worker is started fresh after each task. It's probably not an elegant solution, but at least it insures memory doesn't leak and I don't leave processes behind. And above all, it doesn't require a manual input from me!

[svleeuwen]

Ah, now I remember. I also used --max-tasks-per-child=1 😄

[mbeacom]

@alesgenova It's odd that you're still experiencing zombie/defunct ffmpeg processes with my referenced fork. Did you uninstall/manually install the fork egg? pip uninstall moviepy -y; pip install -e git+git://github.com/mbeacom/moviepy.git@fix-ffmpeg-zombies#egg=moviepy. All reading jobs in my test scenarios no longer result in zombie processes. Hope it helps!

[mbeacom]

@svleeuwen @alesgenova Please pull an update from master directly and let me know if you're still experiencing this issue. Hopefully we'll be able to merge the update to PyPi soon.

[whelly]

I'm still experiencing this issue, even with trying all the solutions in this thread. I'm using the master branch currently, Python 2.7, and Windows 10. Any other ideas to force close the FFMPEG processes? My end goal is to delete some of the source video files after the render is complete, but I'm getting Error 32 since FFMPEG still has control of those files. Using reader.close() on the various clips does exit out some of the FFMPEG processes but I can't get all of them for some reason. If there is the audio clip equivalent of reader.close() maybe that would solve the issue.

Edit: Also, even though I'm able to use close() for video clips and close_proc() for audio, there's still one FFMPEG process left. I have no idea where it comes from or how to close it manually.

[Mordokkai]

Hi, I had also an issue like that. Since I was running my program on thousands of videos, I got my RAM rapidly full. I've just looked at the moviepy code and it seems that the destructor of VideoFileClip is never called when I use "del clip". It's seems to be related to the fact that "del" doesn't call "del()" in every case (cf http://stackoverflow.com/questions/41516287/python-del-does-not-work-as-destructor/41516416). So I explicitly used clip.__del__()every time. I did the same in AudioFileClip, adding self.reader.__del__() in the destructor and idem in VideoFileClip i added self.audio.__del__() and self.reader.__del__() in the destructor. It's very ugly but it seems to work for me.

[whelly]

Thanks @Mordokkai, I did something a little different because del() didn't seem to work for me. I figured out recently that the audio portion of the clip also has a reader you can close, but the function is named differently from the video reader. Call both these on the clip you need to take control of: clip1.reader.close() and clip1.audio.reader.close_proc().

[Mordokkai]

Yes, sorry, soon after I sent my answer, I realized that my solution raised other issues (about missing attributes). I tried to fix it but I didn't succeed. I finally used subprocess.call(["pkill -9 -f " + video_name[2:]],shell=True) but it's a very ugly code. I'll try your solution. Thanks a lot :)

[ghost]

This is a note to the developers of moviepy (which also includes myself))..

We probably need to modify the close_proc function so that it matches the video reader class and rename the function to close.

[kstohr]

Hey there... just wanted to check in and see if this issue is being worked on... I am not an expert in processes.. but getting zombie processes running a subprocess to split audio from a movie file.

[Mordokkai]

I use this function to close my video clips and it works for me. There is probably a better solution, and maybe this issue was fixed since april.

def close_clip(clip):
        try:
                clip.reader.close()
                del clip.reader
                if clip.audio != None:
                        clip.audio.reader.close_proc()
                        del clip.audio
                del clip
        except Exception as e:
                sys.exc_clear()

[nathanielws93]

I'm still running into this issue.- I'm trying to render about 100 videos (each one consists of several VideoFileClip subclips), but each time I render 5 videos or so, I get the "Handle is Invalid" error. I've had to restart & clear my Jupyter kernel after every 4-5 videos I make. I tried using Mordokkai's script, but I'm still getting the error. Any tips?

[cesarandreslopez]

@Mordokkai 's solution worked for me. I just call that function after every clip is used.

[raj6996]

Today I got error:

C:\Users\YL>python "F:\Python36\videogrep1.py" --input "D:\I.mp4" --search "Rule" D:\I.srt [+] Searching for video file corresponding to 'D:\I.srt'. [+] Found 'D:\I.mp4'. [+] Search term 'Rule' was found in 1 places. [+] Creating clips. 125.25 to 128.836: Rule number one, never take your eye off your opponent. [+] Concatenating clips. [+] Writing ouput file. [MoviePy] >>>> Building video supercut.mp4 [MoviePy] Writing audio in temp-audio.m4a 100%|█████████████████████████████████████████████████████████████████████████████████| 80/80 [00:00<00:00, 170.67it/s] [MoviePy] Done. [MoviePy] Writing video supercut.mp4 100%|███████████████████████████████████████████████████████████████████████████████| 108/108 [00:00<00:00, 216.00it/s] [MoviePy] Done. [MoviePy] >>>> Video ready: supercut.mp4

Exception ignored in: <bound method FFMPEG_AudioReader.del of <moviepy.audio.io.readers.FFMPEG_AudioReader object at 0x00000245288DB5C0>> Traceback (most recent call last): File "F:\Python36\lib\site-packages\moviepy\audio\io\readers.py", line 251, in del self.close_proc() File "F:\Python36\lib\site-packages\moviepy\audio\io\readers.py", line 147, in close_proc self.proc.terminate() File "F:\Python36\lib\subprocess.py", line 1139, in terminate _winapi.TerminateProcess(self._handle, 1) OSError: [WinError 6] The handle is invalid

[GE3999]

Thanks I solved this problem add clip.close() after write_gif()

[jaykm]

I am still having this issue, is there any known fix? I am closing every single clip/video in my program.

[ajiteshsingh]

Yes, sorry, soon after I sent my answer, I realized that my solution raised other issues (about missing attributes). I tried to fix it but I didn't succeed. I finally used subprocess.call(["pkill -9 -f " + video_name[2:]],shell=True) but it's a very ugly code. I'll try your solution. Thanks a lot :)

I was about to remove moviepy as a dependency then saw your solution and it worked. Thanks buddy

[ghost]

If anyone is still having this issue now (especially with any Audio/VideoFileClip) when trying to delete any tmp files, please try this function. tmp_dir is the path with all your files you wish to delete, and the audio_file_clip is any Audio/VideoFileClip object that is holding you back in life.

def zombie_killer(tmp_dir, audio_file_clip) -> None:
        os.chdir(os.path.dirname(os.path.abspath(__file__)))
        for real_file in os.listdir(tmp_dir):
            try:
                os.unlink(os.path.join(tmp_dir, real_file))
            except PermissionError:
                try:
                    file_name = os.path.join(tmp_dir, real_file)
                    for proc in psutil.process_iter():
                        try:
                            files = proc.open_files()
                            for tmp_file in files:
                                if tmp_file.path == file_name:
                                    if sys.platform == "win32":
                                        subprocess.call(['taskkill', '/F', '/PID', str(proc.pid)])
                                    else:
                                        os.kill(proc.pid, signal.SIGTERM)
                        except (psutil.NoSuchProcess, psutil.AccessDenied, psutil.ZombieProcess):
                            pass
                    audio_file_clip.close()
                    os.unlink(os.path.join(tmp_dir, real_file))
                except Exception as e:
                    print(e)
                    pass