Preprocessing Dataset: Step2-Normalize Durations

[altuwairqi-s]

Hello

I'm in Preprocessing Dataset, I use UBI_FIGHTS dataset. When running the step2 normalize_videos.py file, it beginning processes the frames as well and then stopping the processes suddenly as the output is shown below:

(venv) sumaya@sumaya-virtual-machine:~/Downloads/human_self_learning_anomaly-master/utils$ python3 normalize_videos.py --root_frames /home/sumaya/Downloads/human_self_learning_anomaly-master/frames --root_sub_videos /home/sumaya/Downloads/human_self_learning_anomaly-master/frames --erase_frames Normalizing video duration... 1 Moviepy - Building video /home/sumaya/Downloads/human_self_learning_anomaly-master/frames/dataset/F_1_0_1_0_1/F_1_0_1_0_1_0.mp4. Moviepy - Writing video /home/sumaya/Downloads/human_self_learning_anomaly-master/frames/dataset/F_1_0_1_0_1/F_1_0_1_0_1_0.mp4

Moviepy - Done !
Moviepy - video ready /home/sumaya/Downloads/human_self_learning_anomaly-master/frames/dataset/F_1_0_1_0_1/F_1_0_1_0_1_0.mp4 Successfully normalized! Normalizing video duration... 5 Moviepy - Building video /home/sumaya/Downloads/human_self_learning_anomaly-master/frames/dataset/F_38_1_2_0_0/F_38_1_2_0_0_0.mp4. Moviepy - Writing video /home/sumaya/Downloads/human_self_learning_anomaly-master/frames/dataset/F_38_1_2_0_0/F_38_1_2_0_0_0.mp4

Moviepy - Done !
Moviepy - video ready /home/sumaya/Downloads/human_self_learning_anomaly-master/frames/dataset/F_38_1_2_0_0/F_38_1_2_0_0_0.mp4 Moviepy - Building video /home/sumaya/Downloads/human_self_learning_anomaly-master/frames/dataset/F_38_1_2_0_0/F_38_1_2_0_0_1.mp4. Moviepy - Writing video /home/sumaya/Downloads/human_self_learning_anomaly-master/frames/dataset/F_38_1_2_0_0/F_38_1_2_0_0_1.mp4

Moviepy - Done !
Moviepy - video ready /home/sumaya/Downloads/human_self_learning_anomaly-master/frames/dataset/F_38_1_2_0_0/F_38_1_2_0_0_1.mp4 Moviepy - Building video /home/sumaya/Downloads/human_self_learning_anomaly-master/frames/dataset/F_38_1_2_0_0/F_38_1_2_0_0_2.mp4. Moviepy - Writing video /home/sumaya/Downloads/human_self_learning_anomaly-master/frames/dataset/F_38_1_2_0_0/F_38_1_2_0_0_2.mp4

Moviepy - Done !
Moviepy - video ready /home/sumaya/Downloads/human_self_learning_anomaly-master/frames/dataset/F_38_1_2_0_0/F_38_1_2_0_0_2.mp4 Moviepy - Building video /home/sumaya/Downloads/human_self_learning_anomaly-master/frames/dataset/F_38_1_2_0_0/F_38_1_2_0_0_3.mp4. Moviepy - Writing video /home/sumaya/Downloads/human_self_learning_anomaly-master/frames/dataset/F_38_1_2_0_0/F_38_1_2_0_0_3.mp4

Moviepy - Done !
Moviepy - video ready /home/sumaya/Downloads/human_self_learning_anomaly-master/frames/dataset/F_38_1_2_0_0/F_38_1_2_0_0_3.mp4 Killed

What is the problem?

[DegardinBruno]

Hello!

The time normalization already started and successfully normalized one of the videos (According to your output). Despite the short description of the problem, the output "Killed" is usually directly related to the python process. I will bet on memory issues. Could you specify your CPU and RAM so that I could reproduce the error?

[altuwairqi-s]

I work with Virtual Machine that has processors = 4 and Memory = 8 GB

[DegardinBruno]

Is the problem persisting? @altuwairqi-s

[altuwairqi-s]

Unfortunately, yes

[DegardinBruno]

Alright, I can recommend you some techniques that I use:

• I am assuming (with your output provided) that you are on a Linux-based system;
• Using the "htop" command, you can follow live the memory size used (RAM and swap) on your PC;
• If the problem persists and happens in the early videos, try to check if a long video causes it (the RAM should start to fill really quick);
• I also recommend you to close any other application (browser, video chats, or other processes), which can "steal" a bit of memory RAM;
• As a final solution, if there is a possibility to have access to 16Gb of ram, it would be better, but I am not sure if it is caused by a long video (just in case)!

[altuwairqi-s]

Thank you for your advice, yes my OS is Linux and I tried all things that you mentioned, and unfortunately didn't work even when increased the RAM to 10GB (This is the max limit to increase).

I found a script that handles this issue. This script alleviates the issue but it still exists, where the problem appears at the beginning of the second video before using the script. And after using it, the problem appears at the end of the second video.

In this experience, I only used four videos where the longest length does not exceed 2 mins. and the script will find in this link: https://serverfault.com/questions/363951/run-a-python-script-that-is-never-killed

[altuwairqi-s]

The problem is solved by using another device with more capabilities. Thank you @DegardinBruno.