search

soumik-kanad / diff2lip

Other
322 stars 38 forks source link

Process killed #8

Closed printerwhale closed 10 months ago

printerwhale commented 10 months ago

Here is my settings

real_video_root='dataset/VoxCeleb2/vox2_test_mp4/mp4/' model_path="checkpoints/e7.24.1.3_model260000_paper.pt" sample_path="output_dir" sample_mode="cross" # or "reconstruction" NUM_GPUS=1

GEN_FLAGS="--is_voxceleb2=False --generate_from_filelist=0 --video_path=../test.mp4 --audio_path=../test.mp3 --out_path=../ --save_orig=False --face_det_batch_size 64 --pads 0,0,0,0"

However it doesn't seem to work. I always get :

MPI.COMM_WORLD.Get_rank() 0 os.environ["CUDA_VISIBLE_DEVICES"] 0 Logging to d2l_gen creating model... scripts/inference.sh: line 40: 31738 Killed

printerwhale commented 10 months ago

In my system log, it's written that I run out of memory. However I have 72Gio of memory and I have nothing else running on this instance.

soumik-kanad commented 10 months ago

Hi, it is hard to debug from my side. I have not faced this error before. I think 72G is enough for running inference on VoxCeleb2 length videos (7-8secs).

I suggest you pinpoint where in the inference python script the memory is going out of bounds and debug accordingly.

printerwhale commented 10 months ago

The error occurs in the face detection step. There is the command causing the error : predictions.extend(detector.get_detections_for_batch(np.array(images[i:i + batch_size])))

face_det_batch_size is 64

printerwhale commented 10 months ago

Seems to work with face_det_batch_size = 32