How to avoid “CUDA out of memory” in PyTorch

[Sagar1998]

hey there.

the preprocessing is working but actually it has only generated audio files of 3.9gb's. I think there should be image aswell( i tried on mac without any gpu and set device-name:'cuda' to device-name:'cpu' and its generating images) , on my desktop(with nvidia gpu) its only giving me .wav audio files and i am getting warning like

“CUDA out of memory” in PyTorch Tried to allocate 20.00 MiB (GPU 0; 10.76 GiB total capacity; 4.29 GiB already allocated; 10.12 MiB free; 4.46 GiB reserved in total by PyTorch)

can someone help me fix the issue?

[Sagar1998]

hey i somehow managed to remove the cuda out of memory error by reducing the batch size but now its showing me this warning (do note: i am only getting .wav files)

0%| | 5/2242 [00:26<2:54:03, 4.67s/it]Traceback (most recent call last): File "preprocess1.py", line 114, in mp_handler process_video_file(vfile, args, gpu_id) File "preprocess1.py", line 86, in process_video_file preds = fa[gpu_id].get_detections_for_batch(np.asarray(fb)) File "E:\Lip2Wav-20210420T094609Z-003\Lip2Wav\face_detection\api.py", line 66, in get_detections_for_batch detected_faces = self.face_detector.detect_from_batch(images.copy()) File "E:\Lip2Wav-20210420T094609Z-003\Lip2Wav\face_detection\detection\sfd\sfd_detector.py", line 42, in detect_from_batch bboxlists = batch_detect(self.face_detector, images, device=self.device) File "E:\Lip2Wav-20210420T094609Z-003\Lip2Wav\face_detection\detection\sfd\detect.py", line 65, in batch_detect imgs = torch.from_numpy(imgs).float().to(device) RuntimeError: CUDA error: an illegal memory access was encountered 0%| | 5/2242 [00:26<2:54:03, 4.67s/it]Traceback (most recent call last): File "preprocess1.py", line 114, in mp_handler process_video_file(vfile, args, gpu_id) File "preprocess1.py", line 86, in process_video_file preds = fa[gpu_id].get_detections_for_batch(np.asarray(fb)) File "E:\Lip2Wav-20210420T094609Z-003\Lip2Wav\face_detection\api.py", line 66, in get_detections_for_batch detected_faces = self.face_detector.detect_from_batch(images.copy()) File "E:\Lip2Wav-20210420T094609Z-003\Lip2Wav\face_detection\detection\sfd\sfd_detector.py", line 42, in detect_from_batch bboxlists = batch_detect(self.face_detector, images, device=self.device) File "E:\Lip2Wav-20210420T094609Z-003\Lip2Wav\face_detection\detection\sfd\detect.py", line 65, in batch_detect imgs = torch.from_numpy(imgs).float().to(device) RuntimeError: CUDA error: an illegal memory access was encountered

please help me thankyou <3

[Sagar1998]

Hey @Rudrabha , thanks for the project. it works flawlessly but the issue is that it needs only RTX gpu's or Tesla P series GPU(for google colab) to preprocess the data (rest other steps work on any nvidia gpu or tesla gpu's).

If we run preprocess.py on chem lectures on gpu's other than gtx or tesla p series then it throws the warning and error mentioned above and its been 2 weeks im trying to figure out why its giving me such error so please if you can help me out then it would be great.

thankyou soo much. also do not i can't run any other speakers other than chem it just doens't preprocess and stops at 1st iteration