3D batch processing and BlazeFace

[esterivera]

Hi @imadtoubal ! Thank you for the amazing code; I would like to know, if processing 3D landmarks for a complete batch is on the roadmap of the library.

Aditionally, I have a problem when I try to run the 2D landmark detector with the BlazeFace detector. I use the same code as in the notebook, with a different video:

fa = face_alignment.FaceAlignment(face_alignment.LandmarksType._2D, device='cpu', face_detector='blazeface')

t_start = time.time() preds = fa.get_landmarks_from_image(frames[0])

`--------------------------------------------------------------------------- AssertionError Traceback (most recent call last)

in 1 t_start = time.time() ----> 2 preds = fa.get_landmarks_from_image(frames[0]) 3 #print(f'BlazeFace: Execution time for a single image: {time.time() - t_start}') ~/anaconda3/envs/bulat/lib/python3.7/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs) 13 def decorate_context(*args, **kwargs): 14 with self: ---> 15 return func(*args, **kwargs) 16 return decorate_context 17 ~/python_custom_packages/face-alignment/face_alignment/api.py in get_landmarks_from_image(self, image_or_path, detected_faces) 138 139 if detected_faces is None: --> 140 detected_faces = self.face_detector.detect_from_image(image[..., ::-1].copy()) 141 142 if len(detected_faces) == 0: ~/python_custom_packages/face-alignment/face_alignment/detection/blazeface/blazeface_detector.py in detect_from_image(self, tensor_or_path) 50 image = self.tensor_or_path_to_ndarray(tensor_or_path) 51 ---> 52 bboxlist = detect(self.face_detector, image, device=self.device)[0] 53 54 return bboxlist ~/python_custom_packages/face-alignment/face_alignment/detection/blazeface/detect.py in detect(net, img, device) 22 orig_size = min(H, W) 23 img, (xshift, yshift) = resize_and_crop_image(img, 128) ---> 24 preds = net.predict_on_image(img) 25 26 if 0 == len(preds): ~/python_custom_packages/face-alignment/face_alignment/detection/blazeface/net_blazeface.py in predict_on_image(self, img) 189 img = torch.from_numpy(img).permute((2, 0, 1)) 190 --> 191 return self.predict_on_batch(img.unsqueeze(0))[0] 192 193 def predict_on_batch(self, x): ~/python_custom_packages/face-alignment/face_alignment/detection/blazeface/net_blazeface.py in predict_on_batch(self, x) 212 213 assert x.shape[1] == 3 --> 214 assert x.shape[2] == 128 215 assert x.shape[3] == 128 216 AssertionError: ` Exactly the same video works correctly with sfd. Thanks!

[imadtoubal]

Hello @esterivera, to answer your first question, I think adding support for 3D faces would not be a difficult addition to the library, I might give it a try this coming weekend!

To answer your specific question: From your code, I can see that your code exactly matches the demo. I think it is possible that I did not account for edge cases like:

• When only part of the face is showing in the frame
• When a face is outside of the square center crop of the image
• When there is no face in the frame, the exceptions afterward are not handled.

If you do not mind sharing that specific video you tried (or at least that first frame), it can help me troubleshoot this issue and possibly fix it.

I am glad you find my contribution useful and I hope we can work together on this issue to help make it more stable and reliable.

Kind regards.

[esterivera]

Hi @imadtoubal , thank you very much, Im looking forward for the 3D batch processing, it would really help me.

Regarding the BlazeFace issue, here is the problematic video:

output1.zip

It is vertical oriented; I tested with one horizontal oriented video, and it worked, so maybe that is the source of the issue.

Thank you in advance, regards

[imadtoubal]

@esterivera Could you try again with your video, the issue should be patched. I am currently looking into batch-processing for 3D landmarks in BlazeFace.

[esterivera]

@imadtoubal I tested, and it worked perfectly! Thank you for your quick response

[esterivera]

The batch 3D processing is working like a charm. Thank you!!

[imadtoubal]

That's awesome, good luck with your project!