search

akashmaity / RobustPPG

9 stars 1 forks source link

How to create file：IMG_2940_landmarks.mat #2

Open xiaoyaozzr opened 1 year ago

xiaoyaozzr commented 1 year ago

FaceMesh interface can only get 478 face 3D landmarks， while in the exsample "IMG_2940_landmarks.mat", it has 1313 points, and the "tm" param cannot get from facemesh. Features also cannot corresponding to the specific location from FaceMesh to matlab code.

akashmaity commented 1 year ago

What FaceMesh interface are you using? The feature mismatch is due to focal length and principal_points being different than the one used in FaceMesh software while generating the mesh.

xiaoyaozzr commented 1 year ago

I refered to the sample code as follows , the "face_landmarks.landmark" only contain 478 points, not found : tm, exp_weights, features, and 1313 landmarks.

"https://google.github.io/mediapipe/solutions/face_mesh.html"

with mp_face_mesh.FaceMesh( static_image_mode=True, max_num_faces=1, refine_landmarks=True, min_detection_confidence=0.5) as face_mesh: for idx, file in enumerate(IMAGE_FILES): image = cv2.imread(file)

Convert the BGR image to RGB before processing.

results = face_mesh.process(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))

# Print and draw face mesh landmarks on the image.
if not results.multi_face_landmarks:
  continue
annotated_image = image.copy()
for face_landmarks in results.multi_face_landmarks:
  print('face_landmarks:', face_landmarks)
  mp_drawing.draw_landmarks(
      image=annotated_image,
      landmark_list=face_landmarks,
      connections=mp_face_mesh.FACEMESH_TESSELATION,
      landmark_drawing_spec=None,
      connection_drawing_spec=mp_drawing_styles
      .get_default_face_mesh_tesselation_style())
  mp_drawing.draw_landmarks(
      image=annotated_image,
      landmark_list=face_landmarks,
      connections=mp_face_mesh.FACEMESH_CONTOURS,
      landmark_drawing_spec=None,
      connection_drawing_spec=mp_drawing_styles
      .get_default_face_mesh_contours_style())
  mp_drawing.draw_landmarks(
      image=annotated_image,
      landmark_list=face_landmarks,
      connections=mp_face_mesh.FACEMESH_IRISES,
      landmark_drawing_spec=None,
      connection_drawing_spec=mp_drawing_styles
      .get_default_face_mesh_iris_connections_style())
cv2.imwrite('/tmp/annotated_image' + str(idx) + '.png', annotated_image)
akashmaity commented 1 year ago

Google's facemesh returns the 3d points of the face features in a different coordinate system than the one (world coordinate) used in this work. The tm matrix is used for converting to pixel coordinate system from world coordinates. Google facmesh estimates this tm matrix in the code itself, but doesn't output it. I am guessing there's a way you can convert the 3d location to face surface normals without needing the tm matrix explicitly.