How to use FaceLandmarksSmoothing subgraph for FaceMesh?

[HarryWindsor]

System information (Please provide as much relevant information as possible)

• OS Platform and Distribution (e.g. Linux Ubuntu 16.04, Android 11, iOS 14.4): Linux Ubuntu 20.10 x86_64
• Compiler version (e.g. gcc/g++ 8 /Apple clang version 12.0.0): gcc version 10.3.0
• Programming Language and version ( e.g. C++ 14, Python 3.6, Java ): Python 3.8
• Installed using virtualenv? pip? Conda? (if python): virtualenv
• MediaPipe version: v0.8.7
• Bazel version: bazel 4.2.0
• XCode and Tulsi versions (if iOS):
• Android SDK and NDK versions (if android):
• Android AAR ( if android):
• OpenCV version (if running on desktop): 4.3

To reduce the jittering, I would like to use the FaceLandmarksSmoothing subgraph from face effect for face mesh. I am new to the concept of graphs and did not find any information how to modify face_mesh_desktop.pbtxt (is this the right one for FaceMesh?). The idea was already discussed, but on such a high level that I do not understand how to implement it.

Thank you

EDIT: What I have tried so far: I copied face_landmarks_smoothing.pbtxt into mediapipe/modules/face_landmark. I can build the python whl. But I don't think the landmakrs are smoother than before, I also tried to add some couts in C, but I see no prints in the console.

diff --git a/mediapipe/modules/face_landmark/BUILD b/mediapipe/modules/face_landmark/BUILD
index 2560bad..c749d38 100644
--- a/mediapipe/modules/face_landmark/BUILD
+++ b/mediapipe/modules/face_landmark/BUILD
@@ -53,6 +53,15 @@ mediapipe_simple_subgraph(
     ],
 )

+mediapipe_simple_subgraph(
+    name = "face_landmarks_smoothing",
+    graph = "face_landmarks_smoothing.pbtxt",
+    register_as = "FaceLandmarksSmoothing",
+    deps = [
+        "//mediapipe/calculators/util:landmarks_smoothing_calculator",
+    ],
+)
+
 mediapipe_simple_subgraph(
     name = "face_landmark_front_cpu",
     graph = "face_landmark_front_cpu.pbtxt",
@@ -61,6 +70,7 @@ mediapipe_simple_subgraph(
         ":face_detection_front_detection_to_roi",
         ":face_landmark_cpu",
         ":face_landmark_landmarks_to_roi",
+        ":face_landmarks_smoothing",
         "//mediapipe/calculators/core:begin_loop_calculator",
         "//mediapipe/calculators/core:clip_vector_size_calculator",
         "//mediapipe/calculators/core:constant_side_packet_calculator",

diff --git a/mediapipe/modules/face_landmark/face_landmark_front_cpu.pbtxt b/mediapipe/modules/face_landmark/face_landmark_front_cpu.pbtxt
index f60ca3d..a657789 100644
--- a/mediapipe/modules/face_landmark/face_landmark_front_cpu.pbtxt
+++ b/mediapipe/modules/face_landmark/face_landmark_front_cpu.pbtxt
@@ -32,7 +32,7 @@ input_side_packet: "NUM_FACES:num_faces"
 # particular timestamp if none of faces detected. However, the MediaPipe
 # framework will internally inform the downstream calculators of the absence of
 # this packet so that they don't wait for it unnecessarily.
-output_stream: "LANDMARKS:multi_face_landmarks"
+output_stream: "multi_smoothed_face_landmarks"

 # Extra outputs (for debugging, for instance).
 # Detected faces. (std::vector<Detection>)
@@ -208,6 +208,36 @@ node {
   output_stream: "ITERABLE:multi_face_landmarks"
 }

+# Extracts a single set of face landmarks associated with the most prominent
+# face detected from a collection.
+node {
+  calculator: "SplitNormalizedLandmarkListVectorCalculator"
+  input_stream: "multi_face_landmarks"
+  output_stream: "face_landmarks"
+  node_options: {
+    [type.googleapis.com/mediapipe.SplitVectorCalculatorOptions] {
+      ranges: { begin: 0 end: 1 }
+      element_only: true
+    }
+  }
+}
+
+# Applies smoothing to the single set of face landmarks.
+node {
+  calculator: "FaceLandmarksSmoothing"
+  input_stream: "NORM_LANDMARKS:face_landmarks"
+  input_stream: "IMAGE_SIZE:input_image_size"
+  output_stream: "NORM_FILTERED_LANDMARKS:smoothed_face_landmarks"
+}
+
+# Puts the single set of smoothed landmarks back into a collection to simplify
+# passing the result into the `FaceGeometryFromLandmarks` subgraph.
+node {
+  calculator: "ConcatenateLandmarListVectorCalculator"
+  input_stream: "smoothed_face_landmarks"
+  output_stream: "multi_smoothed_face_landmarks"
+}
+
 # Collects a NormalizedRect for each face into a vector. Upon receiving the
 # BATCH_END timestamp, outputs the vector of NormalizedRect at the BATCH_END
 # timestamp.

diff --git a/mediapipe/calculators/util/landmarks_smoothing_calculator.cc b/mediapipe/calculators/util/landmarks_smoothing_calculator.cc
index 6673816..3fa9a27 100644
--- a/mediapipe/calculators/util/landmarks_smoothing_calculator.cc
+++ b/mediapipe/calculators/util/landmarks_smoothing_calculator.cc
@@ -141,7 +141,9 @@ class VelocityFilter : public LandmarksFilter {
       : window_size_(window_size),
         velocity_scale_(velocity_scale),
         min_allowed_object_scale_(min_allowed_object_scale),
-        disable_value_scaling_(disable_value_scaling) {}
+        disable_value_scaling_(disable_value_scaling) {
+  std::cout << "VelocityFilter()" << std::endl;
+        }

   absl::Status Reset() override {
     x_filters_.clear();
@@ -358,6 +360,8 @@ class LandmarksSmoothingCalculator : public CalculatorBase {
 REGISTER_CALCULATOR(LandmarksSmoothingCalculator);

 absl::Status LandmarksSmoothingCalculator::GetContract(CalculatorContract* cc) {
+  std::cout << "LandmarksSmoothingCalculator::GetContract(CalculatorContract* cc)" << std::endl;
+
   if (cc->Inputs().HasTag(kNormalizedLandmarksTag)) {
     cc->Inputs().Tag(kNormalizedLandmarksTag).Set<NormalizedLandmarkList>();
     cc->Inputs().Tag(kImageSizeTag).Set<std::pair<int, int>>();
@@ -381,6 +385,8 @@ absl::Status LandmarksSmoothingCalculator::GetContract(CalculatorContract* cc) {
 }

 absl::Status LandmarksSmoothingCalculator::Open(CalculatorContext* cc) {
+  std::cout << "LandmarksSmoothingCalculator::Open(CalculatorContext* cc)" << std::endl;
+
   cc->SetOffset(TimestampDiff(0));

   // Pick landmarks filter.
@@ -410,6 +416,8 @@ absl::Status LandmarksSmoothingCalculator::Open(CalculatorContext* cc) {
 }

 absl::Status LandmarksSmoothingCalculator::Process(CalculatorContext* cc) {
+  std::cout << "LandmarksSmoothingCalculator::Process(CalculatorContext* cc)" << std::endl;
+
   // Check that landmarks are not empty and reset the filter if so.
   // Don't emit an empty packet for this timestamp.
   if ((cc->Inputs().HasTag(kNormalizedLandmarksTag) &&

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[HarryWindsor]

Thank you, but I would prefer a solution. Someone must have tried to use FaceLandmarksSmoothing with FaceMesh.

[sgowroji]

@HarryWindsor, This is how you can try with FaceMesh . It was in selfie segmentation. But you can use the same calculator in your graph. Thanks!

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[ecartman2208]

@HarryWindsor were you able to find a solution?

[HarryWindsor]

@ecartman2208 no I did not find a solution yet. Right now, I am working on the same problem in another repo https://github.com/61315/mediapipe-iris-ios/issues/2. But I habe the problem that I am Unable to find Calculator.

EDIT I forgot to register the calculator. Now my last issue is how to get from NORM_FILTERED_LANDMARKS to LANDMARKS.

[HarryWindsor]

@61315 was able to set up a working graph. The only thing thats missing now are the right smoothing parameters for the one_euro_filter. https://github.com/61315/mediapipe-iris-ios/issues/2#issuecomment-922240599

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No