LamboAventador
commented
3 years ago Its similar to #1953 (comment).
Actually, I mainly modify two parts in Iris tracking project. Firstly, the snippets below was insert to file iris_tracking_gpu.pbtxt
#############################################################
# Generates an environment that describes the current virtual scene.
node {
calculator: "FaceGeometryEnvGeneratorCalculator"
output_side_packet: "ENVIRONMENT:environment"
node_options: {
[type.googleapis.com/mediapipe.FaceGeometryEnvGeneratorCalculatorOptions] {
environment: {
origin_point_location: TOP_LEFT_CORNER
perspective_camera: {
vertical_fov_degrees: 63.0 # 63 degrees
near: 1.0 # 1cm
far: 10000.0 # 100m
}
}
}
}
}
# Computes the face geometry for a single face. The input source is defined
# through `use_face_detection_input_source`.
node {
calculator: "SingleFaceGeometryFromLandmarksGpu"
input_stream: "IMAGE:throttled_input_video"
input_side_packet: "ENVIRONMENT:environment"
output_stream: "MULTI_FACE_GEOMETRY:multi_face_geometry"
}
# Renders the plane
node{
calculator: "FaceGeometryEffectRendererCalculator"
input_stream: "IMAGE_GPU:throttled_input_video"
input_stream: "MULTI_FACE_GEOMETRY:multi_face_geometry"
input_side_packet: "ENVIRONMENT:environment"
output_stream: "IMAGE_GPU:plane_effect_video"
node_options: {
[type.googleapis.com/mediapipe.FaceGeometryEffectRendererCalculatorOptions] {
effect_texture_path: "mediapipe/graphs/face_effect/data/plane.pngblob"
effect_mesh_3d_path: "mediapipe/graphs/face_effect/data/plane.binarypb"
}
}
}
# Renders the plane + line
node{
calculator: "FaceGeometryEffectRendererCalculator"
input_stream: "IMAGE_GPU:plane_effect_video"
input_stream: "MULTI_FACE_GEOMETRY:multi_face_geometry"
input_side_packet: "ENVIRONMENT:environment"
output_stream: "IMAGE_GPU:output_effect_video"
node_options: {
[type.googleapis.com/mediapipe.FaceGeometryEffectRendererCalculatorOptions] {
effect_texture_path: "mediapipe/graphs/face_effect/data/line.pngblob"
effect_mesh_3d_path: "mediapipe/graphs/face_effect/data/line.binarypb"
}
}
}
#############################################################
# Renders annotations and overlays them on top of the input images.
node {
calculator: "IrisAndDepthRendererGpu"
# input_stream: "IMAGE:throttled_input_video"
input_stream: "IMAGE_GPU:output_effect_video"
input_stream: "FACE_LANDMARKS:updated_face_landmarks"
input_stream: "EYE_LANDMARKS_LEFT:left_eye_contour_landmarks"
input_stream: "EYE_LANDMARKS_RIGHT:right_eye_contour_landmarks"
input_stream: "IRIS_LANDMARKS_LEFT:left_iris_landmarks"
input_stream: "IRIS_LANDMARKS_RIGHT:right_iris_landmarks"
input_stream: "NORM_RECT:face_rect"
input_stream: "LEFT_EYE_RECT:left_eye_rect_from_landmarks"
input_stream: "RIGHT_EYE_RECT:right_eye_rect_from_landmarks"
input_stream: "DETECTIONS:face_detections"
input_side_packet: "FOCAL_LENGTH:focal_length_pixel"
output_stream: "IRIS_LANDMARKS:iris_landmarks"
output_stream: "IMAGE:output_video"
}Then I modify both BUILD file in path graph/iris_tracking and examples/android/.../apps/iristrackinggpu,
In graph dictionary
cc_library(
name = "iris_tracking_gpu_deps",
deps = [
"//mediapipe/calculators/core:constant_side_packet_calculator",
"//mediapipe/calculators/core:flow_limiter_calculator",
"//mediapipe/calculators/core:split_vector_calculator",
"//mediapipe/graphs/iris_tracking/calculators:update_face_landmarks_calculator",
"//mediapipe/graphs/iris_tracking/subgraphs:iris_and_depth_renderer_gpu",
"//mediapipe/modules/face_landmark:face_landmark_front_gpu",
"//mediapipe/modules/iris_landmark:iris_landmark_left_and_right_gpu",
#################################
"//mediapipe/calculators/core:gate_calculator",
"//mediapipe/calculators/core:immediate_mux_calculator",
"//mediapipe/calculators/image:image_properties_calculator",
"//mediapipe/graphs/face_effect/subgraphs:single_face_geometry_from_detection_gpu",
"//mediapipe/graphs/face_effect/subgraphs:single_face_geometry_from_landmarks_gpu",
"//mediapipe/modules/face_geometry:effect_renderer_calculator",
"//mediapipe/modules/face_geometry:env_generator_calculator",
#################################
],
)In android dictionary
# Copyright 2019 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
licenses(["notice"])
package(default_visibility = ["//visibility:private"])
cc_binary(
name = "libmediapipe_jni.so",
linkshared = 1,
linkstatic = 1,
deps = [
"//mediapipe/graphs/iris_tracking:iris_tracking_gpu_deps",
"//mediapipe/java/com/google/mediapipe/framework/jni:mediapipe_framework_jni",
],
)
cc_library(
name = "mediapipe_jni_lib",
srcs = [":libmediapipe_jni.so"],
alwayslink = 1,
)
##################################################################
cc_binary(
name = "libmediapipe_effect_jni.so",
linkshared = 1,
linkstatic = 1,
deps = [
"//mediapipe/graphs/face_effect:face_effect_gpu_deps",
"//mediapipe/java/com/google/mediapipe/framework/jni:mediapipe_framework_jni",
],
)
cc_library(
name = "mediapipe_effect_jni_lib",
srcs = [":libmediapipe_effect_jni.so"],
alwayslink = 1,
)
##################################################################
android_binary(
name = "iristrackinggpu",
srcs = glob(["*.java"]),
assets = [
"//mediapipe/graphs/iris_tracking:iris_tracking_gpu.binarypb",
"//mediapipe/modules/face_landmark:face_landmark.tflite",
"//mediapipe/modules/iris_landmark:iris_landmark.tflite",
"//mediapipe/modules/face_detection:face_detection_front.tflite",
##################################################################
"//mediapipe/graphs/face_effect/data:plane.binarypb",
"//mediapipe/graphs/face_effect/data:plane.pngblob",
"//mediapipe/graphs/face_effect/data:line.binarypb",
"//mediapipe/graphs/face_effect/data:line.pngblob",
"//mediapipe/graphs/face_effect:face_effect_gpu.binarypb",
"//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata_detection.binarypb",
"//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata_landmarks.binarypb",
##################################################################
],
assets_dir = "",
manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",
manifest_values = {
"applicationId": "com.google.mediapipe.apps.iristrackinggpu",
"appName": "Iris Tracking",
"mainActivity": ".MainActivity",
"cameraFacingFront": "True",
"binaryGraphName": "iris_tracking_gpu.binarypb",
"inputVideoStreamName": "input_video",
"outputVideoStreamName": "output_video",
"flipFramesVertically": "True",
"converterNumBuffers": "2",
},
multidex = "native",
deps = [
###########################
":mediapipe_effect_jni_lib",
###########################
":mediapipe_jni_lib",
"//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:basic_lib",
"//mediapipe/framework/formats:landmark_java_proto_lite",
"//mediapipe/java/com/google/mediapipe/framework:android_framework",
"@com_google_protobuf//:protobuf_javalite",
],
)
brucechou1983
google-ml-butler[bot]
Platform: Linux, Ubuntu 18.04
I wish I could render face effect during Iris tracking project. what I have done yet was that modify the iris_tracking_gpu.pbtxt file to let IrisAndDepthRendererGpu calculator received the processed video after rendered. Beside that I copy the libraries and assets in BUILD file used by faceeffct project to iris_tracking project BUILD file. But I got the below ERROR when I Sync this project with new BUILD file.
I'm not sure about the correctness of this approach. Can someone told me which part I was missing or give me some suggestion about how to achieve that. Thanks in advance.