search

google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.
https://mediapipe.dev
Apache License 2.0
26.76k stars 5.08k forks source link

Set up iris landmark in python solution #1530

Closed overdescriptive closed 3 years ago

overdescriptive commented 3 years ago

Hi, I'm trying to run iris tracking (mediapipe/graphs/iris_tracking/iris_tracking_cpu.pbtxt) as a python solution (copied face_mesh solution). I've tried several things:

alexandre01 commented 3 years ago

I managed to install it via setup.py, but when instantiating IrisLandmark in Python, it throws the following error:

TfLiteInferenceCalculator: ; Either model as side packet or model path in options is required

All other modules compatible with Python use the InferenceCalculator calculator instead of TfLiteInferenceCalculator used in the Iris graph...

overdescriptive commented 3 years ago

Great to hear @alexandre01
Can you please share the files changed? (setup py and build files and if there's something else?)

overdescriptive commented 3 years ago

@alexandre01 In my case the it can't find other calculators:

RuntimeError: ValidatedGraphConfig Initialization failed.
No registered object with name: IrisLandmarkLeftAndRightCpu; Unable to find Calculator "IrisLandmarkLeftAndRightCpu"
No registered object with name: ConcatenateNormalizedLandmarkListCalculator; Unable to find Calculator "ConcatenateNormalizedLandmarkListCalculator"
No registered object with name: UpdateFaceLandmarksCalculator; Unable to find Calculator "UpdateFaceLandmarksCalculator"
No registered object with name: IrisRendererCpu; Unable to find Calculator "IrisRendererCpu"
No registered object with name: ConcatenateNormalizedLandmarkListCalculator; Unable to find Calculator "ConcatenateNormalizedLandmarkListCalculator"
GNUaquin commented 3 years ago

Have you now managed to get the example working in Python? I have already created the iris_tracking_cpu.binarypb and adapted a copy of the face_mesh solution. The files were copied manually into the python package. Now I am stuck in the same spot that the calculators cannot be found.

weiyiferonia commented 3 years ago

Hi, I'm trying to run iris tracking (mediapipe/graphs/iris_tracking/iris_tracking_cpu.pbtxt) as a python solution (copied face_mesh solution). I've tried several things:

  • Added mediapipe_simple_subgraph(name="iris_tracking_cpu", graph="iris_tracking_cpu.pbtxt", register_as="IrisTrackingCpu", deps = ["iris_tracking_cpu_deps"],) to the build, and manually building the binarypb. However, it doesn't work when running setup.py (it deletes it) and when not running it, it can't find the calculators.
  • Somehow moving the graph to modules directory? The script in cpp works great. Can you please help (maybe with an high level direction) for creating the iris python solution? Thanks!

I tried the same things but failed with following errors:

INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Traceback (most recent call last):
  File "F:/proj/MpExamples/iris_tracking.py", line 27, in <module>
    results = face_mesh.process(image)
  File "F:\baidu\AP\miniconda\lib\site-packages\mediapipe\python\solutions\iris.py", line 239, in process
    return super().process(input_data={'input_video': image})
  File "F:\baidu\AP\miniconda\lib\site-packages\mediapipe\python\solution_base.py", line 293, in process
    data).at(self._simulated_timestamp))
RuntimeError: Graph has errors: 
Calculator::Open() for node "irislandmarkleftandrightcpu__irislandmarkcpu_1__TfLiteTensorsToLandmarksCalculator_1" failed: ; Must provide input with/height for getting normalized landmarks.dth()dmarks_calculator.cc:151) 
Calculator::Open() for node "irislandmarkleftandrightcpu__irislandmarkcpu_1__TfLiteTensorsToLandmarksCalculator_2" failed: ; Must provide input with/height for getting normalized landmarks.dth()dmarks_calculator.cc:151) 
[ WARN:0] global C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-52oirelq\opencv\modules\videoio\src\cap_msmf.cpp (435) `anonymous-namespace'::SourceReaderCB::~SourceReaderCB terminating async callback
weiyiferonia commented 3 years ago

Hi, I'm trying to run iris tracking (mediapipe/graphs/iris_tracking/iris_tracking_cpu.pbtxt) as a python solution (copied face_mesh solution). I've tried several things:

  • Added mediapipe_simple_subgraph(name="iris_tracking_cpu", graph="iris_tracking_cpu.pbtxt", register_as="IrisTrackingCpu", deps = ["iris_tracking_cpu_deps"],) to the build, and manually building the binarypb. However, it doesn't work when running setup.py (it deletes it) and when not running it, it can't find the calculators.
  • Somehow moving the graph to modules directory? The script in cpp works great. Can you please help (maybe with an high level direction) for creating the iris python solution? Thanks!

I tried the same things but failed with following errors:

INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
Traceback (most recent call last):
  File "F:/proj/MpExamples/iris_tracking.py", line 27, in <module>
    results = face_mesh.process(image)
  File "F:\baidu\AP\miniconda\lib\site-packages\mediapipe\python\solutions\iris.py", line 239, in process
    return super().process(input_data={'input_video': image})
  File "F:\baidu\AP\miniconda\lib\site-packages\mediapipe\python\solution_base.py", line 293, in process
    data).at(self._simulated_timestamp))
RuntimeError: Graph has errors: 
Calculator::Open() for node "irislandmarkleftandrightcpu__irislandmarkcpu_1__TfLiteTensorsToLandmarksCalculator_1" failed: ; Must provide input with/height for getting normalized landmarks.dth()dmarks_calculator.cc:151) 
Calculator::Open() for node "irislandmarkleftandrightcpu__irislandmarkcpu_1__TfLiteTensorsToLandmarksCalculator_2" failed: ; Must provide input with/height for getting normalized landmarks.dth()dmarks_calculator.cc:151) 
[ WARN:0] global C:\Users\appveyor\AppData\Local\Temp\1\pip-req-build-52oirelq\opencv\modules\videoio\src\cap_msmf.cpp (435) `anonymous-namespace'::SourceReaderCB::~SourceReaderCB terminating async callback

Solved this by from mediapipe.calculators.tflite import tflite_tensors_to_landmarks_calculator_pb2

jiuqiant commented 3 years ago

Iris Python API can be implemented in the following way:

Step 1. Add iris.py in mediapipe/python/solutions:

# Copyright 2021 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""MediaPipe Iris."""

from typing import NamedTuple

import numpy as np

from mediapipe.python.solution_base import SolutionBase

BINARYPB_FILE_PATH = 'mediapipe/graphs/iris_tracking/iris_tracking_cpu.binarypb'

class Iris(SolutionBase):
  """MediaPipe Iris."""

  def __init__(self):
    super().__init__(
        binary_graph_path=BINARYPB_FILE_PATH,
        outputs=['face_landmarks_with_iris'])

  def process(self, image: np.ndarray) -> NamedTuple:
    return super().process(input_data={'input_video': image})

Step 2. Modify the following files: 

diff --git a/MANIFEST.in b/MANIFEST.in
index ba8014d..33881c9 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -7,6 +7,8 @@ include MANIFEST.in
 include README.md
 include requirements.txt

+recursive-include mediapipe/models *.tflite *.txt *.binarypb
+recursive-include mediapipe/graphs *.tflite *.txt *.binarypb
 recursive-include mediapipe/modules *.tflite *.txt *.binarypb
 exclude mediapipe/modules/objectron/object_detection_3d_chair_1stage.tflite
 exclude mediapipe/modules/objectron/object_detection_3d_sneakers_1stage.tflite

diff --git a/mediapipe/graphs/iris_tracking/BUILD b/mediapipe/graphs/iris_tracking/BUILD
index 86e667b..9887800 100644
--- a/mediapipe/graphs/iris_tracking/BUILD
+++ b/mediapipe/graphs/iris_tracking/BUILD
@@ -84,3 +84,10 @@ mediapipe_binary_graph(
     output_name = "iris_tracking_gpu.binarypb",
     deps = [":iris_tracking_gpu_deps"],
 )
+
+mediapipe_binary_graph(
+    name = "iris_tracking_cpu_binary_graph",
+    graph = "iris_tracking_cpu.pbtxt",
+    output_name = "iris_tracking_cpu.binarypb",
+    deps = [":iris_tracking_cpu_deps"],
+)
diff --git a/mediapipe/python/BUILD b/mediapipe/python/BUILD
index 3bd188d..1dd8450 100644
--- a/mediapipe/python/BUILD
+++ b/mediapipe/python/BUILD
@@ -71,5 +71,6 @@ cc_library(
         "//mediapipe/modules/pose_detection:pose_detection_cpu",
         "//mediapipe/modules/pose_landmark:pose_landmark_by_roi_cpu",
         "//mediapipe/modules/pose_landmark:pose_landmark_cpu",
+        "//mediapipe/graphs/iris_tracking:iris_tracking_cpu_deps",
     ],
 )
diff --git a/mediapipe/python/solution_base.py b/mediapipe/python/solution_base.py
index 1bed385..2da0669 100644
--- a/mediapipe/python/solution_base.py
+++ b/mediapipe/python/solution_base.py
@@ -228,7 +228,7 @@ class SolutionBase:
       self._modify_calculator_options(canonical_graph_config_proto,
                                       calculator_params)
     self._graph = calculator_graph.CalculatorGraph(
-        graph_config=canonical_graph_config_proto)
+        binary_graph_path=os.path.join(root_path, binary_graph_path))
     self._simulated_timestamp = 0
     self._graph_outputs = {}

diff --git a/mediapipe/python/solutions/__init__.py b/mediapipe/python/solutions/__init__.py
index 8cd9af3..5aa53a3 100644
--- a/mediapipe/python/solutions/__init__.py
+++ b/mediapipe/python/solutions/__init__.py
@@ -17,6 +17,7 @@
 import mediapipe.python.solutions.drawing_utils
 import mediapipe.python.solutions.face_detection
 import mediapipe.python.solutions.face_mesh
+import mediapipe.python.solutions.iris
 import mediapipe.python.solutions.hands
 import mediapipe.python.solutions.holistic
 import mediapipe.python.solutions.objectron
diff --git a/setup.py b/setup.py
index 5acad7b..ea402c9 100644
--- a/setup.py
+++ b/setup.py
@@ -44,7 +44,10 @@ SUBDIR_INIT_PY_FILES = [
     os.path.join(MP_ROOT_PATH, 'mediapipe/modules/__init__.py'),
     os.path.join(MP_ROOT_PATH,
                  'mediapipe/modules/holistic_landmark/__init__.py'),
-    os.path.join(MP_ROOT_PATH, 'mediapipe/modules/objectron/__init__.py')
+    os.path.join(MP_ROOT_PATH, 'mediapipe/modules/objectron/__init__.py'),
+    os.path.join(MP_ROOT_PATH, 'mediapipe/graphs/__init__.py'),
+    os.path.join(MP_ROOT_PATH, 'mediapipe/graphs/iris_tracking/__init__.py'),
+    os.path.join(MP_ROOT_PATH, 'mediapipe/graphs/iris_tracking/calculators__init__.py')
 ]
 if not os.path.exists(ROOT_INIT_PY):
   open(ROOT_INIT_PY, 'w').close()
@@ -177,7 +180,7 @@ class GeneratePyProtos(setuptools.Command):
     for pattern in [
         'mediapipe/framework/**/*.proto', 'mediapipe/calculators/**/*.proto',
         'mediapipe/gpu/**/*.proto', 'mediapipe/modules/**/*.proto',
-        'mediapipe/util/**/*.proto'
+        'mediapipe/util/**/*.proto' 'mediapipe/graphs/iris_tracking/calculators/*.proto'
     ]:
       for proto_file in glob.glob(pattern, recursive=True):
         # Ignore test protos.
@@ -230,6 +233,20 @@ class BuildBinaryGraphs(build.build):
       sys.stderr.write('generating binarypb: %s\n' %
                        os.path.join('mediapipe/modules/', binary_graph))
       self._generate_binary_graph(binary_graph)
+     
+    # iris graph
+    bazel_command = [
+        'bazel',
+        'build',
+        '--compilation_mode=opt',
+        '--define=MEDIAPIPE_DISABLE_GPU=1',
+        '--action_env=PYTHON_BIN_PATH=' + _normalize_path(sys.executable),
+        'mediapipe/graphs/iris_tracking/iris_tracking_cpu_binary_graph',
+    ]
+    output_name = 'iris_tracking/iris_tracking_cpu.binarypb'
+    output_file = os.path.join('mediapipe/graphs', output_name)
+    shutil.copyfile(
+        os.path.join('bazel-bin/mediapipe/graphs/', output_name), output_file)

   def _generate_binary_graph(self, graph_path):
     """Generate binary graph for a particular MediaPipe binary graph target."""

Step 3. Build MediaPipe Python by following https://google.github.io/mediapipe/getting_started/python#building-mediapipe-python-package.

Step 4. Run

import mediapipe as mp
import cv2
mp_iris = mp.solutions.iris
iris = mp_iris.Iris()
image = cv2.imread(file)
results = iris.process(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
print(results.face_landmarks_with_iris)
iris.close()

You can find the eye landmarks indices at https://github.com/google/mediapipe/blob/350fbb2100ad531bc110b93aaea23d96af5a5064/mediapipe/graphs/iris_tracking/calculators/update_face_landmarks_calculator.cc#L33 For example, you can get one of the left eye lower contour coordinates by print(results.face_landmarks_with_iris.landmark[33]).

GNUaquin commented 3 years ago

@jiuqiant Thanks a lot! It works.

jiuqiant commented 3 years ago

Glad to hear that! I am going to close this issue.

overdescriptive commented 3 years ago

@jiuqiant Works for me as well. thanks a lot!

ghost commented 3 years ago

Step 3. Build MediaPipe Python by following https://google.github.io/mediapipe/getting_started/python#building-mediapipe-python-package.

Why am i facing such a problem? Снимок

fredrikmalmberg commented 3 years ago

Step 3. Build MediaPipe Python by following https://google.github.io/mediapipe/getting_started/python#building-mediapipe-python-package.

Why am i facing such a problem? Снимок

Getting the same problem

RafalSiwek commented 3 years ago

@fredrikmalmberg, @Egorovlvan Hi! I had the same issue, so I had to review @jiuqiant 's git diff. I have spotted some places where changes had to be applied:

diff --git a/MANIFEST.in b/MANIFEST.in
index ba8014d..33881c9 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -7,6 +7,8 @@ include MANIFEST.in
 include README.md
 include requirements.txt

+recursive-include mediapipe/models *.tflite *.txt *.binarypb
+recursive-include mediapipe/graphs *.tflite *.txt *.binarypb
 recursive-include mediapipe/modules *.tflite *.txt *.binarypb
 exclude mediapipe/modules/objectron/object_detection_3d_chair_1stage.tflite
 exclude mediapipe/modules/objectron/object_detection_3d_sneakers_1stage.tflite
diff --git a/mediapipe/graphs/iris_tracking/BUILD b/mediapipe/graphs/iris_tracking/BUILD
index 86e667b..220737a 100644
--- a/mediapipe/graphs/iris_tracking/BUILD
+++ b/mediapipe/graphs/iris_tracking/BUILD
@@ -84,3 +84,11 @@ mediapipe_binary_graph(
     output_name = "iris_tracking_gpu.binarypb",
     deps = [":iris_tracking_gpu_deps"],
 )
+mediapipe_binary_graph(
+    name = "iris_tracking_cpu_binary_graph",
+    graph = "iris_tracking_cpu.pbtxt",
+    output_name = "iris_tracking_cpu.binarypb",
+    deps = [":iris_tracking_cpu_deps"],
+)
+
+
diff --git a/mediapipe/python/BUILD b/mediapipe/python/BUILD
index 3bd188d..1dd8450 100644
--- a/mediapipe/python/BUILD
+++ b/mediapipe/python/BUILD
@@ -71,5 +71,6 @@ cc_library(
         "//mediapipe/modules/pose_detection:pose_detection_cpu",
         "//mediapipe/modules/pose_landmark:pose_landmark_by_roi_cpu",
         "//mediapipe/modules/pose_landmark:pose_landmark_cpu",
+        "//mediapipe/graphs/iris_tracking:iris_tracking_cpu_deps",
     ],
 )
diff --git a/mediapipe/python/solution_base.py b/mediapipe/python/solution_base.py
index 7b535c8..1991b19 100644
--- a/mediapipe/python/solution_base.py
+++ b/mediapipe/python/solution_base.py
@@ -234,7 +234,7 @@ class SolutionBase:
       self._modify_calculator_options(canonical_graph_config_proto,
                                       calculator_params)
     self._graph = calculator_graph.CalculatorGraph(
-        graph_config=canonical_graph_config_proto)
+        binary_graph_path=os.path.join(root_path, binary_graph_path))
     self._simulated_timestamp = 0
     self._graph_outputs = {}

diff --git a/mediapipe/python/solutions/__init__.py b/mediapipe/python/solutions/__init__.py
index 8cd9af3..5aa53a3 100644
--- a/mediapipe/python/solutions/__init__.py
+++ b/mediapipe/python/solutions/__init__.py
@@ -17,6 +17,7 @@
 import mediapipe.python.solutions.drawing_utils
 import mediapipe.python.solutions.face_detection
 import mediapipe.python.solutions.face_mesh
+import mediapipe.python.solutions.iris
 import mediapipe.python.solutions.hands
 import mediapipe.python.solutions.holistic
 import mediapipe.python.solutions.objectron
diff --git a/setup.py b/setup.py
index 8a4c712..2a700e9 100644
--- a/setup.py
+++ b/setup.py
@@ -44,7 +44,11 @@ SUBDIR_INIT_PY_FILES = [
     os.path.join(MP_ROOT_PATH, 'mediapipe/modules/__init__.py'),
     os.path.join(MP_ROOT_PATH,
                  'mediapipe/modules/holistic_landmark/__init__.py'),
-    os.path.join(MP_ROOT_PATH, 'mediapipe/modules/objectron/__init__.py')
+    os.path.join(MP_ROOT_PATH, 'mediapipe/modules/objectron/__init__.py'),
+    os.path.join(MP_ROOT_PATH, 'mediapipe/graphs/__init__.py'),
+    os.path.join(MP_ROOT_PATH, 'mediapipe/graphs/iris_tracking/__init__.py'),
+    os.path.join(MP_ROOT_PATH, 'mediapipe/graphs/iris_tracking/calculators/__init__.py')
+
 ]
 if not os.path.exists(ROOT_INIT_PY):
   open(ROOT_INIT_PY, 'w').close()
@@ -177,7 +181,7 @@ class GeneratePyProtos(setuptools.Command):
     for pattern in [
         'mediapipe/framework/**/*.proto', 'mediapipe/calculators/**/*.proto',
         'mediapipe/gpu/**/*.proto', 'mediapipe/modules/**/*.proto',
-        'mediapipe/util/**/*.proto'
+        'mediapipe/util/**/*.proto', 'mediapipe/graphs/iris_tracking/calculators/*.proto'
     ]:
       for proto_file in glob.glob(pattern, recursive=True):
         proto_dir = os.path.dirname(os.path.abspath(proto_file))
@@ -231,6 +235,24 @@ class BuildBinaryGraphs(build.build):
                        os.path.join('mediapipe/modules/', binary_graph))
       self._generate_binary_graph(binary_graph)

+    # iris graph
+    bazel_command = [
+      'bazel',
+      'build',
+      '--compilation_mode=opt',
+      '--define=MEDIAPIPE_DISABLE_GPU=1',
+      '--action_env=PYTHON_BIN_PATH=' + _normalize_path(sys.executable),
+      'mediapipe/graphs/iris_tracking/iris_tracking_cpu_binary_graph',
+    ]
+    if not self.link_opencv and not IS_WINDOWS:
+      bazel_command.append('--define=OPENCV=source')
+    if subprocess.call(bazel_command) != 0:
+      sys.exit(-1)
+    output_name = 'iris_tracking/iris_tracking_cpu.binarypb'
+    output_file = os.path.join('mediapipe/graphs', output_name)
+    shutil.copyfile(
+      os.path.join('bazel-bin/mediapipe/graphs/', output_name), output_file)
+
   def _generate_binary_graph(self, graph_path):
     """Generate binary graph for a particular MediaPipe binary graph target."""

diff --git a/third_party/BUILD b/third_party/BUILD
index 654f0cb..460ecca 100644
--- a/third_party/BUILD
+++ b/third_party/BUILD
@@ -90,7 +90,7 @@ OPENCV_MODULES = [
 # still only builds the shared libraries, so we have to choose one or the
 # other. We build shared libraries by default, but this variable can be used
 # to switch to static libraries.
-OPENCV_SHARED_LIBS = True
+OPENCV_SHARED_LIBS = False

 OPENCV_SO_VERSION = "3.4"

Maybe this problem occured due to the new repo commit. Moreover it is crutial to follow the steps mentioned in step 3 by @jiuqiant and after applying: python3 setup.py gen_protos in bash, it is crutial to remove the protobuf framework from the system and leave that one installed at mp_env with pip3.

After all you might be able to visualize it just like in the hand-keypoints-detection example here And the easiest way (and least elegant) to do this is to add a function in mp_env/lib/python3.9/site-packages/mediapipe/python/solutions/drawing_utils.py:

def draw_iris_landmarks(
    image: np.ndarray,
    landmark_list: landmark_pb2.NormalizedLandmarkList,
    eye_key_indicies=[
        # Left eye
      # eye lower contour
      33,
      7,
      163,
      144,
      145,
      153,
      154,
      155,
      133,
      # eye upper contour (excluding corners)
      246,
      161,
      160,
      159,
      158,
      157,
      173,

      # Right eye
      # eye lower contour
      263,
      249,
      390,
      373,
      374,
      380,
      381,
      382,
      362,
      # eye upper contour (excluding corners)
      466,
      388,
      387,
      386,
      385,
      384,
      398

    ],
    iris_key_indicies=[

        # Left iris points
      468, # middle  
      469, # right 
      470, # up 
      471, # left 
      472, # down

        # Right iris points
      473, # middle 
      474, # left 
      475, # upper 
      476, # right 
      477 # lower 

    ],
    eye_drwing_color=(255,0,0),
    iris_drawing_color=(0,0,255)
    ):

  if not landmark_list:
    return
  if image.shape[2] != RGB_CHANNELS:
    raise ValueError('Input image must contain three channel rgb data.')
  image_rows, image_cols, _ = image.shape
  idx_to_eye_coordinates = {}
  idx_to_iris_coordinates = {}
  for idx, landmark in enumerate(landmark_list.landmark):

    landmark_px = _normalized_to_pixel_coordinates(landmark.x, landmark.y,
                                                   image_cols, image_rows)
    if landmark_px and idx in eye_key_indicies:
      idx_to_eye_coordinates[idx]= landmark_px

    elif landmark_px and idx in iris_key_indicies:
      idx_to_iris_coordinates[idx]= landmark_px

  for landmark_px in idx_to_eye_coordinates.values():
    cv2.circle(image, landmark_px, 2,
               eye_drwing_color, 1)

  for landmark_px in idx_to_iris_coordinates.values():
    cv2.circle(image, landmark_px, 2,
               iris_drawing_color, 1)

And write a code like this:

import mediapipe as mp
import cv2
mp_iris = mp.solutions.iris
mp_draw = mp.solutions.drawing_utils

cap = cv2.VideoCapture(0)
with mp_iris.Iris() as iris:
  while cap.isOpened():
    success, image = cap.read()
    if not success:
      print("Ignoring empty camera frame.")
      # If loading a video, use 'break' instead of 'continue'.
      continue

    # Flip the image horizontally for a later selfie-view display, and convert
    # the BGR image to RGB.
    image = cv2.cvtColor(cv2.flip(image, 1), cv2.COLOR_BGR2RGB)
    # To improve performance, optionally mark the image as not writeable to
    # pass by reference.
    image.flags.writeable = False
    results = iris.process(image)

    # Draw the eye and iris annotations on the image.
    image.flags.writeable = True
    image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)
    if results.face_landmarks_with_iris:
        mp_draw.draw_iris_landmarks(image,results.face_landmarks_with_iris)
    cv2.imshow('MediaPipe Iris', image)
    if cv2.waitKey(5) & 0xFF == 27:
      break
cap.release()
markson14 commented 3 years ago

@RafalSiwek Thanks for the sum up. It does work.

DSgUY commented 3 years ago

Can anyone explain an updated guide to make Iris work on python?. I generate the protos with python setup.py gen_protos but then get and error when python setup.py install --link-opencv

ERROR: An error occurred during the fetch of repository 'local_execution_config_python': ERROR: Error fetching repository: Traceback (most recent call last): ERROR: Analysis of target '//mediapipe/modules/face_detection:face_detection_front_cpu' failed; build aborted: Repository command failed INFO: Could not find files for the given pattern(s). INFO: Elapsed time: 0.248s INFO: 0 processes. FAILED: Build did NOT complete successfully (0 packages loaded, 0 targets configured)

julienguegan commented 3 years ago

Does this work only on Ubuntu/Mac @jiuqiant @RafalSiwek ? I am struggling on Windows at the last step when building with "python setup.py bdist_wheel" it returns ERROR : C:/users/gueganj/mediapipe/WORKSPACE:260:23: fetching android_ndk_repository rule //external:androidndk: Either the path attribute of android_ndk_repository or the ANDROID_NDK_HOME environment variable must be set. I see that there is a file named setup_android_sdk_and_ndk.sh that could help but it is a bash script and I don't really need to make mediapipe worked on android ...

Even so, I have tried to follow exactly the steps mentioned by @RafalSiwek on a ubuntu VM and this time, I succeeded to build mediapipe but when calling mediapipe.solutions in my python env, it give me module 'mediapipe' has no attribute 'solutions' ... If someone as an idea, I will be glad to hear it !

Thank you

mertkslkc commented 3 years ago

@fredrikmalmberg , @Egorovlvan Merhaba! Ben de aynı sorunu yaşadım, bu yüzden @jiuqiant'ın git diff'ini incelemem gerekti . Değişikliklerin uygulanması gereken bazı yerler tespit ettim:

diff --git a/MANIFEST.in b/MANIFEST.in
dizin ba8014d..33881c9 100644
--- a/MANIFEST.in 
+++ b/MANIFEST.in 
@@ -7,6 +7,8 @@ MANIFEST.in içerir
 README.md'yi dahil et
 gereksinimleri.txt'i dahil et

+ özyinelemeli-mediapipe/modelleri içerir *.tflite *.txt *.binarypb 
+ özyinelemeli-içerir mediapipe/graphs *.tflite *.txt *.binarypb
 özyinelemeli-mediapipe/modülleri içerir *.tflite *.txt *.binarypb
 mediapipe/modules/objectron/object_detection_3d_chair_1stage.tflite'ı hariç tut
 mediapipe/modules/objectron/object_detection_3d_sneakers_1stage.tflite'ı hariç tut
diff --git a/mediapipe/graphs/iris_tracking/BUILD b/mediapipe/graphs/iris_tracking/BUILD
dizin 86e667b..220737a 100644
--- a/mediapipe/graphs/iris_tracking/BUILD 
+++ b/mediapipe/graphs/iris_tracking/BUILD 
@@ -84,3 +84,11 @@ mediapipe_binary_graph(
     output_name = "iris_tracking_gpu.binarypb",
     deps = [":iris_tracking_gpu_deps"],
 )
+ Mediapipe_binary_graph ( 
+     adı = "iris_tracking_cpu_binary_graph", 
+     grafik = "iris_tracking_cpu.pbtxt", 
+     output_name = "iris_tracking_cpu.binarypb", 
+     deps = [ ": iris_tracking_cpu_deps"], 
+ ) 
+ 
+
 fark --git a / mediapipe / python/BUILD b/mediapipe/python/BUILD
dizin 3bd188d..1dd8450 100644
--- a/mediapipe/python/BUILD 
+++ b/mediapipe/python/BUILD 
@@ -71,5 +71,6 @@ cc_library(
         "//mediapipe/modules/pose_detection:pose_detection_cpu",
         "//mediapipe/modules/pose_landmark:pose_landmark_by_roi_cpu",
         "//mediapipe/modules/pose_landmark:pose_landmark_cpu",
+         "//mediapipe/graphs/iris_tracking:iris_tracking_cpu_deps",
     ],
 )
diff --git a/mediapipe/python/solution_base.py b/mediapipe/python/solution_base.py
dizin 7b535c8..1991b19 100644
--- a/mediapipe/python/solution_base.py 
+++ b/mediapipe/python/solution_base.py 
@@ -234,7 +234,7 @@ class SolutionBase:
       self._modify_calculator_options(canonical_graph_config_proto,
                                       hesaplayıcı_params)
     self._graph = calculator_graph.CalculatorGraph(
-         graph_config=canonical_graph_config_proto) 
+         binary_graph_path=os.path.join(root_path, binary_graph_path))
     self._simulated_timestamp = 0
     self._graph_outputs = {}

diff --git a/mediapipe/python/solutions/__init__.py b/mediapipe/python/solutions/__init__.py
dizin 8cd9af3..5aa53a3 100644
--- a/mediapipe/python/solutions/__init__.py 
+++ b/mediapipe/python/solutions/__init__.py 
@@ -17,6 +17,7 @@
 mediapipe.python.solutions.drawing_utils'i içe aktar
 mediapipe.python.solutions.face_detection'ı içe aktar
 mediapipe.python.solutions.face_mesh'i içe aktar
+ mediapipe.python.solutions.iris'i içe aktar
 mediapipe.python.solutions.hands'i içe aktar
 mediapipe.python.solutions.holistic'i içe aktar
 mediapipe.python.solutions.objectron'u içe aktar
diff --git a/setup.py b/setup.py
dizin 8a4c712..2a700e9 100644
--- a/setup.py 
+++ b/setup.py 
@@ -44,7 +44,11 @@ SUBDIR_INIT_PY_FILES = [
     os.path.join(MP_ROOT_PATH, 'mediapipe/modules/__init__.py'),
     os.path.join(MP_ROOT_PATH,
                  'mediapipe/modules/holistic_landmark/__init__.py'),
-     os.path.join(MP_ROOT_PATH, 'mediapipe/modules/objectron/__init__.py') 
+     os.path.join(MP_ROOT_PATH, 'mediapipe/modules/objectron/__init__.py'), 
+     os.path.join( MP_ROOT_PATH, 'mediapipe/graphs/__init__.py'), 
+     os.path.join(MP_ROOT_PATH, 'mediapipe/graphs/iris_tracking/__init__.py'), 
+     os.path.join(MP_ROOT_PATH, 'mediapipe/graphs/iris_tracking /hesap makineleri/__init__.py') 
+
 ]
 os.path.exists(ROOT_INIT_PY) değilse:
   open(ROOT_INIT_PY, 'w').close()
@@ -177,7 +181,7 @@ sınıfı GeneratePyProtos(setuptools.Command):
     desen için [
         'mediapipe/framework/**/*.proto', 'mediapipe/hesaplayıcılar/**/*.proto',
         'mediapipe/gpu/**/*.proto', 'mediapipe/modules/**/*.proto',
-         'mediapipe/util/**/*.proto' 
+         'mediapipe/util/**/*.proto', 'mediapipe/graphs/iris_tracking/calculators/*.proto'
     ]:
       glob.glob'daki proto_file için(desen, özyinelemeli=Doğru):
         proto_dir = os.path.dirname(os.path.abspath(proto_file))
@@ -231,6 +235,24 @@ sınıfı BuildBinaryGraphs(build.build):
                        os.path.join('mediapipe/modules/', binary_graph))
       self._generate_binary_graph(binary_graph)

+     # iris grafiği 
+     bazel_command = [ 
+       'bazel', 
+       'build', 
+       '--compilation_mode=opt', 
+       '--define=MEDIAPIPE_DISABLE_GPU=1', 
+       '--action_env=PYTHON_BIN_PATH=' + _normalize_path( sys.executable), 
+       'mediapipe/graphs/iris_tracking/iris_tracking_cpu_binary_graph', 
+     ] 
+     self.link_opencv değilse ve IS_WINDOWS değilse: 
+       bazel_command.append('--define=OPENCV=source') 
+     subprocess.call(bazel_com) ise ) != 0: 
+       sys.exit(-1) 
+    output_name = 'iris_tracking/iris_tracking_cpu.binarypb' 
+
 output_file     = os.path.join('mediapipe/graphs', output_name) +     Shutil.copyfile( 
+       os.path.join('bazel-bin/mediapipe/graphs/', output_name) ), çıktı_dosyası) 
+
   def _generate_binary_graph(self, graph_path):
     """Belirli bir MediaPipe ikili grafik hedefi için ikili grafik oluştur."""

diff --git a/ Third_party/BUILD b/ Third_party/BUILD
dizin 654f0cb..460ecca 100644
--- a/üçüncü taraf/BUILD 
+++ b/ 
üçüncü taraf /BUILD @@ -90,7 +90,7 @@ OPENCV_MODULES = [
 # hala yalnızca paylaşılan kitaplıkları oluşturur, bu nedenle birini veya
 # diğer. Varsayılan olarak paylaşılan kitaplıklar oluşturuyoruz, ancak bu değişken kullanılabilir
 # statik kitaplıklara geçmek için.
- OPENCV_SHARED_LIBS = Doğru 
+ OPENCV_SHARED_LIBS = Yanlış

 OPENCV_SO_VERSION = "3.4"

Belki de bu sorun yeni repo taahhüdü nedeniyle ortaya çıkmıştır. Ayrıca @jiuqiant tarafından 3. adımda belirtilen adımları takip etmek ve uyguladıktan sonra çok önemlidir : bash'ta protobuf python3 setup.py gen_protos çerçevesini sistemden çıkarmak ve mp_env'de kurulu olanı pip3 ile bırakmak çok önemlidir.

Sonuçta sadece el keypoints algılama örnekte olduğu gibi görselleştirmek mümkün olabilir burada Ve bunu yapmanın en kolay yolu (ve az zarif) mp_env / lib / python3.9 / site paketlerinde bir işlev eklemektir / mediapipe/python/solutions/drawing_utils.py:

def  draw_iris_landmarks (
     resim : np . ndarray ,
     landmark_list : landmark_pb2 . NormalizedLandmarkList ,
     eye_key_indicies = [
         # Sol göz 
      # göz alt kontur 
      33 ,
       7 ,
       163 ,
       144 ,
       145 ,
       153 ,
       154 ,
       155 ,
       133 ,
       # gözü üst kontur (hariç köşeler ) 
      246 ,
       161,
       160 ,
       159 ,
       158 ,
       157 ,
       173 ,

      # Sağ göz 
      # göz alt konturu 
      263 ,
       249 ,
       390 ,
       373 ,
       374 ,
       380 ,
       381 ,
       382 ,
       362 ,
       # göz üst konturu (köşeler hariç) 
      466 ,
       388 ,
       387 ,
       386 ,
       385 ,
       384 ,
       398

    ],
    iris_key_indicies = [

        # Sol iris noktaları 
      468 , # orta   
      469 , # sağ 
      470 , # yukarı 
      471 , # sol 
      472 , # aşağı

        # Sağ iris noktaları 
      473 , # orta 
      474 , # sol 
      475 , # üst 
      476 , # sağ 
      477  # alt

    ],
    eye_drwing_color = ( 255 , 0 , 0 ),
     iris_drawing_color = ( 0 , 0 , 255 )
    ):

  eğer  değil  landmark_list :
     dönüş 
  eğer  resmin . şekil [ 2 ] ! =  RGB_CHANNELS :
     kaldırmak  ValueError ( 'giriş görüntüsü üç kanal RGB verileri içermelidir.' )
   image_rows , image_cols , _  =  görüntüsü . şekil 
  idx_to_eye_coordinates  = {}
   idx_to_iris_coordinates  = {}
   için  IDX , dönüm  olarak  enumerate ( landmark_list .yer işareti ):

    landmark_px  =  _normalized_to_pixel_coordinates ( dönüm . X , merkezi nokta . y ,
                                                    image_cols , image_rows )
     halinde  landmark_px  ve  IDX  içinde  eye_key_indicies :
       idx_to_eye_coordinates [ IDX ] =  landmark_px

    Elif  landmark_px  ve  IDX  içinde  iris_key_indicies :
       idx_to_iris_coordinates [ IDX ] =  landmark_px

  için  landmark_px  içinde  idx_to_eye_coordinates . değerler ():
     cv2 . çember ( resim , landmark_px , 2 ,
                eye_drwing_color , 1 )

  için  landmark_px  içinde  idx_to_iris_coordinates . değerler ():
     cv2 . çember ( resim , landmark_px , 2 ,
                iris_drawing_color , 1 )

Ve şöyle bir kod yazın:

içe  mediapipe  olarak  en 
içe  CV2 
mp_iris  =  erime noktası . çözümler . iris 
mp_draw  =  mp . çözümler . çizim_utils

kap  =  cv2 . mp_iris ile VideoCapture ( 0 )
 . iris () as iris :
   while cap . isOpened ():
     başarı , resim = cap . okumak ()
     eğer değil başarı :
       print ( "boş kamera çerçevesini görmezden gelmek." )
       # bir video yükleme, kullanmayı 'mola' yerine 'devam' arasında. devam et       

    # Daha sonraki bir selfie görünümü için görüntüyü yatay olarak çevirin ve 
    # BGR görüntüsünü RGB'ye 
    dönüştürün . resim  =  cv2 . cvtColor ( cv2 . flip ( görüntü , 1 ), cv2 . COLOR_BGR2RGB )
     # Performansı artırmak için, isteğe bağlı olarak görüntüyü 
    referansa göre # iletmeye 
    yazılamaz olarak işaretleyin . görüntü . bayraklar . yazılabilir  =  Yanlış 
    sonuçlar  =  iris . süreç ( resim )

    # Resmin üzerine göz ve iris açıklamalarını çizin. 
    görüntü . bayraklar . yazılabilir  =  Gerçek 
    görüntü  =  cv2 . cvtColor ( resim , cv2 . COLOR_RGB2BGR )
     eğer  sonuçlarının . face_landmarks_with_iris :
         mp_draw . draw_iris_landmarks ( resim , sonuçlar . face_landmarks_with_iris )
     cv2 . imshow ( 'MediaPipe Iris' , resim )
     eğer özgeçmiş2 . waitKey ( 5 ) ve  0xFF  ==  27 :
       ara 
kap . serbest bırakmak ()

Can you share the edited files?

mertkslkc commented 3 years ago

Iris Python API aşağıdaki şekilde uygulanabilir:

Adım 1. iris.pyMediapipe/python/solutions'ı ekleyin :

# Copyright 2021 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""MediaPipe Iris."""

from typing import NamedTuple

import numpy as np

from mediapipe.python.solution_base import SolutionBase

BINARYPB_FILE_PATH = 'mediapipe/graphs/iris_tracking/iris_tracking_cpu.binarypb'

class Iris(SolutionBase):
  """MediaPipe Iris."""

  def __init__(self):
    super().__init__(
        binary_graph_path=BINARYPB_FILE_PATH,
        outputs=['face_landmarks_with_iris'])

  def process(self, image: np.ndarray) -> NamedTuple:
    return super().process(input_data={'input_video': image})

Adım 2. Aşağıdaki dosyaları değiştirin:

diff --git a/MANIFEST.in b/MANIFEST.in
index ba8014d..33881c9 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -7,6 +7,8 @@ include MANIFEST.in
 include README.md
 include requirements.txt

+recursive-include mediapipe/models *.tflite *.txt *.binarypb
+recursive-include mediapipe/graphs *.tflite *.txt *.binarypb
 recursive-include mediapipe/modules *.tflite *.txt *.binarypb
 exclude mediapipe/modules/objectron/object_detection_3d_chair_1stage.tflite
 exclude mediapipe/modules/objectron/object_detection_3d_sneakers_1stage.tflite

diff --git a/mediapipe/graphs/iris_tracking/BUILD b/mediapipe/graphs/iris_tracking/BUILD
index 86e667b..9887800 100644
--- a/mediapipe/graphs/iris_tracking/BUILD
+++ b/mediapipe/graphs/iris_tracking/BUILD
@@ -84,3 +84,10 @@ mediapipe_binary_graph(
     output_name = "iris_tracking_gpu.binarypb",
     deps = [":iris_tracking_gpu_deps"],
 )
+
+mediapipe_binary_graph(
+    name = "iris_tracking_cpu_binary_graph",
+    graph = "iris_tracking_cpu.pbtxt",
+    output_name = "iris_tracking_cpu.binarypb",
+    deps = [":iris_tracking_cpu_deps"],
+)
diff --git a/mediapipe/python/BUILD b/mediapipe/python/BUILD
index 3bd188d..1dd8450 100644
--- a/mediapipe/python/BUILD
+++ b/mediapipe/python/BUILD
@@ -71,5 +71,6 @@ cc_library(
         "//mediapipe/modules/pose_detection:pose_detection_cpu",
         "//mediapipe/modules/pose_landmark:pose_landmark_by_roi_cpu",
         "//mediapipe/modules/pose_landmark:pose_landmark_cpu",
+        "//mediapipe/graphs/iris_tracking:iris_tracking_cpu_deps",
     ],
 )
diff --git a/mediapipe/python/solution_base.py b/mediapipe/python/solution_base.py
index 1bed385..2da0669 100644
--- a/mediapipe/python/solution_base.py
+++ b/mediapipe/python/solution_base.py
@@ -228,7 +228,7 @@ class SolutionBase:
       self._modify_calculator_options(canonical_graph_config_proto,
                                       calculator_params)
     self._graph = calculator_graph.CalculatorGraph(
-        graph_config=canonical_graph_config_proto)
+        binary_graph_path=os.path.join(root_path, binary_graph_path))
     self._simulated_timestamp = 0
     self._graph_outputs = {}

diff --git a/mediapipe/python/solutions/__init__.py b/mediapipe/python/solutions/__init__.py
index 8cd9af3..5aa53a3 100644
--- a/mediapipe/python/solutions/__init__.py
+++ b/mediapipe/python/solutions/__init__.py
@@ -17,6 +17,7 @@
 import mediapipe.python.solutions.drawing_utils
 import mediapipe.python.solutions.face_detection
 import mediapipe.python.solutions.face_mesh
+import mediapipe.python.solutions.iris
 import mediapipe.python.solutions.hands
 import mediapipe.python.solutions.holistic
 import mediapipe.python.solutions.objectron
diff --git a/setup.py b/setup.py
index 5acad7b..ea402c9 100644
--- a/setup.py
+++ b/setup.py
@@ -44,7 +44,10 @@ SUBDIR_INIT_PY_FILES = [
     os.path.join(MP_ROOT_PATH, 'mediapipe/modules/__init__.py'),
     os.path.join(MP_ROOT_PATH,
                  'mediapipe/modules/holistic_landmark/__init__.py'),
-    os.path.join(MP_ROOT_PATH, 'mediapipe/modules/objectron/__init__.py')
+    os.path.join(MP_ROOT_PATH, 'mediapipe/modules/objectron/__init__.py'),
+    os.path.join(MP_ROOT_PATH, 'mediapipe/graphs/__init__.py'),
+    os.path.join(MP_ROOT_PATH, 'mediapipe/graphs/iris_tracking/__init__.py'),
+    os.path.join(MP_ROOT_PATH, 'mediapipe/graphs/iris_tracking/calculators__init__.py')
 ]
 if not os.path.exists(ROOT_INIT_PY):
   open(ROOT_INIT_PY, 'w').close()
@@ -177,7 +180,7 @@ class GeneratePyProtos(setuptools.Command):
     for pattern in [
         'mediapipe/framework/**/*.proto', 'mediapipe/calculators/**/*.proto',
         'mediapipe/gpu/**/*.proto', 'mediapipe/modules/**/*.proto',
-        'mediapipe/util/**/*.proto'
+        'mediapipe/util/**/*.proto' 'mediapipe/graphs/iris_tracking/calculators/*.proto'
     ]:
       for proto_file in glob.glob(pattern, recursive=True):
         # Ignore test protos.
@@ -230,6 +233,20 @@ class BuildBinaryGraphs(build.build):
       sys.stderr.write('generating binarypb: %s\n' %
                        os.path.join('mediapipe/modules/', binary_graph))
       self._generate_binary_graph(binary_graph)
+     
+    # iris graph
+    bazel_command = [
+        'bazel',
+        'build',
+        '--compilation_mode=opt',
+        '--define=MEDIAPIPE_DISABLE_GPU=1',
+        '--action_env=PYTHON_BIN_PATH=' + _normalize_path(sys.executable),
+        'mediapipe/graphs/iris_tracking/iris_tracking_cpu_binary_graph',
+    ]
+    output_name = 'iris_tracking/iris_tracking_cpu.binarypb'
+    output_file = os.path.join('mediapipe/graphs', output_name)
+    shutil.copyfile(
+        os.path.join('bazel-bin/mediapipe/graphs/', output_name), output_file)

   def _generate_binary_graph(self, graph_path):
     """Generate binary graph for a particular MediaPipe binary graph target."""

Adım 3. https://google.github.io/mediapipe/getting_started/python#building-mediapipe-python-package adresini izleyerek MediaPipe Python'u oluşturun .

Adım 4. Çalıştır

import mediapipe as mp
import cv2
mp_iris = mp.solutions.iris
iris = mp_iris.Iris()
image = cv2.imread(file)
results = iris.process(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
print(results.face_landmarks_with_iris)
iris.close()

Göz yer işaretleri indekslerini şu adreste bulabilirsiniz:

https://github.com/google/mediapipe/blob/350fbb2100ad531bc110b93aaea23d96af5a5064/mediapipe/graphs/iris_tracking/calculators/update_face_landmarks_calculator.cc#L33

Örneğin, sol göz alt kontur koordinatlarından birini ile alabilirsiniz print(results.face_landmarks_with_iris.landmark[33]).

Can you share the edited files?

sandesh1994199 commented 3 years ago

@sgowroji Could you please let me know, how to carry out step 2 ? which file needs to be modified ?

adijawla commented 3 years ago

@jiuqiant @sgowroji @mertkslkc @RafalSiwek @GNUaquin

We are trying to setup the python API for iris tracking in media pipe , please could you explain how to proceed with second step as it is not clear which file need to be modified or how to deal with step 2. Thanks

sandesh1994199 commented 2 years ago

I am fallowing #1530 to implement iris in python in docker. But i run python3 setup.py bdist_wheel this command

image

modules/imgcodecs/CMakeFiles/opencv_imgcodecs.dir/build.make:81: recipe for target 'modules/imgcodecs/CMakeFiles/opencv_imgcodecs.dir/src/loadsave.cpp.o' failed make[2]: [modules/imgcodecs/CMakeFiles/opencv_imgcodecs.dir/src/loadsave.cpp.o] Error 1 CMakeFiles/Makefile2:1697: recipe for target 'modules/imgcodecs/CMakeFiles/opencv_imgcodecs.dir/all' failed make[1]: [modules/imgcodecs/CMakeFiles/opencvimgcodecs.dir/all] Error 2 Makefile:181: recipe for target 'all' failed make: *** [all] Error 2 ____ END BUILD LOGS _____ rules_foreign_cc: Build wrapper script location: bazel-out/k8-opt/bin/third_party/opencv_cmake_scripts/CMake_wrapper_script.sh rules_foreign_cc: Build script location: bazel-out/k8-opt/bin/third_party/opencv_cmake_scripts/CMake_script.sh rules_foreign_cc: Build log location: bazel-out/k8-opt/bin/third_party/opencv_cmake_logs/CMake.log

Target //mediapipe/modules/face_detection:face_detection_short_range_cpu failed to build Use --verbose_failures to see the command lines of failed build steps. INFO: Elapsed time: 1338.266s, Critical Path: 1115.33s INFO: 3474 processes: 126 internal, 3348 processwrapper-sandbox. FAILED: Build did NOT complete successfully

sgowroji commented 2 years ago

Iris landmarks are now available in the python face mesh solution by setting refine_landmarks to True.

ianni67 commented 2 years ago

Hello @sgowroji, would you, please, nicely let us know which landmark numbers correspond to the iris landmarks when retrieving the landmarks with the python face mesh solution? The scheme I used in the past ( https://github.com/google/mediapipe/blob/master/mediapipe/modules/face_geometry/data/canonical_face_model_uv_visualization.png) does not contain the iris landmarks. It would also be great to have the coordinates of the iris points from the canonical model, as the file canonical_face_model.obj does not contain them (I guess). I would need them for my project.

jiuqiant commented 2 years ago

Hello @sgowroji, would you, please, nicely let us know which landmark numbers correspond to the iris landmarks when retrieving the landmarks with the python face mesh solution? The scheme I used in the past ( https://github.com/google/mediapipe/blob/master/mediapipe/modules/face_geometry/data/canonical_face_model_uv_visualization.png) does not contain the iris landmarks. It would also be great to have the coordinates of the iris points from the canonical model, as the file canonical_face_model.obj does not contain them (I guess). I would need them for my project.

See https://github.com/google/mediapipe/blob/master/mediapipe/python/solutions/face_mesh_connections.py#L32 and https://github.com/google/mediapipe/blob/master/mediapipe/python/solutions/face_mesh_connections.py#L47.

ianni67 commented 2 years ago

Hello @sgowroji, would you, please, nicely let us know which landmark numbers correspond to the iris landmarks when retrieving the landmarks with the python face mesh solution? The scheme I used in the past ( https://github.com/google/mediapipe/blob/master/mediapipe/modules/face_geometry/data/canonical_face_model_uv_visualization.png) does not contain the iris landmarks. It would also be great to have the coordinates of the iris points from the canonical model, as the file canonical_face_model.obj does not contain them (I guess). I would need them for my project.

See https://github.com/google/mediapipe/blob/master/mediapipe/python/solutions/face_mesh_connections.py#L32 and https://github.com/google/mediapipe/blob/master/mediapipe/python/solutions/face_mesh_connections.py#L47.

Thank you very much for your fast and kind reply, @jiuqiant. Maybe you can also help me with the coordinates of the iris landmarks in the canonical model? I'm still struggling to find them...

lghasemzadeh commented 1 year ago

Iris landmarks are now available in the python face mesh solution by setting refine_landmarks to True.

Thank you @sgowroji 1) from which version of python mediapipe facemesh we can achieve those 8 irises landmarks? 2) so we can use mediapipe's iris solution in python now, correct? if yes why the table in this link is not updated? the table still shows that we cannot use iris solution in python yet, am I wrong? 3) if we can use, should we use the last version of mediapipe to run iris solution in python?

Itamar651 commented 1 year ago

I'm trying to create eye tracking using mediapipe and I'm stuck due to inaccuracy issues.

I have some questions:

  1. Did you work on an eye tracking project using a webcam and are you interested in sharing?
  2. Have you had a breakthrough regarding eye tracking using only a webcam?
  3. Do you have any advice for me regarding webcam based eye tracking only?

Thanks, Itamar