Node: mediapipe models

[liyier90]

Partially addresses https://github.com/aisingapore/PeekingDuck-Private/issues/31

• Supports "Face Detection" and "Pose" MediaPipe Solution
• Face Detection is implemented as an object_detection task with a face subtask.
  • Omits face keypoints results (for output type compatibility)
  • Implemented as a subtask as MediaPipe has a general object detection solution which may become available in Python
• Pose estimation is implemented as a pose_estimation task with a body subtask
  • Converts the model's 33 keypoint output into COCO's 17 keypoint format to be compatible with other pose models.
  • Uses body subtask instead of person following convention in COCO dataset
• Added a pose utility module for handling some common processing functions for COCO's 17-keypoint body keypoints (not yet applied to existing pose estimation models'

[ongtw]

I encountered the following error when performing a make clean html in PeekingDuck/docs/:

building [mo]: targets for 0 po files that are out of date
building [html]: targets for 0 source files that are out of date
updating environment: 0 added, 1 changed, 0 removed
reading sources... [100%] nodes/model.mediapipev1
WARNING: autodoc: failed to import class 'Node' from module 'model.mediapipev1'; the following exception was raised:
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/temp4/lib/python3.8/site-packages/sphinx/util/inspect.py", line 376, in safe_getattr
    return getattr(obj, name, *defargs)
AttributeError: module 'model.mediapipev1' has no attribute 'Node'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniforge/base/envs/temp4/lib/python3.8/site-packages/sphinx/ext/autodoc/importer.py", line 98, in import_object
    obj = attrgetter(obj, mangled_name)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/temp4/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 306, in get_attr
    return autodoc_attrgetter(self.env.app, obj, name, *defargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/temp4/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 2804, in autodoc_attrgetter
    return safe_getattr(obj, name, *defargs)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/temp4/lib/python3.8/site-packages/sphinx/util/inspect.py", line 392, in safe_getattr
    raise AttributeError(name) from exc
AttributeError: Node

looking for now-outdated files... none found

Are you able to build the docs for model.mediapipev1? Wonder if this is a M1 Mac issue? (c.f. I was able to build the docs for PeekingDuck's latest dev branch)

[ongtw]

How to get a list of MediaPipe model types? Calling peekingduck model-hub --help only shows huggingface:

Invoking 'peekingduck model-hub --help'...
Usage: cli model-hub [OPTIONS] COMMAND [ARGS]...

  Utility commands for model hub models.

Options:
  --help  Show this message and exit.

Commands:
  huggingface  Utility commands for Hugging Face Hub models.

 ~/s/ais/y/PeekingDuck │ node-mediapipe !2 ?2

[ongtw]

Can provide a sample object detection yaml and a pose estimation yaml for testing MediaPipe's models?

[ongtw]

Running pytest -s -vv tests/pipeline/nodes/model/mediapipev1 returns the following error on M1 macOS, because mediapipe package is named mediapipe-silicon on M1 macOS:

❯ pytest -s -vv tests/pipeline/nodes/model/mediapipev1
======================================= test session starts =======================================
platform darwin -- Python 3.8.13, pytest-7.1.3, pluggy-1.0.0 -- /opt/homebrew/Caskroom/miniforge/base/envs/temp4/bin/python
cachedir: .pytest_cache
rootdir: /Users/dotw/src/aisg/yier/PeekingDuck, configfile: pyproject.toml
plugins: typeguard-2.13.3
collecting ... 2022-10-09 17:16:57 peekingduck.utils.requirement_checker  INFO:  mediapipe>=0.8.11 not found and is required, attempting auto-update...
ERROR: Could not find a version that satisfies the requirement mediapipe>=0.8.11 (from versions: none)
ERROR: No matching distribution found for mediapipe>=0.8.11
2022-10-09 17:16:57 peekingduck.utils.requirement_checker  ERROR:  Command '['pip', 'install', 'mediapipe>=0.8.11']' returned non-zero exit status 1.
collected 0 items / 1 error

Output of conda env export for M1 macOS env:

[ conda stuff deleted ]
    - matplotlib==3.6.1
    - mediapipe-silicon==0.8.11
    - oauthlib==3.2.1
    - opencv-contrib-python==4.6.0.66
[ more pip stuff deleted ]

[liyier90]

I encountered the following error when performing a make clean html in PeekingDuck/docs/:

Fixed by adding mediapipev1 to omit list when building docs.

How to get a list of MediaPipe model types? Calling peekingduck model-hub --help only shows huggingface:

Apologies, I missed out this part when finishing up. Intially I thought mediapipe will have an API similar to huggingface so we could build some CLI commands with minimal maintanence on our side. But turns out there isn't any.

We will have to copy out the model type documentation from the official site ourselves. Then would it be better to: 1) just leave it in the Node docstrings (like normal model nodes) or 2) create some CLI commands which just prints out some predefined string. Option 2 will have a more consistent interface like huggingface but may possible create 2 locations to maintain.

Can provide a sample object detection yaml and a pose estimation yaml for testing MediaPipe's models?

nodes:
  - input.visual:
      source: https://storage.googleapis.com/peekingduck/videos/wave.mp4
  - model.mediapipe:
      #   task: object_detection
      #   subtask: face
      task: pose_estimation
      subtask: body
      model_type: 0
  - draw.bbox
  - draw.poses
  - output.screen

Running pytest -s -vv tests/pipeline/nodes/model/mediapipev1 returns the following error on M1 macOS, because mediapipe package is named mediapipe-silicon on M1 macOS:

May I know if mediapipe-silicon is installed through pip or conda? If it's through pip, I can try to modify RequirementChecker to accept an "either ... or" statement. But if it's through conda would it then be ok to make mediapipe an compulsory requirement so it skips the check?

[ongtw]

Thanks for the updates.

1. The docs now build correctly.
2. Re CLI model-hub, it'll be good to have the same interface as HF to provide a consistent look-and-feel. Is there any chance this of this reading the relevant data from model.mediapipe node? So that we only have a single point of update.
3. The mediapipe-silicon is installed through pip. So can modify RequirementChecker to handle this as you suggested.

Updates needed:

1. model.mediapipe node needs description of both object detection and pose estimation.
2. Running your mediapipe yaml above produces the below output, but I think this would be fixed once 3 above is implemented.

   ❯ python . --config_path mp.yml
   2022-10-10 18:25:05 __main__  INFO:  Run path: mp.yml
   2022-10-10 18:25:05 peekingduck.declarative_loader  INFO:  Successfully loaded pipeline file.
   2022-10-10 18:25:05 peekingduck.declarative_loader  INFO:  Initializing input.visual node...
   2022-10-10 18:25:05 peekingduck.declarative_loader  INFO:  Config for node input.visual is updated to: 'source': https://storage.googleapis.com/peekingduck/videos/wave.mp4
   2022-10-10 18:25:05 peekingduck.pipeline.nodes.input.visual  INFO:  Input size: 710 by 540
   2022-10-10 18:25:05 peekingduck.declarative_loader  INFO:  Initializing model.mediapipe node...
   2022-10-10 18:25:05 peekingduck.utils.requirement_checker  INFO:  mediapipe>=0.8.11 not found and is required, attempting auto-update...
   ERROR: Could not find a version that satisfies the requirement mediapipe>=0.8.11 (from versions: none)
   ERROR: No matching distribution found for mediapipe>=0.8.11
   2022-10-10 18:25:06 peekingduck.utils.requirement_checker  ERROR:  Command '['pip', 'install', 'mediapipe>=0.8.11']' returned non-zero exit status 1.

[ongtw]

@liyier90 Can resolve peekingduck/pipeline/utils/bbox/transforms.py conflict? Think this was caused by Hongnan's PR.

[liyier90]

@ongtw The merge conflict has been fixed.