Hands solution Side packet issue

[Hyempire]

Description

I've just done your tutorial (FaceMesh) and trying to make a Hand Tracking project, refering to the same tutorial. But in the middle of the phases, I got a novel error. (I am very very new to C#, so probably my code is ridiculous..)

This is the error

MediaPipeException: INVALID_ARGUMENT: ValidateRequiredSidePackets failed to validate: 
; Side packet "input_horizontally_flipped" is required but was not provided.
; Side packet "input_rotation" is required but was not provided.
; Side packet "input_vertically_flipped" is required but was not provided.
; Side packet "num_hands" is required but was not provided.
Mediapipe.Status.AssertOk () (at Packages/com.github.homuler.mediapipe/Runtime/Scripts/Framework/Port/Status.cs:149)
HandsPractice+<Start>d__10.MoveNext () (at Assets/HandLandmarkPractice/HandsPractice.cs:60)
UnityEngine.SetupCoroutine.InvokeMoveNext (System.Collections.IEnumerator enumerator, System.IntPtr returnValueAddress) (at <d3b66f0ad4e34a55b6ef91ab84878193>:0)

This is my code so far

using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;
using Mediapipe;
using Mediapipe.Unity;
using Mediapipe.Unity.CoordinateSystem;

using Stopwatch = System.Diagnostics.Stopwatch;

public class HandsPractice : MonoBehaviour
{
    [SerializeField] private TextAsset _configAsset;
    [SerializeField] private RawImage _screen;
    [SerializeField] private int _width;
    [SerializeField] private int _height;
    [SerializeField] private int _fps;

    private CalculatorGraph _graph;
    private ResourceManager _resourceManager;

    private WebCamTexture _webCamTexture;
    private Texture2D _inputTexture;
    private Color32[] _pixelData;

    private IEnumerator Start()
    {
        if (WebCamTexture.devices.Length == 0)
        {
            throw new System.Exception("Web Camera devices are not found");
        }

        // Turn on Webcam
        var webcamDevice = WebCamTexture.devices[0];
        _webCamTexture = new WebCamTexture(webcamDevice.name, _width, _height, _fps);
        _webCamTexture.Play();

        // return coroutine function
        yield return new WaitUntil(() => _webCamTexture.width > 16);

        _screen.rectTransform.sizeDelta = new Vector2(_width, _height);
        _screen.texture = _webCamTexture;

        // Bring Hands moedel
        _resourceManager = new LocalResourceManager();
        yield return _resourceManager.PrepareAssetAsync("hand_landmark_full.bytes");
        yield return _resourceManager.PrepareAssetAsync("hand_landmark_lite.bytes");
        yield return _resourceManager.PrepareAssetAsync("hand_recrop.bytes");
        yield return _resourceManager.PrepareAssetAsync("handedness.txt");

        var stopwatch = new Stopwatch();

        // image frame for input stream
        _inputTexture = new Texture2D(_width, _height, TextureFormat.RGBA32, false);
        _pixelData = new Color32[_width * _height];

        // CalculatorGraph initialize
        _graph = new CalculatorGraph(_configAsset.text);

        // Bring landmarks ourputstream
        var handLandmarksStream = new OutputStream<NormalizedLandmarkListVectorPacket, List<NormalizedLandmarkList>>(
            _graph, "hand_landmarks");

        // polling
        handLandmarksStream.StartPolling().AssertOk();

        // Run graph
        _graph.StartRun().AssertOk();

        stopwatch.Start();

        // Bring Rectransform of Screen
        var screenRect = _screen.GetComponent<RectTransform>().rect;

        while (true)
        {
            _inputTexture.SetPixels32(_webCamTexture.GetPixels32(_pixelData));

            // ImageFrame initialize
            var imageFrame = new ImageFrame(
                ImageFormat.Types.Format.Srgba,
                _width, _height,
                _width * 4,
                _inputTexture.GetRawTextureData<byte>());
            // Make Timestamp
            var currentTimestamp = stopwatch.ElapsedTicks / (System.TimeSpan.TicksPerMillisecond / 1000);

            // imageFrame을 packet에 싸서 inputstream으로 보내기
            _graph.AddPacketToInputStream("input_video", new ImageFramePacket(imageFrame, new Timestamp(currentTimestamp))).AssertOk();

            yield return new WaitForEndOfFrame();

            // Get landmarks values
            if (handLandmarksStream.TryGetNext(out var handLandmarks))
            {
                foreach (var landmarks in handLandmarks)
                {
                    var fingerTip = landmarks.Landmark[8];
                    Debug.Log(screenRect.GetPoint(fingerTip));
                }
            }
        }
    }

    private void OnDestroy()
    {
        if (_webCamTexture != null)
        {
            _webCamTexture.Stop();
        }
        if (_graph != null)
        {
            try
            {
                _graph.CloseInputStream("input_video").AssertOk();
                _graph.WaitUntilDone().AssertOk();
            }
            finally
            {
                _graph.Dispose();
            }
        }
    }
}

I tried to provide side packets by adding this code right below running graph.

var sidePacket = new SidePacket();
sidePacket.Emplace("input_horizontally_flipped", new BoolPacket(true));
sidePacket.Emplace("input_rotation", new IntPacket(0));
sidePacket.Emplace("input_vertically_flipped", new BoolPacket(true));
sidePacket.Emplace("num_hands", new IntPacket(2));

but error still exists.

This is how I set objects and Inspector. Exactly same with the tutorial.

I have no idea where to start T.T Please give me any advices you can give.

[G93-7]

I think you must create a side packet before you run the graph. After that you must pass the side packet as a parameter to the StartRun function like this: _graph.StartRun(sidePacket).AssertOk();

[homuler]

See also https://github.com/homuler/MediaPipeUnityPlugin/wiki/API-Overview#start-running.

[Hyempire]

I think you must create a side packet before you run the graph. After that you must pass the side packet as a parameter to the StartRun function like this: _graph.StartRun(sidePacket).AssertOk();

Thank you so much. But it didn't solve my problem. The errors still exist T.T

[homuler]

Then maybe your hand_tracking_cpu has an error.

[Hyempire]

The hand_tracking_cpu.txt is the exact one in MediaPipeUnity package..!

# Copyright 2019 The MediaPipe Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Copied from mediapipe/graphs/hand_tracking/hand_tracking_desktop_live.pbtxt
#
# CHANGES:
#   - Add ImageTransformationCalculator and rotate the input
#   - Remove HandRendererSubgraph

# MediaPipe graph that performs hand tracking on desktop with TensorFlow Lite
# on CPU.
# Used in the example in
# mediapipie/examples/desktop/hand_tracking:hand_tracking_cpu.

# CPU image. (ImageFrame)
input_stream: "input_video"

# Max number of hands to detect/process. (int)
input_side_packet: "num_hands"

# Collection of detected/predicted hands, each represented as a list of
# landmarks. (std::vector<NormalizedLandmarkList>)
output_stream: "hand_landmarks"

# Collection of detected/predicted hand world landmarks.
# (std::vector<LandmarkList>)
output_stream: "hand_world_landmarks"

output_stream: "handedness"
output_stream: "palm_detections"
output_stream: "hand_rects_from_landmarks"
output_stream: "hand_rects_from_palm_detections"

node {
  calculator: "FlowLimiterCalculator"
  input_stream: "input_video"
  input_stream: "FINISHED:hand_landmarks"
  input_stream_info: {
    tag_index: "FINISHED"
    back_edge: true
  }
  output_stream: "throttled_input_video"
}

node: {
  calculator: "ImageTransformationCalculator"
  input_stream: "IMAGE:throttled_input_video"
  input_side_packet: "ROTATION_DEGREES:input_rotation"
  input_side_packet: "FLIP_HORIZONTALLY:input_horizontally_flipped"
  input_side_packet: "FLIP_VERTICALLY:input_vertically_flipped"
  output_stream: "IMAGE:transformed_input_video"
}

# Detects/tracks hand landmarks.
node {
  calculator: "HandLandmarkTrackingCpu"
  input_stream: "IMAGE:transformed_input_video"
  input_side_packet: "MODEL_COMPLEXITY:model_complexity"
  input_side_packet: "NUM_HANDS:num_hands"
  output_stream: "LANDMARKS:hand_landmarks"
  output_stream: "WORLD_LANDMARKS:hand_world_landmarks"
  output_stream: "HANDEDNESS:handedness"
  output_stream: "PALM_DETECTIONS:palm_detections"
  output_stream: "HAND_ROIS_FROM_LANDMARKS:hand_rects_from_landmarks"
  output_stream: "HAND_ROIS_FROM_PALM_DETECTIONS:hand_rects_from_palm_detections"
}

[azmath37]

Hi @Hyempire I am facing same issue, did you find any solution..?