Which ImageFormat to use.

[elblogbruno]

Hello! As I told you on error #23 I am integrating ar foundation into this. You told me to create a ImageFrame, the thing is I don't know which Format Should be use, Glad if you or someone could help me out Cordially Bruno

[elblogbruno]

https://pastebin.com/NhJ8n5UT As you seeit shows the message DEBUGGING VALUES FROM STREAM and prints out handTrackingValue.PalmDetections[0] and handTrackingValue.Handednesses[0] once. and it does not show anymore.

[elblogbruno]

Log file created at: 2021/02/02 13:46:16 Running on machine: localhost Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg I20210202 13:46:16.792347 13498 gl_context_egl.cc:163] Successfully initialized EGL. Major : 1 Minor: 5 Glog log

[homuler]

Error Unity ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.

I think simply your code has some bugs.

Debug.Log(handTrackingValue.Handednesses[0]);

[elblogbruno]

Ok I will investigate furthermore! and why I can't fetch next packet from OfficalDemoGraph stream? public override void RenderOutput(WebCamScreenController screenController, ImageFrame textureFrame) { // MediaPipe renders to the texture directly. //var landmarks = FetchNext(handLandmarksStreamPoller, handLandmarksPacket, "hand_landmarks"); //Debug.Log("render image frame: " + landmarks); return; } I have uncommented it and texture is displayed, but when I call the function FetchNext it does not work. I followed #23 tip from your part on how to read values synchronously. thanks

[homuler]

Did you read this comment?

// Note that the above method blocks the thread, and if the output packet is empty, it never returns (the process hangs).
// see https://github.com/homuler/MediaPipeUnityPlugin/blob/master/Assets/MediaPipe/Examples/Graphs/HandTracking/Scripts/HandTrackingGraph.cs

Before calling FetchNext, you have to make sure that the output packet exists. If you use ObserveOutputStream, this problem won't occur.

[elblogbruno]

oops, I am so sorry I missed it! Thanks

[elblogbruno]

I was implementing ObserveOutputStream and always get this issue, that has popped up before. It is because private Status OutputCallback(NormalizedLandmarkListVectorPacket packet) should be static or decorator [AOT.MonoPInvokeCallback(typeof(GlCalculatorHelper.NativeGlStatusFunction))] needs to be added?

https://github.com/homuler/MediaPipeUnityPlugin/blob/master/Assets/MediaPipe/Examples/Graphs/OfficialDemo/Scripts/OfficialDemoDesktop.cs I have followed this as a reference!

2021/02/02 16:20:22.948 22850 22893 Error Unity ArgumentException: Object contains non-primitive or non-blittable data.
2021/02/02 16:20:22.948 22850 22893 Error Unity   at System.Runtime.InteropServices.GCHandle..ctor (System.Object value, System.Runtime.InteropServices.GCHandleType type) [0x00000] in <00000000000000000000000000000000>:0 
2021/02/02 16:20:22.948 22850 22893 Error Unity   at System.Runtime.InteropServices.GCHandle.Alloc (System.Object value, System.Runtime.InteropServices.GCHandleType type) [0x00000] in <00000000000000000000000000000000>:0 
2021/02/02 16:20:22.948 22850 22893 Error Unity   at Mediapipe.CalculatorGraph.ObserveOutputStream[T,U] (System.String streamName, Mediapipe.CalculatorGraph+PacketCallback`2[T,U] packetCallback, System.Runtime.InteropServices.GCHandle& callbackHandle) [0x00000] in <00000000000000000000000000000000>:0 
2021/02/02 16:20:22.948 22850 22893 Error Unity   at OfficialDemoAndroid.StartRun (UnityEngine.Texture texture) [0x00000] in <00000000000000000000000000000000>:0 
2021/02/02 16:20:22.948 22850 22893 Error Unity   at WebCamScreenController+<RunGraph>d__45.MoveNext () [0x00000] in <00000000000000000000000000000000>:0 
2021/02/02 16:20:22.948 22850 22893 Error Unity   at UnityEngine.SetupCoroutine.InvokeMoveNext (System.Collections.IEnumerator enumerator, System.IntPtr returnValueAddress) [
2021/02/02 16:20:24.695 22850 23180 Error native motion_tracking_context.cc:1064 Unable to find measurements corresponding to VIO status at timestamp 2568738454048105.

[homuler]

The callback type is CalculatorGraph.NativePacketCallback and it should return IntPtr, not Status (the reason is same as https://github.com/homuler/MediaPipeUnityPlugin/issues/43#issuecomment-770498550). See also https://github.com/homuler/MediaPipeUnityPlugin/blob/master/Assets/MediaPipe/SDK/Scripts/Framework/CalculatorGraph.cs#L59 .

[AOT.MonoPInvokeCallback(typeof(CalculatorGraph.NativePacketCallback))]
static IntPtr HandLandmarksCallback(IntPtr ptr) {
  using (var packet = new NormalizedLandmarksListVectorPacket(ptr, false)) {
    using (var landmarks = packet.Get()) {
      // do something
    }
    return Status.Ok().mpPtr;
  }
}

[elblogbruno]

Ok I see what was the problem! Thanks for all again!

[homuler]

I'm sorry that ObserveOutputStream is not documented well and no examples exist for IL2CPP, but note that the packet the callback receives is still owned by MediaPipe, so the constructor's second argument has to be false.

var packet = new NormalizedLandmarksListVectorPacket(ptr, false);

[elblogbruno]

Don't worry about having less documentation. You are doing a great job with this repo!

[elblogbruno]

it expects an status

[elblogbruno]

I got it working 😄 but can't read from the stream consistently. Hand annotations are shown as always but as you see by the code, i am debugging some values.

public override void RenderOutput(WebCamScreenController screenController, ImageFrame textureFrame) { var handTrackingValue = FetchNextHandTrackingValue(); RenderAnnotation(screenController, handTrackingValue); Debug.Log(handTrackingValue.PalmDetections.Count); Debug.Log(handTrackingValue.Handednesses[0]); screenController.DrawScreen(textureFrame); }

Values are debugged for some seconds until it stops with the unexpected END OF FILE issue. Even though values are not being printed out on the console annotations still appear on screen. I will also try now to debug landmarks on the official graph. (i am getting better performance with official graph, but can't print out values on console)

2021/02/02 12:01:53.617 21374 21908 Info native I20210202 12:01:53.617295 21908 scheduler.cc:455] active queues: 1 2021/02/02 12:01:53.617 21374 21904 Info native I20210202 12:01:53.617719 21904 calculator_node.cc:812] Calling Calculator::Process() for node: handlandmarktrackinggpu__handlandmarkgpu__ImageToTensorCalculator timestamp: 36 2021/02/02 12:01:53.618 21374 21904 Info native I20210202 12:01:53.618667 21904 calculator_node.cc:825] Called Calculator::Process() for node: handlandmarktrackinggpu__handlandmarkgpu__ImageToTensorCalculator timestamp: 36 2021/02/02 12:01:53.618 21374 21904 Info native I20210202 12:01:53.618875 21904 calculator_node.cc:812] Calling Calculator::Process() for node: handlandmarktrackinggpu__handlandmarkgpu__InferenceCalculator timestamp: 36 0001/01/01 00:00:00.000 -1 -1 Info read: unexpected EOF! Again thanks for your help @homuler !

getting this problem again wth

[homuler]

Please see the link in the comment https://github.com/homuler/MediaPipeUnityPlugin/issues/43#issuecomment-771730648 . ObserveOutputStream is overloaded.

[elblogbruno]

I am using this version of the ObserveOutputStream https://github.com/homuler/MediaPipeUnityPlugin/blob/master/Assets/MediaPipe/SDK/Scripts/Framework/CalculatorGraph.cs#L59 As the other one expects a status in the static function callback

[elblogbruno]

it expects an status

See this

[homuler]

It's because the arity is different...

graph.ObserveOutputStream("hand_landmarks", HandLandmarksCallback).AssertOk();

[elblogbruno]

It's because the arity is different...

graph.ObserveOutputStream("hand_landmarks", HandLandmarksCallback).AssertOk();

Yes I am using it like this

[elblogbruno]

I got it working 😄 but can't read from the stream consistently. Hand annotations are shown as always but as you see by the code, i am debugging some values.

public override void RenderOutput(WebCamScreenController screenController, ImageFrame textureFrame) { var handTrackingValue = FetchNextHandTrackingValue(); RenderAnnotation(screenController, handTrackingValue); Debug.Log(handTrackingValue.PalmDetections.Count); Debug.Log(handTrackingValue.Handednesses[0]); screenController.DrawScreen(textureFrame); }

Values are debugged for some seconds until it stops with the unexpected END OF FILE issue. Even though values are not being printed out on the console annotations still appear on screen. I will also try now to debug landmarks on the official graph. (i am getting better performance with official graph, but can't print out values on console)

2021/02/02 12:01:53.617 21374 21908 Info native I20210202 12:01:53.617295 21908 scheduler.cc:455] active queues: 1 2021/02/02 12:01:53.617 21374 21904 Info native I20210202 12:01:53.617719 21904 calculator_node.cc:812] Calling Calculator::Process() for node: handlandmarktrackinggpu__handlandmarkgpu__ImageToTensorCalculator timestamp: 36 2021/02/02 12:01:53.618 21374 21904 Info native I20210202 12:01:53.618667 21904 calculator_node.cc:825] Called Calculator::Process() for node: handlandmarktrackinggpu__handlandmarkgpu__ImageToTensorCalculator timestamp: 36 2021/02/02 12:01:53.618 21374 21904 Info native I20210202 12:01:53.618875 21904 calculator_node.cc:812] Calling Calculator::Process() for node: handlandmarktrackinggpu__handlandmarkgpu__InferenceCalculator timestamp: 36 0001/01/01 00:00:00.000 -1 -1 Info read: unexpected EOF! Again thanks for your help @homuler !

getting this problem again wth

And getting the same thing again

[homuler]

It's because the arity is different...

graph.ObserveOutputStream("hand_landmarks", HandLandmarksCallback).AssertOk();

Yes I am using it like this

??? https://github.com/homuler/MediaPipeUnityPlugin/issues/43#issuecomment-772326221 Then I don't understand what the above comment means.

And getting the same thing again

Without knowing the exact code, I can't help you. unexpected EOF occurs usually when the process is killed or aborted (Gog did not finish writing its buffer to the log file), and probably the root cause does not appear in the shared log.

[elblogbruno]

I will investigate what the problem could be furthermore. Maybe it is related because I am using Unity 2020.2.2?

[elblogbruno]

public override Status StartRun(Texture texture) {
    Debug.Log("This graph is for testing official examples. You can customize the graph by editing `official_demo_android.txt` (default is `hand_tracking_mobile.pbtxt`)");

    graph.ObserveOutputStream(handLandmarksStream, HandLandmarksCallback).AssertOk();

    sidePacket = new SidePacket();
    sidePacket.Emplace("num_hands", new IntPacket(2));

    destinationNativeTexturePtr = texture.GetNativeTexturePtr();
    destinationWidth = texture.width;
    destinationHeight = texture.height;

    gpuHelper.RunInGlContext(BuildDestination).AssertOk();
    sidePacket.Emplace(destinationBufferName, outputPacket);

    stopwatch.Start();
    Debug.LogWarning("HAND TRACKING GRAPH START RUN");

    return graph.StartRun(sidePacket);
  }

  [AOT.MonoPInvokeCallback(typeof(CalculatorGraph.NativePacketCallback))]
  static IntPtr HandLandmarksCallback(IntPtr ptr) {
    using (var packet = new NormalizedLandmarkListVectorPacket(ptr, false))
    {
      List<NormalizedLandmarkList> landmarks = packet.Get();
      Debug.Log("CALLBACK ");
      Debug.Log(landmarks[0]);
      return Status.Ok().mpPtr;
    }
  }

Or maybe there is something wrong here 🤨

[homuler]

This code won't work, because landmarks can be empty.

Debug.Log(landmarks[0]);

And note that HandLandmarksCallback must not throw exceptions, because native code can't handle it.

[elblogbruno]

This code won't work, because landmarks can be empty.

Debug.Log(landmarks[0]);

And note that HandLandmarksCallback must not throw exceptions, because native code can't handle it.

so maybe it is that because it is null it throws an exception that blocks the thread?

[homuler]

I don't know. I think the behavior is not defined. At least you need to handle exceptions.

static IntPtr HandleLandmarksCallback(IntPtr packetPtr) {
  try {
    using (var packet = new NormalizedLandmarkListVectorPacket(ptr, false)) {
      List<NormalizedLandmarkList> landmarks = packet.Get();
      Debug.Log("CALLBACK ");
      Debug.Log(landmarks[0]);
      return Status.Ok().mpPtr;
    }
  } catch (Exception e) {
    return Status.FailedPrecondition(e.ToString()).mpPtr;
  }
}

[elblogbruno]

hmm ok thanks

[elblogbruno]

Still can't get to read data from the graph. I have tried both synchronously and asynchronously.

[elblogbruno]

I will try latest commit of repo

[holaworlds]

Hi @elblogbruno could you figure out this? I am confused about how to assign the output to a variable. I am using ObjectDetection3dGraph example but concept is same as I see.

should'nt we have a return value from HandLandmarksCallback function so that we can assign it to a variable and use it? HandleLandmarksCallback returns only status.

@homuler if you have any idea can you share it?