Can't get packet from Official Demo Graph

[elblogbruno]

Hello @homuler Thanks for all the support lately. Following #23 I wanted to read landmark data from official demo graph. I have tried synchronously like this:

public override void RenderOutput(WebCamScreenController screenController, TextureFrame textureFrame) {

  public override Status StartRun() {
#if UNITY_IOS
    // On iOS, it's faster to get output packets synchronously than asynchronously.
    outputStreamPoller = graph.AddOutputStreamPoller<ImageFrame>(outputStream).ConsumeValueOrDie();
    outputPacket = new ImageFramePacket();
#elif UNITY_EDITOR || !UNITY_ANDROID
    graph.ObserveOutputStream<ImageFramePacket, ImageFrame>(outputStream, OutputVideoCallback, out outputVideoCallbackHandle).AssertOk();
#endif

    handLandmarksStreamPoller = graph.AddOutputStreamPoller<List<NormalizedLandmarkList>>(handLandmarksStream).ConsumeValueOrDie();
    handLandmarksPacket = new NormalizedLandmarkListVectorPacket();

    return graph.StartRun(sidePacket);
  }

public override void RenderOutput(WebCamScreenController screenController, TextureFrame textureFrame) {

#if UNITY_ANDROID && !UNITY_EDITOR
    if (handLandmarksStreamPoller != null)
    {
        Debug.Log("Fetching landmarks");
        var landmarks = FetchNext(handLandmarksStreamPoller, handLandmarksPacket, "hand_landmarks");
        Debug.Log(landmarks[0]);
    }else{
        Debug.Log("can't fetch landmarks");
    }
    // MediaPipe renders the result to the screen directly.

as well as asynchronously:

  graph.ObserveOutputStream(handLandmarksStream, HandLandmarksCallback).AssertOk();

  [AOT.MonoPInvokeCallback(typeof(CalculatorGraph.NativePacketCallback))]
  static IntPtr HandLandmarksCallback(IntPtr ptr) {
    try {
      Debug.LogWarning("CALLBACK");
      using (var packet = new NormalizedLandmarkListVectorPacket(ptr, false)) 
      {
          List<NormalizedLandmarkList> landmarks = packet.Get();
          Debug.Log("CALLBACK ");
          Debug.Log(landmarks[0]);
          return Status.Ok().mpPtr;
      }
    } catch (Exception e) {
      return Status.FailedPrecondition(e.ToString()).mpPtr;
    }
  }

To make things clear, I am not using ARFoundation like in #43 I am using the latest repo, with nothing modified but the code on top added to OfficialDemoGraph.cs.

Hope there is a solution Cordially, Bruno

[elblogbruno]

InternalException: Status is not ok: NOT_FOUND: ; Unable to attach observer to output stream "hand_landmarks" because it doesn't exist. I get this running on windows. hmm..

[homuler]

It's because official_demo_graph_cpu.txt does not have an output stream called hand_landmarks, and on Windows it's used automatically instead of official_demo_graph_gpu.txt (GPU is not supported).

To test on Windows, add this line

output_stream: "hand_landmarks"

and replace LANDMARKS:landmarks with LANDMARKS:hand_landmarks

[elblogbruno]

Maybe having errors on windows is related to having issues on Android?

[elblogbruno]

Ok so I got callback working on windows editor. I will try on android.

[elblogbruno]

It does not print anything on android, will debug again!

[elblogbruno]

Logcat crashes

[homuler]

It seems that you override OfficialDemo#StartRun() but it'll never be called on Android. cf. https://github.com/homuler/MediaPipeUnityPlugin/blob/master/Assets/MediaPipe/Examples/Graphs/OfficialDemo/Scripts/OfficialDemoGraph.cs#L90

You need to override OfficialDemo#StartRun(Texture) instead.

By the way, below code does not avoid https://github.com/homuler/MediaPipeUnityPlugin/issues/43#issuecomment-771651776 and it will blocks the thread if no hand landmarks are found.

if (handLandmarksStreamPoller != null)
{
  Debug.Log("Fetching landmarks");
  var landmarks = FetchNext(handLandmarksStreamPoller, handLandmarksPacket, "hand_landmarks");
   Debug.Log(landmarks[0]);
} else {
  Debug.Log("can't fetch landmarks");
}

Please refer to HandTrackingGraph to see how to check if the landmarks exist.

[elblogbruno]

Oh, I feel I get your point now, Thanks!

[elblogbruno]

Finally got it working synchronously 😍, can't seem to fetch packets asynchronously 😒 on android, I will investigate furthermore.

[elblogbruno]

2021/02/21 12:47:22.596 15736 16136 Info native performance_monitor.cc:115 Event: FeatureExtraction is taking too long, it took 111.977ms
2021/02/21 12:47:22.606 15736 16271 Info Unity { "landmark": [ { "x": 0.794568062, "y": -0.05212976, "z": -8.479337E-05 }, { "x": 0.6678199, "y": -0.07344505, "z": 0.111126311 }, { "x": 0.5772773, "y": -0.0421902537, "z": 0.158718139 }, { "x": 0.5074707, "y": -0.00244815648, "z": 0.200248182 }, { "x": 0.46561265, "y": 0.0272077471, "z": 0.254246742 }, { "x": 0.558099568, "y": 0.0591590255, "z": 0.0268175416 }, { "x": 0.459399134, "y": 0.185355544, "z": 0.048740007 }, { "x": 0.4297872, "y": 0.255797863, "z": 0.09469208 }, { "x": 0.416626543, "y": 0.301086247, "z": 0.135174841 }, { "x": 0.6121756, "y": 0.116829187, "z": -0.0048311674 }, { "x": 0.5194821, "y": 0.281116962, "z": -0.009289826 }, { "x": 0.479342222, "y": 0.368067622, "z": 0.0260099731 }, { "x": 0.458572537, "y": 0.416899562, "z": 0.0609227531 }, { "x": 0.678285241, "y": 0.160105765, "z": -0.02347347 }, { "x": 0.59997046, "y": 0.3156631, "z": -0.028917158 }, { "x": 0.5606027, "y": 0.3981544, "z": 0.001263613 }, { "x": 0.5328014, "y": 0.4516604, "z": 0.0311656073 }, { "x": 0.7491905, "y": 0.1907
2021/02/21 12:47:22.625 15736 16078 Info Unity Getting status android
2021/02/21 12:47:22.625 15736 16078 Info Unity DemoGraph:GetStatusAndroid()
2021/02/21 12:47:22.625 15736 16078 Info Unity 
2021/02/21 12:47:22.626 15736 15799 Info Unity Sending ImageFrame 
2021/02/21 12:47:22.626 15736 15799 Info Unity WebCamScreenController:OnCameraFrameReceived(ARCameraFrameEventArgs)
2021/02/21 12:47:22.626 15736 15799 Info Unity System.Action`1:Invoke(T)
2021/02/21 12:47:22.626 15736 15799 Info Unity UnityEngine.XR.ARFoundation.ARCameraManager:InvokeFrameReceivedEvent(XRCameraFrame)
2021/02/21 12:47:22.626 15736 15799 Info Unity UnityEngine.XR.ARFoundation.ARCameraManager:Update()
2021/02/21 12:47:22.626 15736 15799 Info Unity 
2021/02/21 12:47:22.630 15736 16271 Error CRASH *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
2021/02/21 12:47:22.630 15736 16271 Error CRASH Version '2020.2.2f1 (068178b99f32)', Build type 'Development', Scripting Backend 'il2cpp', CPU 'arm64-v8a'
2021/02/21 12:47:22.630 15736 16271 Error CRASH Build fingerprint: 'OPPO/CPH1907EEA/OP4B83L1:10/QKQ1.190915.002/1606195512:user/release-keys'
2021/02/21 12:47:22.630 15736 16271 Error CRASH Revision: '0'
2021/02/21 12:47:22.630 15736 16271 Error CRASH ABI: 'arm64'
2021/02/21 12:47:22.630 15736 16271 Error CRASH Timestamp: 2021-02-21 12:47:22+0100
2021/02/21 12:47:22.630 15736 16271 Error CRASH pid: 15736, tid: 16271, name: mediapipe/16271  >>> com.glassear.mediapipe <<<
2021/02/21 12:47:22.630 15736 16271 Error CRASH uid: 11081
2021/02/21 12:47:22.630 15736 16271 Error CRASH signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
2021/02/21 12:47:22.630 15736 16271 Error CRASH     x0  0000000000000000  x1  0000000000003f8f  x2  0000000000000006  x3  0000007b34dfb340
2021/02/21 12:47:22.631 15736 16271 Error CRASH     x4  fefeff7a0652cb47  x5  fefeff7a0652cb47  x6  fefeff7a0652cb47  x7  7f7f7f7f7f7fff7f
2021/02/21 12:47:22.631 15736 16271 Error CRASH     x8  00000000000000f0  x9  391236b1cf827fb8  x10 0000000000000001  x11 0000000000000000
2021/02/21 12:47:22.631 15736 16271 Error CRASH     x12 fffffff0fffffbdf  x13 0000007d15d78360  x14 0000007cc295e318  x15 0000007cc35ca690
2021/02/21 12:47:22.631 15736 16271 Error CRASH     x16 0000007db6db1740  x17 0000007db6d8f7e0  x18 0000007cc35cd000  x19 0000000000003d78
2021/02/21 12:47:22.631 15736 16271 Error CRASH     x20 0000000000003f8f  x21 00000000ffffffff  x22 0000000000000000  x23 0000007db6dbbea0
2021/02/21 12:47:22.631 15736 16271 Error CRASH     x24 0000007bb419a039  x25 0000007b34dfc020  x26 0000007b34dfc020  x27 0000007b34dfb5c1
2021/02/21 12:47:22.631 15736 16271 Error CRASH     x28 0000007beee3d680  x29 0000007b34dfb3e0
2021/02/21 12:47:22.631 15736 16271 Error CRASH     sp  0000007b34dfb320  lr  0000007db6d3f484  pc  0000007db6d3f4b0
2021/02/21 12:47:22.631 15736 16271 Error CRASH 
2021/02/21 12:47:22.631 15736 16271 Error CRASH backtrace:
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #00 pc 00000000000744b0  /apex/com.android.runtime/lib64/bionic/libc.so (abort+160) (BuildId: f46dcba52f09252fc392ba4a54b5b398)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #01 pc 000000000005f760  /apex/com.android.runtime/lib64/bionic/libc.so (ifree+1136) (BuildId: f46dcba52f09252fc392ba4a54b5b398)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #02 pc 000000000005f86c  /apex/com.android.runtime/lib64/bionic/libc.so (je_free+120) (BuildId: f46dcba52f09252fc392ba4a54b5b398)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #03 pc 000000000072ec88  /data/app/com.glassear.mediapipe-os5NXFlHNtJJpj4wZgMmmQ==/lib/arm64/libil2cpp.so (BuildId: 7820877e2a943eda750d97d3e3f2fd09f9fa1720)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #04 pc 0000000000963f20  /data/app/com.glassear.mediapipe-os5NXFlHNtJJpj4wZgMmmQ==/lib/arm64/libil2cpp.so (BuildId: 7820877e2a943eda750d97d3e3f2fd09f9fa1720)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #05 pc 000000000081cb34  /data/app/com.glassear.mediapipe-os5NXFlHNtJJpj4wZgMmmQ==/lib/arm64/libil2cpp.so (BuildId: 7820877e2a943eda750d97d3e3f2fd09f9fa1720)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #06 pc 0000000000b7c674  /data/app/com.glassear.mediapipe-os5NXFlHNtJJpj4wZgMmmQ==/lib/arm64/libil2cpp.so (BuildId: 7820877e2a943eda750d97d3e3f2fd09f9fa1720)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #07 pc 0000000000416620  /data/app/com.glassear.mediapipe-os5NXFlHNtJJpj4wZgMmmQ==/lib/arm64/libil2cpp.so (BuildId: 7820877e2a943eda750d97d3e3f2fd09f9fa1720)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #08 pc 0000000000c7aeb8  /data/app/com.glassear.mediapipe-os5NXFlHNtJJpj4wZgMmmQ==/lib/arm64/libmediapipe_jni.so (std::__ndk1::__function::__func<mp_CalculatorGraph__ObserveOutputStream__PKc_PF::$_0, std::__ndk1::allocator<mp_CalculatorGraph__ObserveOutputStream__PKc_PF::$_0>, absl::lts_2020_09_23::Status (mediapipe::Packet const&)>::operator()(mediapipe::Packet const&)+24)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #09 pc 0000000000cca974  /data/app/com.glassear.mediapipe-os5NXFlHNtJJpj4wZgMmmQ==/lib/arm64/libmediapipe_jni.so (mediapipe::internal::OutputStreamObserver::Notify()+176)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #10 pc 0000000000cc2ff4  /data/app/com.glassear.mediapipe-os5NXFlHNtJJpj4wZgMmmQ==/lib/arm64/libmediapipe_jni.so (std::__ndk1::__function::__func<mediapipe::CalculatorGraph::PrepareForRun(std::__ndk1::map<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, mediapipe::Packet, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::pair<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const, mediapipe::Packet> > > const&, std::__ndk1::map<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, mediapipe::Packet, std::__ndk1::less<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::pair<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const, mediapipe::P...
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #11 pc 0000000000cf2e1c  /data/app/com.glassear.mediapipe-os5NXFlHNtJJpj4wZgMmmQ==/lib/arm64/libmediapipe_jni.so (mediapipe::InputStreamHandler::AddPackets(mediapipe::CollectionItemId, std::__ndk1::list<mediapipe::Packet, std::__ndk1::allocator<mediapipe::Packet> > const&)+328)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #12 pc 0000000000cf13c8  /data/app/com.glassear.mediapipe-os5NXFlHNtJJpj4wZgMmmQ==/lib/arm64/libmediapipe_jni.so (mediapipe::OutputStreamManager::PropagateUpdatesToMirrors(mediapipe::Timestamp, mediapipe::OutputStreamShard*)+608)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #13 pc 0000000000ceff74  /data/app/com.glassear.mediapipe-os5NXFlHNtJJpj4wZgMmmQ==/lib/arm64/libmediapipe_jni.so (mediapipe::OutputStreamHandler::PropagateOutputPackets(mediapipe::Timestamp, mediapipe::internal::Collection<mediapipe::OutputStreamShard, (mediapipe::internal::CollectionStorage)0, mediapipe::internal::CollectionErrorHandlerFatal<mediapipe::OutputStreamShard> >*)+148)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #14 pc 0000000000ce60ac  /data/app/com.glassear.mediapipe-os5NXFlHNtJJpj4wZgMmmQ==/lib/arm64/libmediapipe_jni.so (mediapipe::CalculatorNode::ProcessNode(mediapipe::CalculatorContext*)+1616)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #15 pc 0000000000cd146c  /data/app/com.glassear.mediapipe-os5NXFlHNtJJpj4wZgMmmQ==/lib/arm64/libmediapipe_jni.so (mediapipe::internal::SchedulerQueue::RunCalculatorNode(mediapipe::CalculatorNode*, mediapipe::CalculatorContext*)+312)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #16 pc 0000000000cd0ec0  /data/app/com.glassear.mediapipe-os5NXFlHNtJJpj4wZgMmmQ==/lib/arm64/libmediapipe_jni.so (mediapipe::internal::SchedulerQueue::RunNextTask()+256)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #17 pc 0000000000d056dc  /data/app/com.glassear.mediapipe-os5NXFlHNtJJpj4wZgMmmQ==/lib/arm64/libmediapipe_jni.so (mediapipe::ThreadPool::RunWorker()+392)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #18 pc 0000000000d0530c  /data/app/com.glassear.mediapipe-os5NXFlHNtJJpj4wZgMmmQ==/lib/arm64/libmediapipe_jni.so (mediapipe::ThreadPool::WorkerThread::ThreadBody(void*)+1616)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #19 pc 00000000000d9770  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+36) (BuildId: f46dcba52f09252fc392ba4a54b5b398)
2021/02/21 12:47:22.631 15736 16271 Error CRASH       #20 pc 0000000000075f2c  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: f46dcba52f09252fc392ba4a54b5b398)

Got to print once and app crashed.

[elblogbruno]

FINALLY what a stupid error. The three where reading from the same stream.

graph.ObserveOutputStream(handLandmarksStream, HandleLandmarksCallback).AssertOk();
    graph.ObserveOutputStream(handLandmarksStream, HandlehandednessCallback).AssertOk();
    graph.ObserveOutputStream(handLandmarksStream, Handlepalm_detectionsCallback).AssertOk();

I did change so it read on every different stream.

graph.ObserveOutputStream(handLandmarksStream, HandleLandmarksCallback).AssertOk();
    graph.ObserveOutputStream(handednessStream, HandlehandednessCallback).AssertOk();
    graph.ObserveOutputStream(palmDetectionsStream, Handlepalm_detectionsCallback).AssertOk();

Thanks for all @homuler !

[elblogbruno]

How can I pass that data to other functions? I have seen that if the function is not static, I can't pass landmark info to other scripts.

[holaworlds]

Hi @homuler, I have same question as elblogbruno, how can we pass these values to another functions or store in a variable ? Since the method doesn't allow us to return anything except Status. Thanks

[homuler]

Callback functions are not meant to work that way. The callback function is called by MediaPipe, so we don't receive its return value. Instead, MediaPipe calls a function by passing output as an argument, and you can receive the data in the function and do what you want.

I will be releasing the sample code by the end of this month, so please check then for more details.