Which ImageFormat to use.

[elblogbruno]

Hello! As I told you on error #23 I am integrating ar foundation into this. You told me to create a ImageFrame, the thing is I don't know which Format Should be use, Glad if you or someone could help me out Cordially Bruno

[elblogbruno]

also can't seem to push and ImageFrame to a graph

[homuler]

Of course it depends on how the buffer (NativeArray<byte>) is aligned. But MediaPipe does not support all the format for all the calculations (e.g. you cannot convert SBGRA to GpuBuffer https://github.com/google/mediapipe/blob/master/mediapipe/gpu/gpu_buffer_format.cc#L176), and in such a case, this plugin will throw an error or aborts in the worst case.

also can't seem to push and ImageFrame to a graph

Probably you are using DemoGraph and DemoGraph receives TextureFrame, not ImageFrame.

By the way, I recommend you not using DemoGraph or scripts under Examples currently. Most of them are implemented only for the demonstration purpose (though some useful classes such as AssetBundleManager may be moved under SDK in the future).

[homuler]

If more generalized (platform independent) graph implementation is found, it will be placed under SDK, but not yet.

[elblogbruno]

So I can simply pass an ImageFrame like in the picture? I have seen TextureFrame has some callbacks and things.

[elblogbruno]

hmm...

[elblogbruno]

i am looking at the logs

[homuler]

I have seen TextureFrame has some callbacks and things.

If you pass OpenGL ES textures directly to MediaPipe, those callbacks are necessary, but as long as you pass ImageFrames to MediaPipe, you can ignore them.

[elblogbruno]

thanks just for trying to get it work I am modifying the DemoGraph

[Codeavr]

If you are wishing to integrate ARFoundation, consider that XRCpuImage.Convert method returns byte array which you can directly pass to the input stream. No need in glcontext, callbacks and etc. Works good for me

[elblogbruno]

Yes exactly @Codeavr I do get a vyte array but don't know how to pass it to the input stream! Could you shine a light on me please? Thanks

[elblogbruno]

i was trying to create an ImageFrame out of it but I can't get it to work

[elblogbruno]

Hi @Codeavr thanks for answering. How do you pass that imageFrame to the graph?

[Codeavr]

@elblogbruno Well, in my project I have quite different pipeline. I need converted video texture preview, so I load ARFoundation texture to Texture2D

//unsafe context
image.Convert(conversionParams, new IntPtr(buffer.GetUnsafePtr()), buffer.Length);

_texture.LoadRawTextureData(buffer);
_texture.Apply();

then I create ImageFrame using this texture pointer to data and pass it to the input stream

imageFrame = new ImageFrame
(
    ImageFormat.Format.SRGBA,
    _texture.width,
    _texture.height,
    4 * _texture.width,
    _texture.GetRawTextureData<byte>()
);

var packet = new ImageFramePacket(imageFrame, timestamp);

_graph.AddPacketToInputStream(InputStream, packet);

There must be more performing way to this, but this save/load overhead is not that high

[elblogbruno]

ooh thanks!

[elblogbruno]

but are you planning to work on Android? I believe for android more things should be done, does this current setup work on android for you?

[Codeavr]

but are you planning to work on Android? I believe for android more things should be done, does this current setup work on android for you?

I started working with this plugin before homuler implemented async gpu texture loading (those things with texture pooling and callbacks), and I'm still at old implementation on android which look kinda:

var status = _gpuHelper.RunInGlContext(() => {
    var texture = _gpuHelper.CreateSourceTexture(imageFrame);
    var gpuFrame = texture.GetGpuBufferFrame();
    texture.Release();

    return _graph.AddPacketToInputStream(InputStream, new GpuBufferPacket(gpuFrame, timestamp));
});

I'm not sure if it's completely correct, but works for me

[elblogbruno]

hhm thanks!

[elblogbruno]

@Codeavr how you are rendering the output?

[Codeavr]

@elblogbruno if you are about annotations, I use orthographic projection on my camera and just drawing landmarks into linerenderers

[elblogbruno]

Well I was speaking about graph.renderoutput() function. I will make the annotations myself in the near future , I still need to get image frame passing working. I can't seem to be able. Thanks @Codeavr

[homuler]

var status = _gpuHelper.RunInGlContext(() => {
    var texture = _gpuHelper.CreateSourceTexture(imageFrame);
    var gpuFrame = texture.GetGpuBufferFrame();
    texture.Release();

   return _graph.AddPacketToInputStream(InputStream, new GpuBufferPacket(gpuFrame, timestamp));
 });

It's correct, but using ImageFrameToGpuBufferCalculator, you can simply call AddPacketToInputStream with ImageFramePacket and it's faster (ImageFrameToGpuBufferCalculator does the same thing in native code).

https://github.com/homuler/MediaPipeUnityPlugin/blob/master/Assets/MediaPipe/Examples/Graphs/FaceDetection/Resources/face_detection_desktop_gpu.txt#L51-L55

using (var packet = new ImageFramePacket(imageFrame, timestamp)) {
  var status = _graph.AddPacketToInputStream(InputStream, packet);
}

I believe for android more things should be done, does this current setup work on android for you?

On Android, you can pass texture pointers to MediaPipe, and the setup becomes complicated if you'd like to do so, but you don't have to do so. Desktop GPU samples should work on Android too (if UNITY_ADNROID pragma is removed. Probably I should have used custom directives).

[homuler]

I was speaking about graph.renderoutput() function.

Except for OfficialDemoAndroid, DemoGraph#RenderOutput is not platform dependent.

[Codeavr]

@elblogbruno I don't understand the question

[elblogbruno]

On Android, you can pass texture pointers to MediaPipe, and the setup becomes complicated if you'd like to do so, but you don't have to do so.

But passing the pointer is a method that improves the performance right?

Desktop GPU samples should work on Android too (if UNITY_ADNROID pragma is removed. Probably I should have used custom directives).

I will continue today trying to get it working on Android! Thanks for the support and tips

[elblogbruno]

@elblogbruno I don't understand the question

Nevermind it was a nonsense question I answered myself 😂

[elblogbruno]

What could this be?

[homuler]

Please uncomment this line, then build and rerun. https://github.com/homuler/MediaPipeUnityPlugin/blob/master/C/WORKSPACE#L48

Glog will output the error log. Or if you can, set GLOG_v=2 and run it.

[elblogbruno]

ok thanks! I will debug and return here with an answer!

[elblogbruno]

Log file created at: 2021/01/29 13:34:57 Running on machine: localhost Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg W20210129 13:34:57.718008 27261 calculator_graph.cc:939] INVALID_ARGUMENT: Packet type mismatch on calculator outputting to stream "input_video": The Packet stores "mediapipe::ImageFrame", but "mediapipe::GpuBuffer" was requested.

[elblogbruno]

what could this be?

[homuler]

Maybe you are pushing ImageFramePackets to input_video, but input_video should be defined as GpuBuffer. Will you share the config you use?

[elblogbruno]

I am using the hand_tracking_android.txt config. https://pastebin.com/izyJ48J6 Here is what I have modified from DemoGraph.cs https://pastebin.com/6UUSz9QJ

And here is the WebcamScreenController.cs modified with arFoundation and pushing imageFrame to modified DemoGraph.cs https://pastebin.com/rhiS8SQr

[elblogbruno]

I did switched from outputTexture to the buffer byte array, and now I get it to show but with a poor,poor performance. Using hand_tracking_gpu config

[homuler]

Stop rendering annotations temporarily, and try ObserveOutputStream as described here https://github.com/homuler/MediaPipeUnityPlugin/issues/23#issuecomment-763247582 . In this case, you can remove PacketPresenceCalculators from the config.

You cannot get output packets faster than this way, so I recommend you check the maximum performance first. (To be precise, if you prepare specialized APIs to send image data, I think you can improve the performance, but I don't know how much it will be.)

[elblogbruno]

hmm ok thanks!

[elblogbruno]

and can I pass an ImageFrame to the OfficialGraph? sorry for asking lot of questions recently

[homuler]

Yes, but note that you need to convert ImageFrame to GpuBuffer (https://github.com/homuler/MediaPipeUnityPlugin/issues/43#issuecomment-769498179) if you use official_demo_android.txt. OfficialDemoGpu would also work, but this graph does not render the output image directly to Texture2D.

[kitmek1]

C:\Users\kitme\OneDrive\Desktop\New folder (2)\MediaPipeUnityPlugin>make cpu ./.build/protobuf-3.13.0/csharp/buildall.sh process_begin: CreateProcess(NULL, bash C:\Users\kitme\OneDrive\Desktop\New folder (2)\MediaPipeUnityPlugin.build\protobuf-3.13.0\csharp\buildall.sh, ...) failed. make (e=2): The system cannot find the file specified. Makefile:61: recipe for target '.build/protobuf-3.13.0/csharp/src/Google.Protobuf/bin/Release/net45/Google.Protobuf.dll' failed mingw32-make.exe: *** [.build/protobuf-3.13.0/csharp/src/Google.Protobuf/bin/Release/net45/Google.Protobuf.dll] Error 2 i have this error while loading model can you help me...

[elblogbruno]

Yes, but note that you need to convert ImageFrame to GpuBuffer (#43 (comment)) if you use official_demo_android.txt. OfficialDemoGpu would also work, but this graph does not render the output image directly to Texture2D.

Thanks!

[elblogbruno]

What could this be?

[elblogbruno]

C:\Users\kitme\OneDrive\Desktop\New folder (2)\MediaPipeUnityPlugin>make cpu ./.build/protobuf-3.13.0/csharp/buildall.sh process_begin: CreateProcess(NULL, bash C:\Users\kitme\OneDrive\Desktop\New folder (2)\MediaPipeUnityPlugin.build\protobuf-3.13.0\csharp\buildall.sh, ...) failed. make (e=2): The system cannot find the file specified. Makefile:61: recipe for target '.build/protobuf-3.13.0/csharp/src/Google.Protobuf/bin/Release/net45/Google.Protobuf.dll' failed mingw32-make.exe: *** [.build/protobuf-3.13.0/csharp/src/Google.Protobuf/bin/Release/net45/Google.Protobuf.dll] Error 2 i have this error while loading model can you help me...

make sure you have installed protobuf, but please don't post your issue on a thread about a non-related issue to your issue.

[homuler]

TL;DR Call RunInGlContext(NativeGlStatusFunction) instead or use Mono as the scripting backend.

GlStatusFunction returns Status, but MediaPipe (C++) doesn't understand Status (C# object), so when RunInGlContext(GlStatusFunction is called, this plugin converts the GlStatusFunction to an anonymous NativeGlStatusFunction (returns a native pointer). https://github.com/homuler/MediaPipeUnityPlugin/blob/09cd496124811a3e2392bafbb07f6825dfc2988d/Assets/MediaPipe/SDK/Scripts/Gpu/GlCalculatorHelper.cs#L37-L56

However, IL2CPP does not support marshaling non-static method to native code, and Unity thorws the above error if we try to do that or pin function pointers to instance methods (the latter in the above case). To avoid this error, define a static NativeGlStatusFunction and call RunInGlContext(NativeGlStatusFunction) with it instead.

See also ↓. https://github.com/homuler/MediaPipeUnityPlugin/blob/09cd496124811a3e2392bafbb07f6825dfc2988d/Assets/MediaPipe/Examples/Graphs/OfficialDemo/Scripts/OfficialDemoAndroid.cs#L56-L67

[elblogbruno]

I will try that! Thanks for all the help 😊

[elblogbruno]

I am still getting the same issue wth.

[homuler]

Will you share the related code? I cannot explain more clearly than https://github.com/homuler/MediaPipeUnityPlugin/issues/43#issuecomment-770498550 .

By the way, it'd be much better to copy code, logs, etc... in text format than in image (e.g. screenshot). It's difficult for us to copy texts in image or control the size of them, and those texts won't be indexed by search engine.

[elblogbruno]

Sure, here is my modified code https://pastebin.com/tfY5tX7u using #43 (comment). Also here is the code that creates and pushes the ImageFrame https://pastebin.com/8FeaWH6E Sorry about sending pictures before it was a quick and syntax highlighted way of sharing. Hope you see what I am doing wrong. Thanks

[homuler]

NativeGlStatusFunction should return IntPtr, not Status.

  [AOT.MonoPInvokeCallback(typeof(GlCalculatorHelper.NativeGlStatusFunction))]
  static IntPtr GetStatusAndroid() {
    var texture = gpuHelper.CreateSourceTexture(currentTextureFrame);
    var gpuFrame = texture.GetGpuBufferFrame();
    texture.Release();
  Debug.Log("Getting status android");
    return graph.AddPacketToInputStream(inputStream, new GpuBufferPacket(gpuFrame, currentTimestamp)).mpPtr;
  }

Note that the above code may throw exceptions, and C++ cannot handle them. e.g. https://github.com/homuler/MediaPipeUnityPlugin/blob/09cd496124811a3e2392bafbb07f6825dfc2988d/Assets/MediaPipe/Examples/Scripts/DemoGraph.cs#L69-L82

[elblogbruno]

Thanks for the help!

[elblogbruno]

I got it working 😄 but can't read from the stream consistently. Hand annotations are shown as always but as you see by the code, i am debugging some values.

public override void RenderOutput(WebCamScreenController screenController, ImageFrame textureFrame) { var handTrackingValue = FetchNextHandTrackingValue(); RenderAnnotation(screenController, handTrackingValue); Debug.Log(handTrackingValue.PalmDetections.Count); Debug.Log(handTrackingValue.Handednesses[0]); screenController.DrawScreen(textureFrame); }

Values are debugged for some seconds until it stops with the unexpected END OF FILE issue. Even though values are not being printed out on the console annotations still appear on screen. I will also try now to debug landmarks on the official graph. (i am getting better performance with official graph, but can't print out values on console)

2021/02/02 12:01:53.617 21374 21908 Info native I20210202 12:01:53.617295 21908 scheduler.cc:455] active queues: 1 2021/02/02 12:01:53.617 21374 21904 Info native I20210202 12:01:53.617719 21904 calculator_node.cc:812] Calling Calculator::Process() for node: handlandmarktrackinggpu__handlandmarkgpu__ImageToTensorCalculator timestamp: 36 2021/02/02 12:01:53.618 21374 21904 Info native I20210202 12:01:53.618667 21904 calculator_node.cc:825] Called Calculator::Process() for node: handlandmarktrackinggpu__handlandmarkgpu__ImageToTensorCalculator timestamp: 36 2021/02/02 12:01:53.618 21374 21904 Info native I20210202 12:01:53.618875 21904 calculator_node.cc:812] Calling Calculator::Process() for node: handlandmarktrackinggpu__handlandmarkgpu__InferenceCalculator timestamp: 36 0001/01/01 00:00:00.000 -1 -1 Info read: unexpected EOF! Again thanks for your help @homuler !

[homuler]

Will you set GLOG_v=5 and share the full log (Glog and Unity log), please? cf. https://github.com/homuler/MediaPipeUnityPlugin/blob/master/Assets/MediaPipe/Examples/Scripts/SceneDirector.cs#L32