mediapipe_data to osc

[jossevessies]

Comparison Product

No response

Feature Category

Enhancement

Feature Description

Dear @endink !

Thanks again for a great tool. I am wondering to contribute. Besides the mocap I'd also like to use the mediapipe_coordinates to move particles around in touchdesigner and overlay this with the mocapped charachter from ue.

I think to dive into the components code and make an extra one, that exposes the media_pipe coordinates into an array. But It would be my first journey into c++. I'm experienced with python/c# & js.

Would you be interested in helping me out with this facility and I can contribute to this git?

Let me know,

sjoes

[jossevessies]

It comes down to me having a few questions:

1. is the build that we receive after downloading the plugin editable? Or do we need an 'unbuilded' version of it? (I am not familiari with 'builds', I know its a basic c++ thing. I'm just being honest haha.

2. Could you point me in the right direction? In which part of the plugins are you appealing to the mediapipe array of positions?

cheers!

[endink]

Hi @jossevessies

Sorry, its a precompiled plugin, you can‘t modify any code, its like a C# library （windows dll, linux so）. The difference is that C# puts the function and class information in the dll (IL symbo), and C++ puts it in the header file .

But you can develop your logic base on this plugin ( include the C++ header file in your project ).

Due to the large size of these precompiled files, they are not suitable for the git development pipline.

If you want to handle the mediapipe raw data, please see UMediaPipeHolisticComponent.h file.

It has some "Trigger" function, Trigger means a C++ native event ( like event/delegate in C# ), you can consume mediapipe data by subscribing to these events.

You can also subscribe these events to hook the lifecycle of the mediapipe graph, but this requires you to be very familiar with google mediapipe.

It's also important to note that MediaPipe4U uses multithreading, and the native event callbacks is not in game thread, so if you need to interact with the UE, you may need to post your callback to the game thread.

//UMediaPipeHolisticComponent.h

FLandmarksOutEvent& PoseLandmarksEventTrigger() { return PoseLandmarksEvent; }
FLandmarksOutEvent& LeftHandLandmarksEventTrigger() { return LeftHandLandmarksEvent; };
FLandmarksOutEvent& RightHandLandmarksEventTrigger() { return RightHandLandmarksEvent; }
FLandmarksOutEvent& FaceLandmarksEventTrigger() { return FaceLandmarksEvent; }
FLandmarksOutEvent& PoseWorldLandmarksEventTrigger() { return PoseWorldLandmarksEvent; }
FaceGeometryOutEvent& FaceGeometryEventTrigger() { return FaceGeometryEvent; }
FClassificationsOutEvent& FaceBlendShapesEventTrigger() { return FaceBlendShapesEvent; }
FVideoTextureCreatedEvent& VideoTextureCreatedEventTrigger() { return VideoTextureCreatedEvent; }

FOnMediaPipeHolisticStateChanged& OnMediaPipeHolisticStoppedTrigger() { return OnStoppedEvent; }
FBeforeMediaPipeHolisticStart& BeforeMediaPipeHolisticStartTrigger() { return BeforeStartEvent;  }
FOnMediaPipeHolisticStateChanged& OnMediaPipeHolisticStartFailedTrigger() { return OnStartFailedEvent;  }
FOnMediaPipeHolisticStateChanged& OnMediaPipeHolisticStartedTrigger() { return OnStartedEvent; }

[github-actions[bot]]

This issue is stale because it has been open for 15 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.