Running the project inside a Windows 11 PC using an external camera and TouchDesigner

[julscloudops]

Hello Jerry,

I am very interesting in implementing this project in a single Windows workstation that has the required specs. Could you advise me how to go about setting up the smart camera system to send pose estimation data to TD on the same computer to run a 3D model animation? Excuse me if this question is obvious but it seems the code is adapted to running on a small raspberry pi.

Best regards and thank you for sharing your work!

[MauiJerry]

Aloha! Single PC? That is the simple part of things. Check out the Hackaday article :
https://hackaday.io/project/188345-pose2art-smartcam-to-touchdesigner-unity-via-osc in particular look into Log 3: PC Code - Python and TouchDesigner I based mine on Cronin4392's work https://github.com/cronin4392/TouchDesigner-OpenCV-OSC I am in the process of revising the Python and TD so the 17 points will match up in TD as named channels with same names as the corresponding Kinect joints. This will make it a LOT easier to use TD apps designed for Kinect/Azure, possibly even supporting both.

I am also digging further into PoseNet (on a google Coral device) as it can handle multiple people.

Once the OSC skeletons are transmitting on network, Unity and Unreal Engine could also read them to puppet 3d avatars. Those are future projects for me.

[julscloudops]

Awesome! I will take a look at that.

Did you ever try doing this using the OpenPose project for body tracking? I found this implementation of it on TD and it seems to work fine: https://github.com/yohki/td_openpose_sample

[MauiJerry]

I havent seen that one -5y old. I did just recently find this one https://github.com/chungbwc/TouchDesigner/tree/main/MediaPipePoseTOP dude has a lot of cool stuff to investigate. One issue (perhaps) with these inside Touch is whether they properly multiprocess. Python in particular has issues with real threads. Doing the OpenCV/ML in a separate process or on separate computer avoids that bottleneck.

[julscloudops]

Will check that one out as well.

I was able to install OpenPose and use it sucessfully as a standalone app but the problem is that it doesn't have OSC implementation and I haven't found a way to use the keypoint information inside of Touch, it saves the keypoint information as JSON files and one might parse them and use them inside of TD but I believe the implementation process of that might be really hard and not worth the effort, it's like recreating the Kinect CHOP in a way. Like you can record keypoint data and use it in an animation but I am looking for real-time motion capture solution to animate a rigged 3D model (like what can be done with the Kinect CHOP).

I found this https://medium.com/@asus4/openpose-with-spout-input-and-osc-output-f02d25e2c578 which seems to implement OSC but isn't really documented and didn't manage to get it to work and https://qiita.com/yohki/items/317a0d96a7494441439a (implementation of OpenPose inside of TD) but it requires the OpenPose Python API which is a nightmare to compile on Windows.

At this point I'm a bit stuck, I am looking to see if I can find some cheap Xbox 360 Kinect or try some other solutions. I am down for chatting/brainstorming or to see if I can help you in your project as I believe you are trying to accomplish something similar. My Discord username is PRODHAB#2164 if you are interested.

[MauiJerry]

I was able to mod some of the python/c++ pose software examples to export via OSC and read them into Touch using the OSCin Chop. I recently got a Google Coral dev board and hooked it up with a usb web cam (intel realsense actually). The better example code there uses the GPU and I'm not familar with that, yet.